This repository contains the core New Relic JFR components. It is used to consume JFR events, transform them into New Relic telemetry, and send them to New Relic (with the New Relic Telemetry SDK) where they can be visualized in a number of ways.
This repository contains the following modules:
- jfr-daemon - A monitoring tool that uses a rotating fileset to consume JFR events and near-continuously send telemetry data to New Relic.
- jfr-mappers - Mappers that transform JFR
RecordedEventobjects into New Relic telemetry data. Also contains registries of all supported mappers. This is used as an implementation library byjfr-daemon. - jfr-jlink - A module that produces jlink binaries of
jfr-daemonthat include a bundled JRE. - jfr-tools - Tools that can be used to analyze
jfr-daemon. - smoke-tests - Smoke tests for the different
jfr-daemonusage scenarios.
This project uses Java 11 and the gradle wrapper. To build it, run:
$ git clone https://github.com/newrelic/newrelic-jfr-core.git
$ cd newrelic-jfr-core
$ ./gradlew build
The resulting jars of interest are:
jfr-mappers/build/libs/jfr-mappers-<version>.jarjfr-daemon/build/libs/jfr-daemon-<version>.jar
Unit tests are run with gradlew:
$ ./gradlew test
There are three different usage scenarios for the jfr-daemon:
- New Relic Java agent JFR service (RECOMMENDED) - JFR monitoring built into the flagship New Relic Java agent. No additional setup needed, simply install the Java agent, make sure that the JFR service is enabled, and JFR data will flow into the same APM application as the Java agent.
- Standalone process - Run the
jfr-daemonas a standalone process and configure it to monitor an existing Java process using remote JMX. - Standalone Java agent - Attach the
jfr-daemonto your Java process as a Java agent. A lightweight alternative to the New Relic Java agent.
For additional details see:
While the jfr-daemon supports any version of Java 11 and above, we do not recommend using any non-LTS version of Java in production environments.
Some vendors have backported JFR to their Java 8 binaries. For instance, OpenJDK backported JFR on version 8u262. The jfr-daemon is compatible with those Java versions.
The recommended way to get JFR data is to use the New Relic Java agent with builtin JFR monitoring. This will provide the richest experience by combining data collected by the New Relic Java agent with JFR data together in a single New Relic One entity.
The New Relic Java agent JFR monitoring has the following requirements:
- New Relic Java agent version 7.0.0+
- APM license key
- JFR must be enabled in the agent config
- Uses a supported Java version
Notice: If you were previously using the jfr-daemon as an agent extension or standalone process you should remove that option to avoid potential conflicts with the JFR service that is now built into the agent.
Install the New Relic Java agent
by adding the -javaagent parameter to your JVM properties and enable JFR monitoring using one of the following mechanisms:
Agent yaml config (the following should be nested under the common stanza):
# Real-time profiling using Java Flight Recorder (JFR).
# This feature reports dimensional metrics to the ingest endpoint configured by
# metric_ingest_uri and events to the ingest endpoint configured by event_ingest_uri.
# Both ingest endpoints default to US production but they will be automatically set to EU
# production when using an EU license key. Other ingest endpoints can be configured manually.
# Requires a JVM that provides the JFR library.
jfr:
# Set to true to enable Real-time profiling with JFR.
# Default is true.
enabled: true
# Set to true to enable audit logging which will display all JFR metrics and events in each harvest batch.
# Audit logging is extremely verbose and should only be used for troubleshooting purposes.
# Default is false.
audit_logging: falseSystem property:
-Dnewrelic.config.jfr.enabled=true
Environment variable:
export NEW_RELIC_JFR_ENABLED=true
By default, the New Relic Java agent will send JFR data to New Relic US production metric and event ingest endpoints.
JFR data will be associated with the APM entity generated by the New Relic Java agent as defined by your app_name and license_key.
To change this or other behavior see configuration for New Relic Java agent JFR service.
When using the New Relic Java agent, the JFR service fully respects agent configuration settings.
The New Relic Java agent is configurable via a yaml configuration file, system properties, and environment variables.
The following agent configuration options affect how and where JFR data is reported:
- app_name
- license_key
- jfr.enabled
- jfr.audit_logging
- metric_ingest_uri
- event_ingest_uri
- proxy_host
- proxy_password
- proxy_port
- proxy_user
- proxy_scheme
The jfr-daemon can be used as a stand-alone process that consumes JFR events
from an existing java process and sends telemetry to New Relic. This daemon
process issues commands over JMX to periodically generate a series of rolling
JFR files. It uses these files to build a "pseudo stream" of telemetry events.
The jfr-daemon standalone process has the following requirements:
- An Insights insert API key or
an APM license key.
To use an APM license key you will need to configure the JFR daemon with
USE_LICENSE_KEY=true. - Uses a supported Java version
The minimum requirements to use the jfr-daemon as standalone process are as follows.
Set the app name that the JFR data should be reported to, and an Insights insert key (to use an APM license key also add export USE_LICENSE_KEY=true):
export NEW_RELIC_APP_NAME=<NAME>
export INSIGHTS_INSERT_KEY=<KEY>
Start the jfr-daemon standalone process, it will attempt to connect to your application's remote JMX MBean server:
java -jar jfr-daemon-<version>.jar
By default, the JFR daemon will connect JMX to localhost on port 1099 and send data to New Relic US production metric and event ingest endpoints.
To change this or other behavior see configuration for standalone usage.
The target application that you wish to monitor must be configured to expose JFR data over remote JMX by adding the following system properties:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
The jfr-daemon can be used as a Java agent that consumes JFR events
by attaching to a java process and sending telemetry to New Relic.
This can be used as a lightweight alternative to the New Relic Java agent JFR service,
if you do not need the additional data captured by the New Relic Java agent, or you require a solution with less overhead.
The jfr-daemon standalone Java agent has the following requirements:
- An Insights insert API key or
an APM license key.
To use an APM license key you will need to configure the JFR daemon with
USE_LICENSE_KEY=true. - Uses a supported Java version
The minimum requirements to use the jfr-daemon as standalone Java agent are as follows.
Set the app name that the JFR data should be reported to, and an Insights insert key (to use an APM license key also add export USE_LICENSE_KEY=true):
export NEW_RELIC_APP_NAME=<NAME>
export INSIGHTS_INSERT_KEY=<KEY>
Add the -javaagent parameter to your JVM properties:
-javaagent:/path/to/jfr-daemon-<version>.jar
By default, the JFR daemon will send data to New Relic US production metric and event ingest endpoints. To change this or other behavior see configuration for standalone usage.
The environment variables in this section apply when using the jfr-daemon as a standalone process
or standalone Java agent, though the JMX settings only apply to the standalone process.
When using the New Relic Java agent JFR service the configuration is done through the agent config mechanisms (see configuration for New Relic Java agent JFR service).
| env var name | required? | default | description |
|---|---|---|---|
INSIGHTS_INSERT_KEY |
Y | n/a | The New Relic insert key or APM license key for your account |
USE_LICENSE_KEY |
N | false |
Use a License Key instead of Insights Insert Key. |
NEW_RELIC_APP_NAME |
N(!) | My Application |
The name of the remote application being monitored. You should probably set this so that your application shows up properly in the NR1 platform. |
REMOTE_JMX_HOST |
N | localhost |
The host to pull JFR data from via JMX |
REMOTE_JMX_PORT |
N | 1099 |
The port to pull JFR data from via JMX |
METRICS_INGEST_URI |
N | US production, EU production | Where to send metric data |
EVENTS_INGEST_URI |
N | US production, EU production | Where to send event data |
JFR_SHARED_FILESYSTEM |
N | false |
Use a shared filesystem instead of streaming data from JMX |
AUDIT_LOGGING |
N | false |
Enables audit logging in the underlying Telemetry SDK |
PROXY_HOST |
N | null |
Proxy host name |
PROXY_PORT |
N | null |
Proxy host port |
PROXY_USER |
N | null |
Proxy user name |
PROXY_PASSWORD |
N | null |
Proxy password |
PROXY_SCHEME |
N | null |
Proxy scheme (http or https) |
HARVEST_INTERVAL |
N | 10 |
How often data from JFR will be sent to New Relic |
QUEUE_SIZE |
N | 250_000 |
How many events are to be sent during each harvest cycle |
THREAD_NAME_PATTERN |
N | ((?<=[\W_]|^)([0-9a-fA-F]){4,}(?=[\W_]|$))|\d+ |
A regex pattern that will be run against thread names, matches will be converted to a # to group those threads |
The JFR daemon and the underlying Telemetry SDK logs with the Slf4j-Simple implementation at the default Info level.
To increase the logging level, you can configure Slf4j with system properties, or a simplelogger.properties file.
For example, set this on the command line to log at the debug level:
-Dorg.slf4j.simpleLogger.defaultLogLevel=debug
Here is a sample of debug level logs.
[JfrController] DEBUG com.newrelic.telemetry.metrics.MetricBuffer - Creating metric batch.
[JfrController] INFO com.newrelic.jfr.daemon.JFRUploader - Sending metric batch of size 61
[JfrController] DEBUG com.newrelic.telemetry.events.EventBuffer - Creating Event batch.
[Thread-0] DEBUG com.newrelic.telemetry.metrics.MetricBatchSender - Sending a metric batch (number of metrics: 61) to the New Relic metric ingest endpoint)
[Thread-0] DEBUG com.newrelic.telemetry.metrics.json.MetricBatchMarshaller - Generating json for metric batch.
[JfrController] INFO com.newrelic.jfr.daemon.JFRUploader - Sending events batch of size 25This module produces executable jlink binaries that bundle the jfr-daemon with a JRE that
provides all required libraries. This allows you to run the jfr-daemon as a standalone process without needing to explicitly
provide a separate JRE for the daemon.
The produced binaries can be found in newrelic-jfr-core/jfr-jlink/build/jlink/jfr-jlink-<version>/bin/.
This has the same configuration and requirements as the jfr-daemon standalone process with the
lone exception being that the binaries can be directly executed without explicitly starting a JVM (e.g. ./jfr-daemon).
This module is a library of reusable JFR (Java Flight Recorder) mappers
used to transform JFR RecordedEvent instances into New Relic telemetry collections
that are compatible with the
telemetry SDK.
We don't intend this library to be used directly. Instead, leverage tools like the JFR daemon that are built upon this library.
<dependency>
<groupId>com.newrelic</groupId>
<artifactId>jfr-mappers</artifactId>
<version>1.9.0</version>
</dependency>
compile group: 'com.newrelic', name: 'jfr-mappers', version: '1.9.0'
A module providing tools that can be used to analyze jfr-daemon. See jfr-tools/README.md for details.
Consists of a SmokeTestApp and a suite of tests to verify that the jfr-daemon attaches and sends data for the various usage scenarios.
Requirements:
- Java 11
- Docker
New Relic One provides a detail rich user experience designed to provide immediate value from
Realtime profiling with JFR metrics.
Additionally, all JFR events and dimensional metrics reported by the jfr-daemon are queryable and facilitate building
custom dashboards and
New Relic One applications.
To view your data in the Realtime Profiling Java UI, go to one.newrelic.com and navigate to the following:
Explorer > All entities (select the entity) > More Views > Realtime Profiling Java
The JVM cluster timeline view shows the JVM behavior across the cluster.
Select How is JVM health determined? for a detailed breakdown of how JVM health is calculated.
Detailed view for individual JVMs.
The flamegraph view for individual JVMs (currently only available when JFR data is linked to an APM entity generated by New Relic Java agent).
Use the Data explorer to dig into JFR data.
JFR events.
JFR dimensional metrics.
Should you need assistance with New Relic products, you are in good hands with several support channels.
Support Channels
- New Relic Documentation: Comprehensive guidance for using our platform
- New Relic Community: The best place to engage in troubleshooting questions
- New Relic Developer: Resources for building a custom observability applications
- New Relic University: A range of online training for New Relic users of every level
At New Relic we take your privacy and the security of your information seriously, and are committed to protecting your information. We must emphasize the importance of not sharing personal data in public forums, and ask all users to scrub logs and diagnostic information for sensitive information, whether personal, proprietary, or otherwise.
We define “Personal Data” as any information relating to an identified or identifiable individual, including, for example, your name, phone number, post code or zip code, Device ID, IP address, and email address.
For more information, review New Relic’s General Data Privacy Notice.
We encourage your contributions to improve newrelic-jfr-core! Keep in mind that when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project.
If you have any questions, or to execute our corporate CLA (which is required if your contribution is on behalf of a company), drop us an email at opensource@newrelic.com.
A note about vulnerabilities
As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.
If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.
If you would like to contribute to this project, review these guidelines.
To all contributors, we thank you! Without your contribution, this project would not be what it is today.
newrelic-jfr-core is licensed under the Apache 2.0 License.
newrelic-jfr-core also uses source code from third party libraries. Full details on which libraries are used and the terms under which they are licensed can be found in the third party notices document.