How are samples collected by honest profiler

Honest-Profiler uses an OpenJDK internal API call AsyncGetCallTrace to facilitate non-safepoint (learn about safepoints here) collection of stack traces. AsyncGetCallTrace is NOT official JVM API. It's original use case was for Solaris Studio, and it provides the following API (see forte.cpp, the name is a left over from the Forte Analyzer days). Here's what the API adds up to:

typedef struct {
  jint lineno;         // BCI in the source file
  jmethodID method_id; // method executed in this frame
} ASGCT_CallFrame;

typedef struct {
  JNIEnv *env_id   //Env where trace was recorded
  jint num_frames; // number of frames in this trace
  ASGCT_CallFrame *frames;
} ASGCT_CallTrace; 

void AsyncGetCallTrace(ASGCT_CallTrace *trace, // pre-allocated trace to fill
                       jint depth,             // max number of frames to walk up the stack
                       void* ucontext)         // signal context

You give it a ucontext and it fills in the trace with call frames.

The 'async' in the name refers to the fact that AGCT is safe to call in a signal handler. The workflow ends up as follows:

• In a JVMTI agent, register a signal handler for signal X.
• Setup a timer, triggering the signal X at the desired sample frequency (e.g N samples per second). Honest Profiler is using the ITIMER_PROF option, which means we'll get signalled based on CPU time (not wall clock time. In particular if the application is consuming more than one CPU to 100% in a given second we will see more than N samples per wall clock second). The signal will be sent to the process and one of the running threads will end up being interrupted and calling into our signal handler. Note that this assumes the OS will distribute the signals fairly between threads so we will get a fair sample of all running threads.
• From signal handler, call AGCT: Note that the interrupted thread (picked at 'random' from the threads currently executing on CPU) is now running your signal handler. The thread is NOT AT A SAFEPOINT. It may not be a Java thread at all.
• Persist the call trace: Note that when running in a signal handler only 'async' code is legal to run. This means for instance that any blocking code is forbidden, including malloc and IO. Honest profiler persists the call trace into a lock free MPSC ring buffer.

The ucontext ingredient is the very same context handed to you by the signal handler (your signal handler is a callback of the signature handle(int signum, siginfo_t *info, void *context)). From it AGCT will dig up the instruction/frame/stack pointer values at the time of the interrupt and do it's best to find out where you are and what the call stack is.

For further details/discussion see this blog post on the pros and cons of AGCT profilers.