The Lies Your JVM Profiler Tells You: Safepoints, Biased Sampling, and WhyYour Hotspot Analysis Is Wrong
You ran your profiler. You found the hotspot. You optimised it. Performance didn’t change. Here’s the precise technical reason why — and how to actually find where your time goes.
1. The Lie, Stated Precisely
The hotspot your profiler highlighted is probably not where your application is spending its time. It is where your profiler was most likely to be standing when the JVM permitted it to look.
This is not a metaphor. It is a direct consequence of how the majority of Java profilers work at the mechanical level, and it has been formally documented since at least a 2010 academic study titled “Evaluating the Accuracy of Java Profilers” — and arguably understood by JVM engineers even earlier. The study found that different Java profilers identify entirely different hotspots on the same benchmarks. Not slightly different hotspots. Different ones entirely. And the reason is not that the profilers have bugs. It is that they are faithfully reporting what they can observe — which is not the same thing as where your program actually spends its time.
Furthermore, as a 2024 ACM SIGPLAN paper on profiler accuracy confirmed, even instrumentation-based profilers — the alternative to sampling — have their own category of lies. The profiler with the lowest overhead in their study still increased application runtime by 82×. That overhead is not neutral: it changes what the JIT compiles, how aggressively it inlines, and consequently what the profiler observes. Both categories of Java profiler have systematic problems. Understanding them is the prerequisite to using them correctly.
2. What a Safepoint Actually Is
To understand safepoint bias, you first need to understand what a safepoint is at the JVM level — not the simplified description, but the actual mechanics.
A safepoint is a point in a thread’s execution where the JVM can guarantee that all object references are in known, walkable locations. Specifically: all threads are in a state where the JVM runtime can inspect their stacks, identify all live object references, and perform operations that require a consistent heap view. The most visible operation requiring all threads to reach a safepoint is Stop-the-World garbage collection. But safepoints are also used for deoptimisation, biased locking revocation, class redefinition, and — critically for our topic — profiler stack sampling.
Safepoints are not uniformly distributed through your code. The HotSpot JIT compiler inserts safepoint poll instructions at specific, deliberate locations. In JIT-compiled code, these are primarily at method exit and at backward loop edges in counted loops. Crucially, they are absent from the interior of compiled method bodies and absent from uncountable loops that the JIT has not instrumented. The result is a fundamentally non-uniform distribution: some points in your code are reachable as safepoints, and many are not.
Where safepoint polls are inserted in HotSpot JIT code: In C1/C2 compiled code: method entry and exit, backward branches in counted loops, and at transitions between compiled and interpreted code. Notably absent: inside tight loops that the JIT has deemed uncountable, inside deeply inlined method chains (the inlined bodies share the caller’s safepoint structure), and in regions the JIT has aggressively optimised. Interpreted code is always at a safepoint between bytecodes.
3. How JVMTI Sampling Profilers Actually Work
The Java Virtual Machine Tool Interface (JVMTI) is the foundation of most production-grade Java profilers. VisualVM, the JVM sampling mode in many commercial profilers, and JMH’s built-in -prof stack all rely on it. The mechanism is straightforward: a background thread fires at a configured interval (say, every 10 milliseconds), requests a stop-the-world safepoint, walks the stacks of all running threads, records the top-of-stack frames, and resumes execution.
The critical sentence is the second one. To collect stack traces, JVMTI profilers must first bring the JVM to a safepoint. As Baeldung’s async-profiler guide states directly: “JVMTI-based sampling profilers collect stack traces at a safepoint. Therefore, these sampling profilers suffer from the safepoint bias problem.” The consequence is immediate: the profiler cannot observe what a thread is doing at an arbitrary moment. It can only observe what a thread is doing at the moment the thread reaches its next safepoint after the profiler requested the sample.
This introduces a systematic wait. When the profiler requests a sample, each running thread must execute until it reaches its next safepoint poll before it stops for inspection. The time between the sample request and the actual sample is the time it takes to reach the next safepoint — which varies dramatically depending on where in the code the thread was when the request arrived.
4. The Bias Mechanism: Where the Error Enters
Now the bias becomes apparent. Imagine a method that spends 99% of its time in a tight inner loop with no safepoint polls, followed by a single method call at the loop exit that has a safepoint. When a JVMTI profiler fires during the tight loop, the thread does not stop immediately — it continues executing until it exits the loop and hits the method-exit safepoint. The profiler sees the thread at the safepoint — which is the method call, not the loop — and attributes the sample there.
The inner loop, which is where the time is actually spent, generates few or no samples. The method call at the loop exit, which executes once and takes negligible time, accumulates all the samples. Your profiler reports the method call as the hotspot. The loop is invisible. You optimise the wrong thing and are confused when performance doesn’t improve.
Nitsan Wakart’s canonical illustration of the problem: In his foundational blog post, Nitsan Wakart demonstrates a specific case: time is clearly spent in a loop before calling setResult(), but the profiler blames setResult(). There is nothing wrong with setResult() — it simply provides a safepoint that the profiler can observe. As he writes: “The hot code can be anywhere between the indicated safepoint poll and the previous one.” This is the safepoint bias in its purest form: the profiler blames the nearest visible safepoint, not the actual hot code.
5. A Concrete Example of a Misattributed Hotspot
The following pattern is representative of cases that produce systematic misattribution in JVMTI-based profilers.
Java — misattributed hotspot pattern
public class DataProcessor {
public void processChunk(int[] data) {
long sum = 0;
// Tight loop — no safepoint in here (JIT may omit polls in
// short counted loops it has fully unrolled or vectorised).
// This is where ~95% of wall-clock time is spent.
for (int i = 0; i < data.length; i++) {
sum += data[i] * data[i]; // ? actual hot code, invisible to JVMTI
}
// Method call at loop exit — provides a safepoint.
// JVMTI will accumulate samples HERE, not in the loop above.
publishResult(sum); // ? blamed by JVMTI profiler
}
private void publishResult(long sum) {
// Negligible work. The profiler will claim this is your bottleneck.
results.add(sum);
}
}
A JVMTI profiler will show publishResult or results.add() as the hotspot. async-profiler will show the loop inside processChunk. Only one of these is actionable. The important takeaway is that the misattribution is not random noise — it is systematic and consistently biases toward safepoint locations, which means you will encounter it repeatedly in real codebases, especially in data-processing, numerical, or tight-loop code.
6. How JIT Inlining Amplifies the Problem
The safepoint bias problem is significantly worsened by one of the JIT compiler’s most important optimisations: inlining. When HotSpot inlines a method call, it removes the call site entirely and fuses the callee’s bytecode into the caller. This is a major performance win — it eliminates call overhead, enables further optimisations across the combined code, and removes the allocation of stack frames.
But it also removes safepoint opportunities. An inlined method has no method-exit safepoint of its own because, from the JVM’s perspective after JIT compilation, it no longer exists as a separate method. The entire inlined chain shares the calling method’s safepoint structure. Consequently, as Jean-Philippe Bempel documents in his analysis of debug non-safepoints: “Only one line is reported, so no safepoint emitted when methods are inlined.” The profiler cannot see the individual contributions of each inlined method — they are collapsed into the outermost caller’s sample.
In a typical Spring Boot service, this means a deep call chain — controller → service → repository → JDBC — can appear in profiler output as a single large block attributed to the outermost method, with none of the intermediate callee contributions visible. Without knowing which level of the call chain is actually slow, you are optimising blind.
The
-XX:+DebugNonSafepointspartial mitigation: When inlining is active, the JIT normally only records debug information (line numbers, method mappings) at safepoint locations. The flag-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepointsinstructs the JIT to record debug information at all program counter locations, not just safepoints. This dramatically improves the line-number and method resolution of out-of-safepoint profilers like async-profiler and JFR. It is enabled automatically when async-profiler attaches, but must be set explicitly if you use JFR without it.
7. The Hidden Overhead Cost of Safepoint-Based Profiling
The accuracy problem would be serious enough on its own. But JVMTI profiling has a second, compounding problem: the profiling itself causes the JVM to behave differently than it would in production.
Bringing the JVM to a safepoint requires every running thread to reach its next safepoint poll and wait there. The latency of this operation is determined by the thread that takes longest to reach a safepoint — and as Wakart notes, in a heavily threaded application like an application server or a SEDA architecture, this cost is unbounded. Deep stack traces (Spring Framework is notorious here) and many threads compound the problem. A 5ms safepoint every 100ms represents 5% steady-state overhead — and that overhead gets worse as load increases, precisely when you most need accurate profiling data.
Profiler overhead comparison: JVMTI vs async-profiler vs JFR
This overhead is not neutral with respect to profiler accuracy. Higher overhead changes the application’s CPU utilisation, its GC frequency, and — most significantly for profiling — which methods the JIT compiler decides are worth optimising. A method that is borderline hot under normal conditions may fall below the JIT’s compilation threshold under 5% profiling overhead, meaning it runs interpreted during profiling and compiled in production. Your profiler report describes a different program than the one you’re trying to optimise.
8. The Tool Landscape: Ranked by Accuracy
| Tool | Sampling mechanism | Safepoint biased? | Inlining visibility | Overhead | Accuracy verdict |
|---|---|---|---|---|---|
| async-profiler | AsyncGetCallTrace + perf_events | No | Good with DebugNonSafepoints | ~1–2% | Best available |
| JFR (JDK 21+) | Thread-interrupt timer + partial safepoint walk | Partially | Good with DebugNonSafepoints | <1% | Good, improving |
| JFR (JDK 25 CPU-timer) | CPU-time timer, not wall-clock | No | Good | <1% | Best in JDK-native |
| VisualVM (sampling) | JVMTI thread dump | Yes — severely | Poor (collapsed inlines) | 3–8% | Misleading on hot loops |
| JProfiler (JVMTI sampling) | JVMTI | Yes | Poor without async mode | 3–8% | Use async mode instead |
| JProfiler (async mode) | AsyncGetCallTrace | No | Moderate | ~1–2% | Good |
| YourKit (sampling) | JVMTI / async hybrid | Partially | Moderate | Low | Good |
| Instrumentation profilers (general) | Bytecode insertion | No bias | Excellent (line-level) | 82–400× overhead | Distorts JIT severely |
9. async-profiler: How It Avoids the Bias
async-profiler, created by Andrei Pangin, is the tool the JVM performance community has converged on as the best general-purpose solution to safepoint bias. Its approach is fundamentally different from JVMTI: rather than waiting for a global safepoint, it uses the AsyncGetCallTrace API — a HotSpot-specific, experimental API that allows stack trace collection without requiring the thread to be at a safepoint.
Specifically, async-profiler uses Linux’s perf_events mechanism to send a signal to specific threads at configured intervals. The signal handler calls AsyncGetCallTrace on the interrupted thread directly, collecting the stack trace at whatever point in the code the thread happened to be. No safepoint needed. No bias toward method exits or loop edges. The thread is sampled at a truly arbitrary point in its execution, satisfying the statistical requirement that any sampling profiler imposes: “samples must be collected with equal probability across all points in the program.”
The result is immediate and concrete. Where a JVMTI profiler would report publishResult() as the hotspot in the example above, async-profiler correctly reports the loop inside processChunk(). It also monitors non-Java threads — GC and JIT compiler threads — and shows native and kernel frames in stack traces. If your application is spending significant time in kernel read() syscalls or in GC work, async-profiler will show it. JVMTI profilers typically cannot.
shell — running async-profiler on a running Java process
# Attach to a running JVM and profile for 30 seconds
# Outputs a flamegraph — the most useful visual format
./asprof -d 30 -f /tmp/flamegraph.html <PID>
# Profile with DebugNonSafepoints explicitly enabled (for better
# inlined method resolution — auto-enabled when async-profiler attaches)
./asprof -d 30 -f /tmp/flamegraph.html \
--jvmargs "-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints" \
<PID>
# Profile allocation as well as CPU — catch GC pressure
./asprof -d 30 -e alloc -f /tmp/alloc.html <PID>
# Profile lock contention — finds monitor bottlenecks JVMTI misses
./asprof -d 30 -e lock -f /tmp/lock.html <PID>
10. JFR: The Changing Picture in JDK 21–25
Java Flight Recorder has a more nuanced story that has changed significantly across recent JDK releases. Historically, JFR’s method sampling worked by selecting up to five running Java threads at configured intervals and collecting their stack traces — but without requiring a full global safepoint. This is better than JVMTI, but still imperfect.
JDK 25 introduced a new CPU-time profiler in JFR that represents a genuine architectural improvement. Unlike the wall-clock sampler, the new jdk.CPUTimeSample event uses a per-CPU timer mechanism to sample threads based on actual CPU time consumed — meaning a thread that is blocked waiting for I/O is not sampled, and a thread doing intensive computation is sampled proportionally to its actual CPU usage. This is semantically closer to what most engineers want when they ask “where is my CPU time going?”
The new JFR CPU-time sampler also records a biased flag on each sample, explicitly indicating whether that particular sample was safepoint-biased. This transparency is a meaningful improvement over earlier profilers, which had no mechanism to indicate which of their samples were distorted. It allows engineers to filter out biased samples and build a more accurate picture from the unbiased ones.
11. The -XX:+DebugNonSafepoints Flag You Should Always Enable
Regardless of which profiler you use, one JVM flag dramatically improves the quality of profiling data and has almost no production performance cost: -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints.
Without this flag, the JIT compiler only records debug information — the mapping from compiled machine code program counter values back to Java method names and line numbers — at safepoint locations. Between safepoints, if a profiler samples the thread at an arbitrary PC value, it has no way to map that PC to a source location. The sample is either dropped or attributed to the nearest known location, which is typically a safepoint.
With DebugNonSafepoints enabled, the JIT records debug information at all PC locations. async-profiler and JFR can then resolve samples collected at arbitrary execution points to precise method names and line numbers. In practice, this transforms a flame graph from one with wide, poorly-attributed method blocks into one with narrow, precise attribution — and that precision is what makes the difference between finding the actual bottleneck and optimising the wrong method.
Add this to every service you profile
-XX: +UnlockDiagnosticVMOptions -XX:+DebugNonSafepointshas negligible performance overhead in production — the additional debug info is stored in the JIT’s code cache and only consulted when a profiler is attached. async-profiler automatically enables it when it attaches, but JFR does not. If you use JFR for continuous profiling in production (a practice with legitimate overhead/accuracy trade-offs), enabling this flag explicitly gives you meaningfully better data.
12. Building the Right Mental Model
The goal of this article is not to make you distrust profilers entirely — it is to give you a mental model that lets you interpret profiler output with appropriate scepticism. Here is a practical set of rules that follows from everything above.
| Situation | What to do | Why |
|---|---|---|
| Profiler shows a utility method or framework call as the hotspot | Treat as a signal, not a diagnosis. Check what called it and what happens before it in the call tree. | Utility methods and framework entry points frequently sit adjacent to safepoints. The real work may be in their callers. |
| Optimising the reported hotspot produced no improvement | Switch to async-profiler. The JVMTI result was likely safepoint-biased. | Null result from an optimisation is the strongest signal that the hotspot report was wrong. |
| Profiling a tight numerical or data-processing loop | Use async-profiler exclusively. Never trust JVMTI output for this pattern. | Tight loops with no safepoint polls are exactly the case where JVMTI bias is most severe. |
| Profiling in production or staging under real load | Use async-profiler or JFR. Enable DebugNonSafepoints on both. | JVMTI overhead (3–8%) at production load is both measurable and distorting. async-profiler overhead is ~1–2%. |
| Flame graph shows a wide flat top in a single method | Check whether inlining has collapsed child methods into the parent. | Wide method attribution in a flame graph often indicates several inlined callees contributing to a shared attribution block. |
| GC pauses are high but CPU profile shows nothing obvious | Run async-profiler in allocation mode (-e alloc). | CPU sampling does not show allocation pressure. async-profiler’s alloc mode records the stack at object allocation sites, directly identifying where GC load originates. |
The Honest Assessment from the Research Community: The 2024 MPLR paper from Stefan Marr’s group concludes bluntly: instrumentation-based profilers interact badly with inlining and other standard JIT optimisations, leading to profiles that are not representative of production performance. Sampling profilers have safepoint bias and cannot correctly attribute observed run time to the correct methods in the presence of inlining. Their prototype — Bubo, which inserts probes late in the compilation pipeline to avoid interfering with optimisations — gets good attribution but is not yet production-ready. The honest state of play in 2026 is that async-profiler plus DebugNonSafepoints is the best available option, and it is considerably less wrong than the alternatives — not perfect.
13. What We Have Learned
JVM profiling has a systematic accuracy problem that most Java developers have never been explicitly told about. Here is the full picture distilled:
- JVMTI-based profilers (VisualVM, JMH
-prof stack, JProfiler in JVMTI mode) require a global safepoint to collect stack traces. Because safepoints are non-uniformly distributed in JIT-compiled code, these profilers systematically over-sample code near safepoints — method exits and counted loop edges — and under-sample everything in between. - Safepoint bias means the profiler blames the nearest safepoint to the hot code, not the hot code itself. In tight loops, the blamed location can be completely unrelated to where the time is spent.
- JIT inlining amplifies the problem by collapsing multiple methods into a single safepoint structure. Inlined methods have no individual safepoint and appear as a single block in the parent method.
- JVMTI profiling overhead (3–8% at normal sampling rates) is not neutral — it changes JIT compilation decisions and makes the profiled application behave differently from the production one.
- async-profiler uses
AsyncGetCallTrace+perf_eventsto sample at arbitrary execution points without requiring a safepoint. It is the current best practice for CPU, allocation, and lock profiling. Overhead is ~1–2%. - JFR is improving: JDK 21+ partially avoids safepoint bias; JDK 25’s CPU-time sampler avoids it entirely and marks remaining biased samples explicitly.
-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepointsshould be added to every service you profile. It enables the JIT to record debug information at non-safepoint locations, allowing async-profiler and JFR to resolve samples to precise methods and line numbers instead of the nearest safepoint.- The honest answer from academic research: no production-ready profiler for the JVM is fully accurate today. async-profiler is considerably less wrong than the alternatives — which is the correct goal to optimise for.
