On Linux, perf_event_open starts and stops counters around any code region. The catch is that starting and stopping has its own cost. With perf stat this overhead disappears into the noise of a full program; in-code measurements have no such cushion, and around a tight loop the bracket itself can be larger than what is being measured.

Two paths exist for interacting with hardware performance counters from within code. The first goes through ioctl calls, asking the kernel to enable and disable counters. This is the standard path, and what most (in-code) profiling tools default to, including perf stat itself. The second reads counter values directly from user space using the rdpmc instruction, bypassing the kernel entirely. Setting aside that it is x86-only, that sounds like a clear win.

However, the perf_event_open man page is careful not to promise that:

Using rdpmc is not necessarily faster than other methods for reading event values.

The only way to know how much each actually costs, and at what measured region size each becomes trustworthy, is to measure it.

What the Counter Actually Sees

Before any numbers, a quick note on what “overhead of start/stop” actually means: the counter does not see the enable or disable operations themselves. It sees the asymmetric window between them.

On the ioctl path, the performance counters start counting at some point inside the syscall handler. Everything that follows — the rest of the handler, the syscall return path (sysret/iret, the KPTI page-table swap, retpoline / IBRS tail, restoring user state) — executes with the counter already running. Then the measured region itself runs. Then ioctl is invoked to disable counting: the full syscall entry sequence (KPTI swap, stack switch, dispatch into the handler) runs with the counter still on, until the disable point inside the handler is reached. Consequently, what the counter records, beyond the body, is the tail of enable, the head of disable, and a thin slice of user-space scaffolding inside the bracket.

On the rdpmc path, the bracket reduces to a few user-space reads. No kernel round-trip, no privilege transition; only the read instructions themselves and whatever harness surrounds them. This is the entire reason the two paths differ in cost.

Methodology

We ran a random-access benchmark across working set sizes from 1 KB to 32 MB, using perf-cpp to drive both counter paths. Each iteration accesses one cache line at a shuffled index: exactly 6 instructions, 2 memory loads, and 1 branch per iteration, regardless of working set size. Those fixed, known baselines are the point: any excess in the measured counts comes from measurement overhead, not the workload.

We picked three machines to separate vendor effects from micro-architecture ones: AMD Zen 4, Intel Skylake (desktop i7), and Intel Sapphire Rapids (Xeon server). The two Intel chips are eight years apart — if the ioctl overhead were purely a micro-architectural cost, they should diverge. perf-cpp exposes the ioctl path through its EventCounter (docs) interface and the rdpmc path through LiveEventCounter (docs).

We made the setup as constant as possible across the three machines: each benchmark pinned to a single core (using taskset), turbo disabled, and kernel.perf_event_paranoid = -1; kernels were 6.8.0-101 (Skylake), 6.8.0-106 (Sapphire Rapids), and 6.17.0-23 (Zen 4).

One note: Spectre / KPTI mitigations were left at distribution defaults. We did not toggle them, so any claim about them below is based on the cross-platform pattern, not on direct A/B tests.

And one more: each platform was tested on one machine. The instruction-count constants should be deterministic (they come from kernel code paths), but cycle numbers can shift with silicon variation, BIOS settings, and kernel version.

We measured four counters: instructions, cycles, L1-dcache-loads, and branches. Instructions, loads, and branches have constant per-iteration baselines (6, 2, and 1) and carry the main analysis. Branches reproduce the instruction thresholds exactly, so we leave them out of the per-platform walkthrough. Cycles have no fixed baseline (cache-miss latency grows with the working set), but the same qualitative pattern shows up there too, and we use them below as a cross-check on the kernel-path argument.

Code and plots. Benchmark source, all plots (including cycles and branches), and supporting files live in the companion repo: see main.cpp for the random-access benchmark and plots/ for the figures.

Anatomy on AMD Zen 4

We start with Zen 4 in detail; the shape we see here repeats on the Intel chips with different magnitudes.

Instructions. At 1 KB of working set (16 iterations or accessed cache lines), ioctl reports 193 instructions per iteration; the true number is 6 (the red dashed line). That works out to roughly 3,000 extra instructions per start/stop bracket, dominating at small working sets. Reminder: this is the overhead of the tail of enable and the head of disable.

rdpmc adds only about 400 instructions per bracket at the same working set size. It settles within 5% of the true baseline at 128 KB (2,048 iterations, ~12,000 total instructions executed). ioctl does not reach that same threshold until 16,384 iterations — 1 MB of randomly accessed data, or ~98,000 total instructions.

L1-dcache-loads. The same pattern holds for memory loads, with a true baseline of 2 per iteration. At 1 KB, ioctl reports 70 loads per iteration. Both turnaround points match the instruction thresholds: rdpmc at 128 KB (~4K total loads), ioctl at 1 MB (~33K total loads).

So on Zen 4 the ioctl bracket costs ~7.5× more than rdpmc per call (3,000 vs. 400 instructions), which translates directly into needing ~8× more workload to dilute that overhead to ≤5%.

Intel: Skylake and Sapphire Rapids

Both Intel chips share the same rdpmc behaviour as Zen 4, but the ioctl overhead is dramatically higher.

On Skylake, ioctl reports 705 instructions per iteration at 16 iterations, roughly 4× the Zen 4 number. The overhead works out to about 11,200 extra instructions per start/stop bracket. rdpmc adds the same ~400 instructions as on Zen 4 and stabilizes at exactly the same threshold: ~12,000 total instructions (128 KB accessed data). ioctl, by contrast, doesn’t settle within 5% until ~393K total instructions (4 MB).

Sapphire Rapids produces nearly the same numbers. At 1 KB, ioctl reports 679 instructions per iteration, ~10,800 instructions of overhead per bracket. The instruction threshold in our random-access benchmark is identical to Skylake: 4 MB for ioctl, 128 KB for rdpmc.

Two Intel chips (eight years apart) show essentially the same ioctl instruction overhead: very likely not a coincidence. Cycles, in contrast, tell a different story: the same ioctl bracket consumes ~28,100 cycles on Skylake but only ~8,525 cycles on Sapphire Rapids.

It is the same kernel work, but different silicon is executing it. Hardware-level optimizations such as deeper buffers and better branch predictors on Sapphire Rapids let the same ~11K-instruction kernel path retire in roughly a third of the cycles. This is exactly the signature of kernel-path cost: the count of overhead instructions is constant across generations (determined by the syscall entry/exit code, not by the hardware substrate), while the cycle count varies with the hardware executing those instructions. The kernel-path account fits both observations; a micro-architectural one fits neither.

The most plausible source of those ~11K instructions is the syscall entry/exit path under Spectre / KPTI mitigations — KPTI page-table swaps on enter and exit, retpoline / IBRS tails on indirect branches. We frame this as consistent with the data rather than proven, since we did not run with mitigations=off.

The L1-dcache-loads results follow the instruction picture, with one minor twist. At 1 KB, ioctl reports 161 loads/iter on Skylake and 157 on Sapphire Rapids against a true value of 2. rdpmc again converges at 128 KB (~4K total loads) on both. For ioctl, Skylake mirrors its instruction threshold at 4 MB (~131K total loads), but Sapphire Rapids converges earlier, at 2 MB (~66K loads). The kernel path is instruction-heavy, not load-heavy, so Sapphire Rapids’ load count crosses the ≤5% line slightly sooner than its instruction count does.

What the Bracket Actually Costs

The thresholds in the next section are all derived from a small set of underlying constants: the counters added by a single enable-to-disable window, on each platform, in each path. Using the small-WSS rows (where the per-iteration overhead is large enough to read off cleanly), the per-bracket cost is:

The Intel ioctl bracket is ~1.5–4× larger than AMD’s in every counter dimension; instructions, loads, and branches all scale together, meaning the kernel is doing more work on the syscall path on Intel. rdpmc, by contrast, is small and roughly constant across platforms (~400 instructions, ~70 branches, ~120–170 loads). Zen 4’s read sequence reads slightly more data than Intel’s, but the differences are small enough to be a footnote.

Cycles are the exception. The two Intel generations show identical kernel-path instruction counts but a factor-of-three cycle gap, exactly the pattern one would expect from kernel work running on different silicon.

Side by Side: Threshold Tables

The thresholds below are the workload size at which the per-iteration overhead from the constants above dilutes to ≤5% and ≤1% of the true baseline.

Instructions (baseline: 6/iteration, ≤5% error = ≤6.30, ≤1% error = ≤6.06):

L1-dcache-loads (baseline: 2/iteration, ≤5% error = ≤2.10, ≤1% error = ≤2.02):

Two conclusions:

rdpmc’s thresholds are the same on every platform. All three machines converge at 128 KB (≤5%) and 512 KB (≤1%) for both instructions and loads. A clean 4× gap between the two thresholds, identical across vendors and generations. The per-bracket cost varies by a small amount between vendors, but not enough to move the threshold.

ioctl requires ~4× more work on Intel than AMD to reach ≤5%. Not because Intel’s hardware is slower, but because its kernel path is longer, confirmed by the per-bracket instruction count (~11K vs ~3K) and backed by the Skylake-vs-Sapphire-Rapids cycle gap, which puts the cost in the kernel rather than in the silicon. Spectre / KPTI mitigations on the syscall path are the most plausible explanation; we did not isolate them experimentally.

When to Use Which

Yes, rdpmc is cheaper. But rdpmc is not always usable.

How many events do you need at once? rdpmc reads physical hardware counters directly, so it cannot multiplex. If you request more events than the hardware exposes simultaneously (typically 4–8 general-purpose counters), ioctl-based measurements rotate events in and out and scale the results; rdpmc has no such fallback. For a handful of events this is no constraint, but if you need many events at once, you’ll have to fall back to ioctl.

How small is your measured region? Below ~12K instructions / ~4K loads, neither path is reliable on any platform; restructure the benchmark. Between those thresholds and ~98K instructions, rdpmc is trustworthy on every platform; ioctl is trustworthy only on AMD. Above ~393K instructions, ioctl becomes acceptable on Intel too, except for ≤1% accuracy, which Intel ioctl never quite reaches in our data.

Bottom line: prefer rdpmc when your event count fits in the hardware slots and your region is small; fall back to ioctl when you need multiplexing or your region is large enough to amortize the kernel cost.

Practical Takeaways

The thresholds aren’t about the number of iterations. They’re about the total volume of work in the measured region.

If the workload falls below the thresholds, there are two options. The first is to restructure your benchmark so the measured region contains enough work — running the core loop multiple times and normalizing the result afterward is a common approach, and most perf APIs support passing the operation count directly so the division happens automatically. The second is to prefer rdpmc-based reads where your tooling supports it: perf-cpp’s LiveEventCounter interface exposes this directly, and the numbers above translate to an 8× reduction in required workload size on AMD and a 32× reduction on Intel, if you stay within the hardware counter limits.

Linux Perf, Intel VTune, and AMD μProf are helpful tools for understanding how applications use system resources. However, as these tools are typically designed as external applications, they profile the entire program, making it difficult to focus on specific code segments like particular functions. This limitation is particularly challenging when analyzing micro-benchmarks, where the measured code may represent only a fraction of the overall runtime, or distinguishing between different phases of an application’s execution.

Counting Hardware Events

At their core, these tools leverage Performance Monitoring Units (PMUs)–specialized components designed to track hardware events like cache misses and branch mispredictions. Although these tools are far more powerful, this discussion will focus on the essentials of hardware event counting.

Scenario: Random Access Pattern

Consider a random access micro-benchmark designed to access a set of cache lines in a random sequence—a scenario that typically baffles the data prefetcher (see the full source code). The benchmark employs two distinct arrays: one holding the data and another containing indices that establish the random access pattern. After initializing these arrays, we execute the micro-benchmark by sequentially scanning through the indices array and access data from the data array, a method that generally leads to approximately one cache miss per access within the contiguous data array.

Perf Stat

To observe the underlying hardware dynamics, we utilize the perf stat command, which quantifies low-level hardware events such as L1 data cache accesses and references during the execution of the micro-benchmark:

perf stat -e instructions,cycles,L1-dcache-loads,L1-dcache-load-misses -- ./random-access-bench --size 16777216

After running, perf stat displays the results on the command line, in combination with metrics such as instructions per cycle:

Performance counter stats for './random-access-bench --size 16777216':

    3,697,089,032      instructions            #    0.63  insn per cycle            
    5,879,736,227      cycles                                                                
    1,186,826,319      L1-dcache-loads                                                       
      103,262,784      L1-dcache-load-misses   #    8.70% of all L1-dcache accesses 

      1.202831289 seconds time elapsed

      0.799309000 seconds user
      0.403155000 seconds sys

Zooming into details, the results reveal 103,262,784 L1d misses for 16,777,216 items, which translates to \(\frac{103,262,784}{16,777,216} \approx 6\) misses per item. This number significantly surpasses the anticipated single cache miss per item. The source of this discrepancy lies in the comprehensive scope of the perf stat command, which records events throughout the entire runtime of the benchmark. This includes the initialization stage of the benchmark where both the data and pattern arrays are allocated and filled. Ideally, however, profiling should be confined to the specific segment of the code that interacts directly with the data array to achieve more accurate metrics.

One effective strategy for more control over profiling is to start and stop hardware counters at specific code segments using file descriptors. This technique is well-documented in the perf stat man page. Pramod Kumbhar provides a practical guide to implementing this technique on his blog, though some might find the approach somewhat cumbersome to implement.

Controlling Performance Counters from C++ Applications

Another strategy for achieving refined control over PMUs is to leverage the perf subsystem directly from C and C++ applications through the perf_event_open system call. Given the complexity of this interface, various libraries have been developed to simplify interaction by embedding the perf_event_open system call into their framework. Notable examples include PAPI, PerfEvent, and perf-cpp, each designed to offer a more accessible gateway to these advanced functionalities.

This article will specifically explore perf-cpp and demonstrate practical examples of how to activate and deactivate hardware performance counters for targeted code segments. The perf::EventCounter class in perf-cpp allows users to define which events to measure and provides start() and stop() methods to manage the counters. Below is a code snippet that sets up the EventCounter and focuses the measurement on the desired code segment:

#include <perfcpp/event_counter.h>

/// Initialize the hardware event counter
auto counters = perf::CounterDefinition{};
auto event_counter = perf::EventCounter{ counters };

/// Specify hardware events to count
event_counter.add({"instructions", "cycles", "cache-references", "cache-misses"});

/// Setup benchmark here (this will not be measured)
struct alignas(64U) cache_line { std::uint64_t value; };
auto data = std::vector<cache_line>{};
auto indices = std::vector<std::uint64_t>{};
/// Fill both vectors here...

auto sum = 0ULL;

/// Run the workload and count hardware events
event_counter.start();
for (const auto index : indices) {
    sum += data[index]; // <-- critical memory access
}
asm volatile("" : : "r,m"(value) : "memory"); // Ensure the compiler will not optimize sum away
event_counter.stop();

Once the EventCounter is initiated and the events of interest are added, we set up the benchmark by initializing the data and pattern arrays. Enclosing the workload we wish to measure with start() and stop() calls enables precise monitoring of that particular code segment. Upon stopping the counter, the EventCounter can be queried to obtain the measured events:

const auto result = event_counter.result();

/// Print the performance counters.
for (const auto [name, value] : result)
{
    std::cout << value << " " << name << " (" << value / 16777216 << " per access)" << std::endl;
}

The output reflects only the activity during the benchmark, effectively excluding the initial setup phase where data is allocated, and patterns are established:

102,284,667 instructions            (6.09664 per access)
992,091,716 cycles                  (59.1333 per access)
 34,227,532 L1-dcache-loads         (2.04012 per access)
 18,944,008 L1-dcache-load-misses   (1.12915 per access)

The results obtained are markedly more explicable than those we got from the perf stat command. We observe two L1d cache references per access: one for the randomly accessed cache line and another for the index of the pattern array. Additionally, there are approximately 1.3 cache misses—one for each data cache line and 0.125 for the access index, as eight indices fit into a single cache line of the pattern array.

Hardware-specific Events

While basic performance metrics such as instructions, cycles, and cache misses shed light on the interplay of hardware and software, modern CPUs offer a far broader spectrum of events to monitor. However, it’s important to note that many of these events are specific to the underlying hardware substrate. The perf subsystem standardizes only a select group of events universally supported across different processors (see a detailed list). To discover the full range of events available on specific CPUs, one can utilize the perf list command. Additionally, Intel provides an extensive catalog of events for various architectures on their perfmon website.

In order to use hardware-specific counters within applications, the readable event names need to be translated into event codes. To that end, Libpfm4 provides a valuable tool that translates event names (from perf list) into codes.

Let us consider the event CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD on the AMD Zen4 architecture as an example. The event quantifies the CPU cycles stalled due to pending memory requests, which is particularly insightful for assessing the effects of cache misses on modern systems. Intel offers analogous events, such as CYCLE_ACTIVITY.STALLS_MEM_ANY on the Cascade Lake architecture, and both EXE_ACTIVITY.BOUND_ON_LOADS and EXE_ACTIVITY.BOUND_ON_STORES on the Sapphire Rapids architecture.

After downloading and compiling Libpfm4, developers can fetch the code for a specific event as shown below:

./examples/check_events CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD

Requested Event: CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD
Actual    Event: amd64_fam19h_zen4::CYCLES_NO_RETIRE:NOT_COMPLETE_MISSING_LOAD:k=1:u=1:e=0:i=0:c=0:h=0:g=0
PMU            : AMD64 Fam19h Zen4
IDX            : 1077936192
Codes          : 0x53a2d6

Incorporating hardware-specific events into an application with perf-cpp would look something like this:

#include <perfcpp/event_counter.h>

/// Initialize the hardware event counter
auto counters = perf::CounterDefinition{};
counters.add("CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD", 0x53a2d6); // <-- Event code from Libpfm4 output
auto event_counter = perf::EventCounter{ counters };

/// Specify hardware events to count
event_counter.add({"cycles", "CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD"});

/// Setup and execute the benchmark as demonstrated above...

This precise tracking reveals that approximately 57 of 59 CPU cycles are spent waiting for memory loads to complete–a finding consistent with the inability of the hardware to predict the benchmark’s random access pattern, relying instead on inherent memory latency:

992,091,716 cycles                                      (59.1333 per access)
967,301,682 CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD  (57.6557 per access)

However, thanks to sophisticated out-of-order execution, the hardware effectively masks much of this latency, which on the specific machine to execute the benchmark is around 700 cycles.

Summary

Profiling tools play a crucial role in identifying bottlenecks and aiding developers in optimizing their code. Yet, the broad granularity often means that key code segments tracked with perf stat can be obscured by extraneous data. Libraries like PAPI, PerfEvent, and perf-cpp offer a solution by allowing direct control over hardware performance counters from within the application itself. By leveraging the perf subsystem (more precisely the perf_event_open system call), these tools enable precise measurements of only the code segments that are truly relevant.