Measure CPU, memory, and IRQ behavior.

Jitter Tools 1.3.0 provides three standalone Linux command-line tools. CPUJitter observes execution gaps on one CPU. MEMjitter measures declared memory operations. IRQjitter observes kernel IRQ and softirq events through tracefs. Use them separately to answer a specific measurement question.

./cpujitter --version
./memjitter --version
./irqjitter --version

The examples below run from an unpacked binary distribution. After a source build, the same executables are in ./bin/. If installed on PATH, omit the ./ prefix. CPU numbers are examples, not a production placement recommendation. Replace them with allowed, online Linux CPU IDs on your host.

grep -E "Cpus_allowed_list|Mems_allowed_list" /proc/self/status
cat /sys/devices/system/cpu/online

./cpujitter -c 4 --duration 10

./memjitter -c 4 --duration 10

sudo ./irqjitter -c 1 --duration 10

Run these examples one at a time. CPU and memory tests are active workloads. IRQjitter is passive in the sense that it does not generate test traffic, but tracing and collection still have overhead. CPU/MEM do not require root, disabled SMT, CPU isolation, an exclusive CPU, or HFTKernel. IRQ collection requires access to an existing tracefs mount and permission to create its own instance; sudo is one way to provide those permissions.

Source reference: S1, S2, S3, S7.

The supplied 1.3.0 archive is source-only. It does not contain ready-to-run ELF executables. Its recorded qualification does not establish a completed Rust build or a live IRQ run. The supported source entry point is test.sh; it builds and tests before publishing binaries.

cd jitter-tools-1.3.0
./test.sh

Cargo runs offline with a locked dependency set and no registry dependencies. Python test dependencies may be provisioned into the project-local .test-env/. Use --offline when the environment is already prepared and package-index downloads must be disabled.

./test.sh --offline --acceptance

After all required gates pass, the workflow publishes bin/cpujitter, bin/memjitter, bin/irqjitter, bin/build-ok.json, and dist/jitter-tools-1.3.0-linux-<architecture>-static.tgz. The release bundle includes BUILD.json, ELF.json, checksums, notices and documentation. Do not treat a partially completed source build as a qualified static distribution.

./test.sh
./cpujitter --build-info
./memjitter --build-info
./irqjitter --build-info

The binary-distribution test.sh requires sh, readelf, sha256sum, and grep. It is not the source build runner. GNU-static executables do not need Rust, Python, an ELF interpreter, or neighboring shared libraries at runtime. They still need an architecture-compatible Linux kernel, the required system calls and /proc and /sys; IRQjitter additionally needs tracefs. ELF checks are packaging evidence, not a guarantee of measurement accuracy or support on every kernel version.

Source reference: S1, S2.

CPU/MEM also accept --cpu and --interval. IRQjitter 1.3.0 uses only -c and -n for these controls. All three accept attached short values such as -c4 -n0.1. Use plain decimal seconds, for example 0.1, not 100ms, 1e-1, or .1. Repeated options are rejected.

./cpujitter -c4 -n0.1 --duration 5

./cpujitter -c 4 --raw --json --samples 1000 > cpu-raw.jsonl 2> cpu-raw.stderr.log

Data goes to stdout and diagnostics to stderr. Redirecting stdout does not select JSON or raw mode. Use a finite duration or Ctrl+C to stop. A handled SIGINT/SIGTERM emits final records when the output channel remains usable and cleanup succeeds. A blocked system call, SIGKILL, failed host, or closed pipe prevents any guarantee of an immediate stop or complete final output.

Source reference: S2, S3, S9.

The metric cpu_iteration_elapsed_ns is the calibrated elapsed time between adjacent hardware-counter reads inside a capture batch. It includes the loop branch, timestamp storage and counter-read overhead. It is not the CPU core cycle count, a device interrupt-delivery measurement, or a direct classification of the cause of a pause.

ordered is the default counter-read mode; relaxed is an explicit alternative with a distinct context. Calibration uses seven segments of at least 50 ms against CLOCK_MONOTONIC_RAW, with a fixed scale for the run. No baseline or counter cost is subtracted. On x86, declared invariant-TSC support and usable counter access are required; unsupported counters are rejected rather than replaced silently.

./cpujitter -c 4 -n 1 --duration 30
./cpujitter -c 4 -n 1 --duration 30 --counter-read relaxed

./cpujitter -c 4 -n 1 --duration 30 \
  --threshold cpu_iteration_elapsed_ns=5000

./cpujitter -c 4 --duration 30 --json > cpu.jsonl 2> cpu.stderr.log

One batch has a seed timestamp and completed observations. Capture is separated from clock anchors, validation, aggregation and output. A cooperative counter budget checks for roughly 20 ms of capture work every 256 records; it does not clip a large observed delay. ACTIVE% describes the covered capture portion, not total CPU utilization. Pauses occurring outside capture are unobserved for this metric.

Nanoseconds are computed with checked integer arithmetic and rounding to nearest: (delta_ticks * 1000000000 + hz / 2) / hz. Event timestamps for window assignment use the bracketed counter-to-MONOTONIC mapping. The value and its mapped start/end interval serve different purposes and need not be numerically identical. Unknown physical accuracy remains null in the stream.

Source reference: S2, S3, S9.

Resident reads and read/modify/write

The default is resident, read, sequential, a 16777216-byte mapping (16 MiB), a system-page-sized stride, offset 0 and seed 1. Preparation touches each base page, warms the access plan and checks initial residency with mincore. Without an explicit lock, that snapshot is not a promise of continued residency.

./memjitter -c 4 --bytes 16777216 --duration 30
./memjitter -c 4 --operation rmw --pattern random --seed 1 --duration 30

read times one volatile byte load. rmw times a byte load, wrapping increment and store; it is not atomic and does not measure globally visible store completion. The timing bracket includes timestamp/fence overhead. A result is not pure DRAM latency.

Dependent pointer chase

./memjitter -c 4 --operation chase --bytes 16777216 \
  --seed 1 --duration 30

--operation chase selects pointer-chase automatically unless a pattern is explicit. An explicit pattern must also be pointer-chase. Other resident patterns are sequential, random with replacement, and permutation. Address selection and bookkeeping are outside each measured operation. A page-strided 16 MiB mapping on a 4096-byte-page host has 4096 sampled positions, not 16 MiB of payload read on each pass.

First-touch writes

./memjitter -c 4 --mode first-touch --reset remap \
  --bytes 16777216 --duration 30

./memjitter -c 4 --mode first-touch --reset madvise \
  --pattern permutation --seed 1 --duration 30

First-touch permits sequential or permutation, without replacement inside an epoch. write is the only operation and need not be specified. After every successful remap/discard, mincore must confirm no resident pages before a new epoch begins. Reset work is excluded service time. A first byte write is not necessarily a separate page fault or physical-page allocation, especially when THP backs multiple offsets. Remap and madvise are different experiments.

Allocation API calls

./memjitter -c 4 --mode alloc --bytes 65536 --duration 30
./memjitter -c 4 --mode alloc --operation mmap --samples 65 --json

Alloc creates at most one owned mapping at a time and unmaps it before the next call. Only successful mmap intervals enter the histogram. mmap establishes a virtual mapping, not physical population of every page. The C ABI/libc wrapper and timestamp bracket remain part of the value. The legacy alloc-cycle, malloc/calloc latency and automatic memory-pressure generation are not implemented by this mode.

Layout and mode options

./memjitter -c 4 --mode first-touch --duration 30 \
  --threshold mem_first_touch_ns=20000

Alloc rejects pattern, stride, offset, seed and reset options, even when their values resemble defaults. No memory mode accepts CPUJitter's --counter-read. Keep the same operation, pattern, bytes, stride, offset, seed, reset and output mode when comparing results.

Source reference: S2, S4, S5.

The default is --lock none, --thp system, inherited NUMA policy, and no explicit HugeTLB request. The tool does not reconfigure the system, enlarge lock limits, change a huge-page pool, or retry under a weaker policy. An explicit unsupported request fails with its reason.

./memjitter -c 4 --bytes 65536 --lock resident --duration 10

./memjitter -c 4 --mode first-touch --lock onfault \
  --reset remap --bytes 65536 --duration 10

./memjitter -c 4 --thp never --duration 10
./memjitter -c 4 --thp prefer --duration 10

./memjitter -c 4 --numa-node 0 --duration 10

./memjitter -c 4 --bytes 2097152 \
  --hugetlb-bytes 2097152 --duration 10

Locking prevents swapping of the mapping while held; it does not put data permanently in CPU cache. An accepted THP advice call does not prove huge-page backing. NUMA policy readback is distinct from observed physical page placement. NUMA IDs are 0..4095 and must be allowed by the host.

An explicit HugeTLB size must be a power of two greater than the OS base page, at most 1 GiB, and divide the mapping size. It is incompatible with explicit locking or non-system THP advice. Resident preparation requires successful MADV_POPULATE_WRITE for that range. The existing pool, quotas, permissions and required kernel operation must support the request; there is no fallback to ordinary pages.

Requested policy, successful operations and observed mapping state are recorded separately. Initial observations appear in meta; final mapping observations are written to stderr before unmap. Unavailable or unattributable information stays null. Proc-file snapshots and policy checks add service work, so compare runs with matching policies.

Source reference: S2, S6.

IRQjitter uses one userspace reader thread and its own tracefs instance. Required IRQ tracepoints are irq_handler_entry, irq_handler_exit, softirq_raise, softirq_entry, and softirq_exit. The instance must support the mono trace clock. It does not mount tracefs, chmod it, change IRQ/RSS affinity, stop irqbalance, or enable an alternative backend.

grep " tracefs " /proc/mounts

sudo ./irqjitter -c 1 --tracefs /sys/kernel/tracing --duration 15

Collector CPU versus observed CPUs

sudo ./irqjitter -c 1 --target-cpus 4-7 --duration 30

Without --target-cpus, supported events on all online CPUs are observed, including the collector CPU. A CPU list such as 2,4-7 restricts collection, not just presentation. IRQ activity that moves outside the selected set is outside coverage. CPU hotplug is monitored within the initial possible-CPU envelope and changes are reported with coverage information.

Three distinct timing metrics

An IRQ source is identified by metric, actual CPU, source number and name. Counts describe completed handler observations, not physical interrupts or packets. Identical source number/name pairs do not prove a persistent device identity across reconfiguration. A zero elapsed value may reflect the resolution of textual trace timestamps, commonly microsecond-level, rather than zero physical time.

Overview, details and raw events

sudo ./irqjitter -c 1 -n 1 --duration 30
sudo ./irqjitter -c 1 -n 1 --details --duration 30
sudo ./irqjitter -c 1 -n 1 --json --duration 30 > irq.jsonl 2> irq.stderr.log

sudo ./irqjitter -c 1 --target-cpus 4 --raw --json \
  --duration 2 > irq-raw.jsonl 2> irq-raw.stderr.log

--details is valid only for compact windows and cannot be combined with --json or --raw. JSON already preserves per-CPU/source window records. There is no timerlat mode or eBPF backend in this release. The real TIMER softirq is still observable.

Optional NAPI work/budget observations

sudo ./irqjitter -c 1 --napi --duration 15
sudo ./irqjitter -c 1 --napi --details --duration 15

--napi requires a supported napi:napi_poll tracepoint. The metric is napi_poll_work, unit work, not nanoseconds. It reports the return work value and budget. at_budget requires a positive budget with work equal to budget; over_budget requires work greater than a positive budget. Neither implies a proven packet backlog by itself. NAPI duration, IRQ-to-NAPI latency and an inferred hardware queue ID are not provided.

Window and resource controls

sudo ./irqjitter -c 1 --target-cpus 4-7 -n 1 \
  --threshold 10000 --duration 30

Late completed samples are reported as late and are not moved into a later window or used to rewrite old output. Slow output can delay trace reading and lead to kernel-buffer loss. The collector checks controls and buffer statistics, but neither empty reads nor lack of a loss notice prove complete coverage. Trace instrumentation runs on event CPUs too, not only on the collector CPU.

Source reference: S2, S7, S8.

CPU/MEM compact columns

TIME(s) CPU COUNT MIN AVG P99<= P99.9<= MAX ACTIVE% LOSS QUALITY PART

A timing threshold adds a suffix such as >5000=N, where N is the count strictly above the limit. An empty population has COUNT=0 and NA statistics, not zero latency. IRQ --details adds METRIC, SOURCE, UNIT and the NAPI-specific BUDGET_MAX, AT_BUDGET and OVER_BUDGET fields; inapplicable values are NA.

IRQ overview columns

TIME(s) IRQ_N IRQ_P99<= IRQ_MAX SOFT_N SOFT_P99<= SOFT_MAX WAIT_N WAIT_P99<= WAIT_MAX PAIR_GAPS LATE BUFFER OTHER PART

IRQ, SOFT and WAIT are separate populations. Each combines compatible source histograms across the observed CPUs for that metric and window; percentiles are recomputed from pooled bin counts, not averaged across source percentiles.

For example, softirq_entry_without_observed_raise increments pairing diagnostics. It does not establish a trace-ring overflow, and a valid subsequent entry/exit duration can still be retained. Buffer statistics and textual loss markers may overlap; do not add them to a fabricated total loss count.

End-of-run summaries

Compact output prints # SUMMARY, # TOTAL rows and an end status after the last window, including the final partial window. CPU/MEM and IRQ details summarize each series; IRQ overview produces separate pooled timing totals and optional NAPI work totals. The IRQ overview summary includes MAX_CPU and MAX_SOURCE, so the largest observed event retains its source.

Totals use exact counts and sums from fully reported windows, weighted mean, extrema and merged histogram bins. They are not averages of window averages or percentiles. scope=reported_windows excludes unreported and late samples. A source associated with a maximum is an observation identity, not a proven root cause.

JSON Lines for automation

Use --json to retain metadata, per-series windows, quality reasons, loss/gap records and the terminal end record. Use --raw --json for individual samples. Compact summary text is not appended to JSON. Wide integer values such as nanoseconds, counts and sums are decimal strings; parse them as integers, not floating-point numbers. A null value is unknown, not zero.

jq 'select(.type == "meta")' cpu.jsonl
jq 'select(.type == "end") | {execution, exit_code, quality, cleanup}' cpu.jsonl
jq 'select(.type == "window") | {series, stats, coverage}' cpu.jsonl

The end record separates execution, quality and cleanup.confirmed. observations_aggregated is important for window mode; a zero observations_emitted does not mean no data was collected if values were aggregated into windows. Retain the complete file and stderr, rather than only selected window lines.

Source reference: S8, S9.

These are the tool's documented codes, not a promise about every shell, timeout wrapper or fatal signal. Preserve a nonzero status; avoid appending || true to make a result appear successful. A pipe closed by head can prevent final records. --samples or --duration is the supported way to bound a capture.

out="jitter-run-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir "$out"
./cpujitter --build-info > "$out/build-info.json"

if ./cpujitter -c 4 -n 1 --duration 30 --json \
    > "$out/cpu.jsonl" 2> "$out/cpu.stderr.log"; then
    rc=0
else
    rc=$?
fi
printf "%s\n" "$rc" > "$out/exit-code.txt"
printf "Saved %s, exit code %s\n" "$out" "$rc"

case "$rc" in
  0) echo "Completed; inspect the end record." ;;
  3) echo "Partial/unqualified; inspect quality and cleanup before comparing." ;;
  *) echo "Run did not complete normally; inspect diagnostics." >&2 ;;
esac

Compare like-for-like runs

For a stock-kernel versus HFTKernel comparison, use the same executable build, CPU role, memory mode, operation, pattern, size, seed, reset policy, explicit memory policies, counter-read mode, batch, output destination, window size and duration. Preserve kernel, topology and scheduler metadata from each run. Change the intended variable, rather than changing the measurement definition at the same time.

Run each experiment separately. Launching two active tools on the same CPU is allowed by this release, but the runs then interfere and are not independent baselines. Keep diagnostic tracing separate from the primary A/B measurement when the goal is to avoid introducing a different tracing load into only one side.

Compare counts, distributions, maximums, threshold exceedances, coverage and quality together. Do not average p99 values across runs or windows. A whole-run compact total pools reported histograms inside that run; exact cross-run population analysis requires retaining compatible underlying data. The 3.x CPU/MEM CLI and measurement populations are not interchangeable with the older 2.x benchmark profiles already published on the HFTKernel site.

Source reference: S1, S3, S6, S9.

Orderly IRQ shutdown stops recording, drains within bounded limits, disables owned events, closes descriptors, removes only its own instance and restores affinity. A handled failure remains a failure even if later cleanup releases resources. SIGKILL and host failure cannot guarantee complete instance removal.

Source reference: S2, S3, S6, S7, S8, S9.

This guide is derived from the supplied Jitter Tools 1.3.0 source, CLI parsers and measurement contracts. It does not certify live timing accuracy or compatibility with every distribution. Source file paths below are relative to the source archive.

Source archive: jitter-tools-1.3.0-source(1).tgz
SHA-256: 39f803db80f8e598af67fa93171b0c0ead0c6350d2911bc9368b639cdee746ec