Skip to content

Preserve Python and Node symbols in memory flamegraphs - #556

Open
not-matthias wants to merge 13 commits into
mainfrom
cod-3654-support-memory-flamegraphs-for-pythonnode
Open

not-matthias wants to merge 13 commits into
mainfrom
cod-3654-support-memory-flamegraphs-for-pythonnode

Conversation

@not-matthias

@not-matthias not-matthias commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Capture per-process perf maps and Python JIT unwind data alongside native memory artifacts.
  • Enable V8 perf maps for memory runs while preserving existing NODE_OPTIONS.
  • Document native-allocation coverage and runtime limitations.
  • One weird edge case where binaries can be mapped twice into a single address space

Verification

  • cargo test --release --bin codspeed writes_keyed_artifacts_and_metadata_for_a_streamed_mapping
  • cargo fmt --all --check
  • Local Python 3.12 perf-trampoline and Node 22 memory runs produced perf maps and resolved language frames when parsed offline.

@codspeed

codspeed Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 10.47%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 2 regressed benchmarks
✅ 29 untouched benchmarks
⏩ 6 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime memtrack track tar 8.4 s 9.4 s -10.71%
❌ Memory encode_events_realistic[8] 169.6 MB 189 MB -10.24%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-3654-support-memory-flamegraphs-for-pythonnode (a01c776) with main (214c040)

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@not-matthias
not-matthias force-pushed the cod-3654-support-memory-flamegraphs-for-pythonnode branch 2 times, most recently from 9108e6c to 1b0d6fd Compare October 5, 2026 14:27
@not-matthias
not-matthias marked this pull request as ready for review October 5, 2026 14:46
@greptile-apps

greptile-apps Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

[High risk] Adds Python and Node.js symbol support to memory profiling.

The PR is not yet safe to merge because plain Node wall-time benchmarks now run with a memory-specific V8 option that can alter their measurements.

Fix All in Claude CodeFindings

  1. P1 Wall-time runs gain profiling overhead ▶
  2. P2 JIT unwind records get discarded ▶
  3. P2 Node symbol support overstated ▶
Fix with agent prompt
### Issue 1
crates/exec-harness/src/node.rs:5
For a plain Node.js wall-time benchmark, this shared option list now adds `--interpreted-frames-native-stack` to every timed child; previously it was memory-only. The flag changes how V8 runs interpreted frames and increases its code footprint, so measured times can include profiling overhead unrelated to the workload. Keep this option specific to memory runs.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
src/executor/memory/module_artifacts.rs:65-70
If appending one process's JIT symbols fails after another process was processed, this fallback discards the successful process's unwind records too, while leaving its symbols in place. The completed profile may name those frames but be unable to unwind through them. Preserve records collected for processes that succeeded.

### Issue 3
README.md:167-168
Removing the `codspeed exec` qualification implies that Node.js memory flamegraphs have JavaScript function names through any supported integration. Exec-harness enables perf maps, but the direct [Node integration's memory-mode flags](https://github.com/codspeedhq/codspeed-node/blob/HEAD/packages/core/src/introspection.ts) omit `--perf-basic-prof`. Users of that path can therefore get native allocations without the JavaScript function names this PR aims to preserve. Retain the qualification or document the required runtime configuration.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR collects runtime perf maps and Python JIT unwind data for memory profiles, retains multiple placements of a mapped native module, and adds interpreter coverage tests.

  • The latest changes make harvest failures nonfatal, extend a Node profiling option to all exec-harness modes, add the interpreter tests to CI, and revise the memory documentation.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  B[Benchmark process] --> M[Memtrack mappings and stacks]
  B --> P[Runtime perf maps and JIT dumps]
  M --> A[Native symbol and unwind artifacts]
  P --> A
  A --> D[memtrack.metadata and profile folder]
Loading

Reviews (3) · Last reviewed commit: "fix(memory): keep every placement of a m..."

Comment thread crates/exec-harness/src/analysis/mod.rs Outdated
Comment thread src/executor/memory/module_artifacts.rs Outdated
Comment thread crates/memtrack/tests/interpreter_tests.rs
@not-matthias
not-matthias force-pushed the cod-3654-support-memory-flamegraphs-for-pythonnode branch 2 times, most recently from 366917f to 25472c3 Compare October 5, 2026 15:58
Comment thread crates/exec-harness/src/node.rs Outdated
Comment thread src/executor/memory/module_artifacts.rs
Comment thread README.md
@not-matthias
not-matthias force-pushed the cod-3654-support-memory-flamegraphs-for-pythonnode branch from a01c776 to 5e00d30 Compare October 6, 2026 10:12
@not-matthias
not-matthias changed the base branch from main to cod-3746-investigate-unresolved-symbols-and-truncated-memory-call October 6, 2026 10:12
@not-matthias
not-matthias added this pull request to stack #568 October 6, 2026 10:20

@GuillaumeLagrange GuillaumeLagrange left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

olgtm, second round will be quick

Comment thread crates/exec-harness/src/analysis/mod.rs Outdated
Comment thread crates/exec-harness/src/node.rs Outdated
Comment thread crates/exec-harness/src/walltime/benchmark_loop.rs Outdated
Comment thread crates/memtrack/tests/interpreter_tests.rs Outdated
Comment thread src/executor/shared/module_artifacts/loaded_module.rs
Comment thread src/executor/wall_time/profiler/perf/jit_dump.rs Outdated
Module artifact extraction checks that a mapped path still names the file
that was mapped by comparing the perf MMAP2 (dev, ino) with stat(path).
On nested overlayfs the two disagree for unchanged files: perf and
/proc/PID/maps report the overlay superblock's device, while stat can
return a per-layer pseudo device when the outer overlay cannot encode its
layer in the inode's high bits (the inner overlay's xino already uses them).
Every system library was rejected as changed, so libc and ld.so shipped no
symbols or unwind data and their frames showed as unresolved addresses.

Map each module once and read the device and inode of that mapping from
/proc/self/maps, which the kernel derives the same way as MMAP2. Replaced
files still get a different inode and are still rejected. Symbols, load
bias and unwind data are parsed from the same mapping, so a path replaced
during extraction cannot pair the verified identity with another file's
contents.
Python and Node write runtime symbols to /tmp/perf-<pid>.map, and Python JIT dumps carry the unwind data needed to walk through interpreter trampolines. Collect both for the benchmark processes before saving the memtrack metadata, reusing the walltime artifact pipeline, so offline allocation stacks keep their runtime frames.
Memory stacks can only name Python and Node frames from the runtime perf maps. Set PYTHONPERFSUPPORT and the Node perf options per benchmark command, as simulation mode already does. Memory mode also passes --interpreted-frames-native-stack, since interpreted JS frames otherwise all resolve to the shared V8 interpreter trampoline.
Track a native allocation made from a Python function (perf trampoline) and a Node function (V8 perf-basic-prof). Assert the allocation carries a captured stack and that the runtime perf map names the allocating function, which offline attribution needs.
A process can map the same file at several addresses at once. V8 remaps its
embedded builtins out of the node binary into its code range, so node runs
from both its original text and that copy. Each placement has its own load
bias, but only the last mapping per (path, pid) was kept, so frames in every
earlier placement lost their symbols and unwind data. In a Node memory
profile, all native node frames showed up as unresolved addresses.

Record every distinct load bias and the unwind data of every executable
mapping for each process, and emit them all in the artifact metadata. The
walltime perf path shares this bookkeeping and gets the same fix.

Add a sample that maps its own text a second time and allocates through
both copies. The test feeds that process's real mappings into the artifact
pipeline and checks that both copies resolve.

Refs COD-1377
A failed perf-map append aborted the whole JIT harvest, discarding unwind
data already collected for other pids while their symbols stayed written.
Handle it per pid like the other per-dump failures. The function can no
longer fail, so return the map directly; wall-time no longer aborts the
benchmark save on this error.
Base automatically changed from cod-3746-investigate-unresolved-symbols-and-truncated-memory-call to main October 6, 2026 13:06
@not-matthias
not-matthias force-pushed the cod-3654-support-memory-flamegraphs-for-pythonnode branch from 5e00d30 to 1f73202 Compare October 6, 2026 13:19
…rapper

The runner (`helpers/env.rs`) and exec-harness each defined the environment
a benchmark process needs: `CODSPEED_RUNNER_MODE`, the Python hash seed and
perf-map switches, and the Java tool options. Move them into one
`runner_shared::runtime_env` module, keyed on `MeasurementMode`, which moves
to runner-shared as well. The runner converts its `RunnerMode` and extends
its injected env from the shared list; the values are unchanged except that
memory mode now also sets `PYTHONPERFSUPPORT=1`, so memory flamegraphs can
name Python frames.

Add a `node` wrapper script that the module installs on `PATH`. It runs the
real node with the V8 flags codspeed-node's `getV8Flags()` would request for
the current `CODSPEED_RUNNER_MODE` and node major version: the deterministic
analysis set for simulation and memory, and the perf-prof/log-code set for
walltime. Most of these flags are rejected in `NODE_OPTIONS`, so exec-harness
targets without a codspeed-node integration had no way to get them before.

The wrapper may run under valgrind with the simulation preload library in
`LD_PRELOAD`, which reports a benchmark result from every process that loads
it. The script therefore drops `LD_PRELOAD` for its only subprocess
(`node --version`), restores it for the final `exec`, and avoids command
substitutions whose forked subshells would report on exit. Under callgrind
only the real node process emits the benchmark dump.

The install writes a staging file and renames it into place, once per
process: writing over an executing script, or a second in-process writer
racing with a concurrent fork+exec, fails with `ETXTBSY`.
Set the benchmark environment once in `execute_benchmarks`, before the mode
dispatch, through `runner_shared::runtime_env::apply_to_process`. Every
spawned command inherits it, so the per-command `set_perf_map_env` and
`set_node_options` calls in the memory, simulation and walltime loops go
away, together with the local `node.rs` and its `NODE_OPTIONS` subset.

Node targets now go through the shared `node` wrapper and receive the full
codspeed-node V8 flag set for the mode, including indirect launches through
npm, npx or `#!/usr/bin/env node` scripts. `MeasurementMode` is re-exported
from runner-shared, so the CLI is unchanged.
codspeed-node's analysis flag set has no `--perf-basic-prof`, so the node
wrapper in memory mode produced no `/tmp/perf-<pid>.map` and memory
flamegraphs lost their JS frames. Add the flag for memory mode only.

Switch the memtrack interpreter tests to the shared runtime env instead of
hand-picked interpreter flags, so they exercise the same Python env and
node wrapper the runner injects.
@not-matthias
not-matthias force-pushed the cod-3654-support-memory-flamegraphs-for-pythonnode branch from 1f73202 to 2b88350 Compare October 6, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants