Repository navigation
fix(metabench): isolate native probes and harden benchmark reporting - #807
Martin Taillefer (geeknoid) wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Captured worker logs remain unbounded and can exhaust filesystem storage.
1 open finding
What changed in this PR
This PR isolates Metabench collectors in native probe workers and strengthens reporting, artifact handling, diagnostics, and CI coverage.
Changes:
- Adds generated native probe discovery/execution with isolated allocation, perf, and VTune collection.
- Hardens report metadata, Criterion estimates, artifact admission, baseline publication, and diagnostics.
- Adds extensive protocol, lifecycle, failure-path, and Linux native-contract tests.
| File | Description |
|---|---|
.github/workflows/repository-checks.yml |
Adds required Linux native-contract checks. |
Cargo.toml |
Updates shared Gungraun-related dependencies. |
Cargo.lock |
Records dependency resolution changes. |
docs/metabench-native-contracts.md |
Documents native CI contracts. |
crates/metabench/Cargo.toml |
Updates Metabench dependencies. |
crates/metabench/README.md |
Regenerates crate documentation. |
crates/metabench/build.rs |
Captures build-time compiler identity. |
crates/metabench/src/__private.rs |
Exposes generated probe support. |
crates/metabench/src/allocation.rs |
Activates tracking only in allocation workers. |
crates/metabench/src/allocator.rs |
Adds collector-aware allocator wrapper. |
crates/metabench/src/arguments.rs |
Adds smoke-mode argument handling. |
crates/metabench/src/artifact.rs |
Hardens artifacts and estimate parsing. |
crates/metabench/src/entrypoint.rs |
Connects generated probes to the runner. |
crates/metabench/src/error.rs |
Adds probe-specific diagnostics. |
crates/metabench/src/identity.rs |
Removes obsolete allocation-selection accessor. |
crates/metabench/src/lib.rs |
Documents measurement and reporting contracts. |
crates/metabench/src/metric_display.rs |
Updates metric labels and ordering. |
crates/metabench/src/perf.rs |
Activates perf only in validated workers. |
crates/metabench/src/probe.rs |
Implements probe discovery and execution. |
crates/metabench/src/report.rs |
Adds schema-9 estimates and safer rendering. |
crates/metabench/src/runner.rs |
Orchestrates probes, logs, smoke runs, and publication. |
crates/metabench/src/vtune.rs |
Activates VTune only in validated workers. |
crates/metabench/examples/instrumentation_controls.rs |
Adds instrumentation comparison fixture. |
crates/metabench/examples/measurement_boundaries.rs |
Adds workload-boundary fixture. |
crates/metabench/examples/native_probe_contracts.rs |
Adds lifecycle and iterator fixture. |
crates/metabench/examples/threaded_allocations.rs |
Adds cross-thread allocation fixture. |
crates/metabench/tests/probe_protocol.rs |
Tests authenticated probe rejection paths. |
crates/metabench/tests/spawned_benchmark.rs |
Expands process-level regressions. |
crates/metabench/tests/fixtures/artifacts/criterion/duplicate/a/new/estimates.json |
Updates Criterion estimate fixture. |
crates/metabench/tests/fixtures/artifacts/criterion/duplicate/b/new/estimates.json |
Updates Criterion estimate fixture. |
crates/metabench/tests/fixtures/artifacts/criterion/invalid_numeric/new/estimates.json |
Updates invalid estimate fixture. |
crates/metabench/tests/fixtures/artifacts/criterion/minimal/new/estimates.json |
Updates minimal estimate fixture. |
crates/metabench/tests/fixtures/artifacts/criterion/missing_function/new/estimates.json |
Updates missing-function fixture. |
crates/metabench/tests/fixtures/artifacts/criterion/unicode/new/estimates.json |
Updates Unicode fixture. |
crates/metabench/tests/fixtures/artifacts/traversal/cycle/new/estimates.json |
Updates traversal fixture. |
crates/metabench_macros/src/lib.rs |
Documents collector-free macro expansion. |
crates/metabench_macros/tests/compile.rs |
Reworks compile-test fixture reuse. |
crates/metabench_macros/tests/ui/pass/generated_name_collisions.rs |
Expands generated-name collision coverage. |
crates/metabench_macros_impl/src/benchmark.rs |
Generates isolated probe adapters. |
crates/metabench_macros_impl/src/shared.rs |
Adds readable collision-safe native names. |
crates/metabench_macros_impl/src/target.rs |
Generates probe tables and lifecycle links. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
5ae2b8f to
6a43159
Compare
|
Fixed the Windows CI baseline failures: artifact-admission tests now respect the reader’s intentional write-sharing denial, and the oversized fixture closes its writer before admission. The actual stream-size bound remains independently tested. Amended and pushed the commit; Windows CI will verify the platform-specific path. |
6a43159 to
187b972
Compare
187b972 to
3d9af32
Compare
3d9af32 to
28320ec
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
It combines proc-macro generation, unsafe allocator routing, cross-platform process control, and report-schema migration without full workspace, Miri, or mutation validation.
0 open findings
3 resolved since last review
🧠 Review effort: Balanced
28320ec to
cb5b75d
Compare
cb5b75d to
7bfd73f
Compare
7bfd73f to
74abcd8
Compare
74abcd8 to
c819757
Compare
c819757 to
44c0878
Compare
- Keep benchmark bodies and ordinary adapters free of collection controls by generating private allocation, perf, and VTune callers from native cases. - Gate alloc_tracker behind the same underlying allocator instance and bypass tracking in ordinary workers; isolate collectors by validated worker mode. - Report Criterion slope or mean estimates with confidence metadata, validate bootstrap intervals, and compare baselines using the build-time compiler. - Preserve explicit native case identities, use executable-derived artifact roots, and prevent empty, smoke, failed, or partial runs replacing baselines. - Bound artifact reads using regular-file handles, create worker logs exclusively, and encode less-trusted terminal and Markdown output. - Add structural macro, indexed protocol, lifecycle, cross-thread allocation, and collector-failure regressions, including generated-name collision checks. - Require Linux native listing and smoke contracts in CI and update generated crate documentation and native-contract guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2cbef463-84aa-4249-91e4-20ba418301f6
44c0878 to
f282f88
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
It substantially changes unsafe allocation instrumentation, cross-platform process control, reporting, and generated macro behavior without full-workspace or runtime-analysis validation.
0 open findings
🧠 Review effort: Balanced
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #807 +/- ##
=======================================
Coverage 100.0% 100.0%
=======================================
Files 798 796 -2
Lines 119402 119357 -45
=======================================
- Hits 119402 119357 -45
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|



Summary
Validation