Skip to content

fix(metabench): isolate native probes and harden benchmark reporting - #807

Open
Martin Taillefer (geeknoid) wants to merge 1 commit into
mainfrom
fix/metabench-native-probes
Open

Martin Taillefer (geeknoid) wants to merge 1 commit into
mainfrom
fix/metabench-native-probes

Conversation

@geeknoid

Copy link
Copy Markdown
Member

Summary

  • Move allocation, perf, and VTune collection into private native-case callers, preserving benchmark bodies and collector-free ordinary adapters.
  • Bypass allocation tracking in ordinary workers and isolate each collector by validated worker mode.
  • Correct Criterion estimate/confidence reporting, native case identities, compiler compatibility, and baseline publication safety.
  • Harden artifact admission and diagnostic output; add structural, protocol, lifecycle, cross-thread, and failure-path regressions.
  • Require Linux native listing/smoke contracts in CI and document the supported measurement boundaries.

Validation

  • Scoped Anvil Clippy and spelling checks, changed-file formatting, and generated README verification passed.
  • 123 runtime unit tests, public API/protocol tests, macro structural and consumer contracts, and targeted native process regressions passed.
  • Required Linux listing/smoke checks passed; missing native prerequisites fail the required tier.
  • Miri, mutation testing, full-workspace tests, and performance measurements were not run locally.

Copilot AI balanced review requested due to automatic review settings October 11, 2026 18:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Captured worker logs remain unbounded and can exhaust filesystem storage.

1 open finding
What changed in this PR

This PR isolates Metabench collectors in native probe workers and strengthens reporting, artifact handling, diagnostics, and CI coverage.

Changes:

  • Adds generated native probe discovery/execution with isolated allocation, perf, and VTune collection.
  • Hardens report metadata, Criterion estimates, artifact admission, baseline publication, and diagnostics.
  • Adds extensive protocol, lifecycle, failure-path, and Linux native-contract tests.
File Description
.github/​workflows/​repository-checks.yml Adds required Linux native-contract checks.
Cargo.toml Updates shared Gungraun-related dependencies.
Cargo.lock Records dependency resolution changes.
docs/​metabench-native-contracts.md Documents native CI contracts.
crates/​metabench/​Cargo.toml Updates Metabench dependencies.
crates/​metabench/​README.md Regenerates crate documentation.
crates/​metabench/​build.rs Captures build-time compiler identity.
crates/​metabench/​src/​__private.rs Exposes generated probe support.
crates/​metabench/​src/​allocation.rs Activates tracking only in allocation workers.
crates/​metabench/​src/​allocator.rs Adds collector-aware allocator wrapper.
crates/​metabench/​src/​arguments.rs Adds smoke-mode argument handling.
crates/​metabench/​src/​artifact.rs Hardens artifacts and estimate parsing.
crates/​metabench/​src/​entrypoint.rs Connects generated probes to the runner.
crates/​metabench/​src/​error.rs Adds probe-specific diagnostics.
crates/​metabench/​src/​identity.rs Removes obsolete allocation-selection accessor.
crates/​metabench/​src/​lib.rs Documents measurement and reporting contracts.
crates/​metabench/​src/​metric_display.rs Updates metric labels and ordering.
crates/​metabench/​src/​perf.rs Activates perf only in validated workers.
crates/​metabench/​src/​probe.rs Implements probe discovery and execution.
crates/​metabench/​src/​report.rs Adds schema-9 estimates and safer rendering.
crates/​metabench/​src/​runner.rs Orchestrates probes, logs, smoke runs, and publication.
crates/​metabench/​src/​vtune.rs Activates VTune only in validated workers.
crates/​metabench/​examples/​instrumentation_controls.rs Adds instrumentation comparison fixture.
crates/​metabench/​examples/​measurement_boundaries.rs Adds workload-boundary fixture.
crates/​metabench/​examples/​native_probe_contracts.rs Adds lifecycle and iterator fixture.
crates/​metabench/​examples/​threaded_allocations.rs Adds cross-thread allocation fixture.
crates/​metabench/​tests/​probe_protocol.rs Tests authenticated probe rejection paths.
crates/​metabench/​tests/​spawned_benchmark.rs Expands process-level regressions.
crates/​metabench/​tests/​fixtures/​artifacts/​criterion/​duplicate/​a/​new/​estimates.json Updates Criterion estimate fixture.
crates/​metabench/​tests/​fixtures/​artifacts/​criterion/​duplicate/​b/​new/​estimates.json Updates Criterion estimate fixture.
crates/​metabench/​tests/​fixtures/​artifacts/​criterion/​invalid_numeric/​new/​estimates.json Updates invalid estimate fixture.
crates/​metabench/​tests/​fixtures/​artifacts/​criterion/​minimal/​new/​estimates.json Updates minimal estimate fixture.
crates/​metabench/​tests/​fixtures/​artifacts/​criterion/​missing_function/​new/​estimates.json Updates missing-function fixture.
crates/​metabench/​tests/​fixtures/​artifacts/​criterion/​unicode/​new/​estimates.json Updates Unicode fixture.
crates/​metabench/​tests/​fixtures/​artifacts/​traversal/​cycle/​new/​estimates.json Updates traversal fixture.
crates/​metabench_macros/​src/​lib.rs Documents collector-free macro expansion.
crates/​metabench_macros/​tests/​compile.rs Reworks compile-test fixture reuse.
crates/​metabench_macros/​tests/​ui/​pass/​generated_name_collisions.rs Expands generated-name collision coverage.
crates/​metabench_macros_impl/​src/​benchmark.rs Generates isolated probe adapters.
crates/​metabench_macros_impl/​src/​shared.rs Adds readable collision-safe native names.
crates/​metabench_macros_impl/​src/​target.rs Generates probe tables and lifecycle links.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread crates/metabench/src/runner.rs Outdated
Copilot AI balanced review requested due to automatic review settings October 11, 2026 18:47
@geeknoid

Copy link
Copy Markdown
Member Author

Fixed the Windows CI baseline failures: artifact-admission tests now respect the reader’s intentional write-sharing denial, and the oversized fixture closes its writer before admission. The actual stream-size bound remains independently tested. Amended and pushed the commit; Windows CI will verify the platform-specific path.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Markdown and baseline diagnostics still permit terminal-direction manipulation through unsanitized text.

3 open findings

🧠 Review effort: Balanced

Comment thread crates/metabench/src/report.rs Outdated
Comment thread crates/metabench/src/runner.rs Outdated
Comment thread crates/metabench/src/allocator.rs Fixed
Comment thread crates/metabench/src/allocator.rs Fixed
Comment thread crates/metabench/src/runner.rs
Comment thread crates/metabench/src/runner.rs Outdated
Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Schema-9 validation accepts confidence bounds without their required confidence level.

1 open finding
3 resolved since last review

🧠 Review effort: Balanced

Comment thread crates/metabench/src/report.rs Outdated
Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Repository report generators remain incompatible with schema 9, and the documented native smoke behavior differs from the implementation.

3 open findings

🧠 Review effort: Balanced

Comment thread crates/metabench/src/report.rs
Comment thread crates/metabench/src/lib.rs Outdated
Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It combines proc-macro generation, unsafe allocator routing, cross-platform process control, and report-schema migration without full workspace, Miri, or mutation validation.

0 open findings

3 resolved since last review

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Schema-9 validation still permits Criterion timing metrics without the statistic needed for safe baseline comparison.

1 open finding

🧠 Review effort: Balanced

Comment thread crates/metabench/src/report.rs
Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Derived probe identities can collide, and existing benchmark snapshots are mislabeled as newly computed primary estimates.

3 open findings
1 resolved since last review

🧠 Review effort: Balanced

Comment thread crates/metabench/src/probe.rs
Comment thread crates/internity/docs/COMPARISON.md Outdated
Comment thread crates/routerama/docs/PERF.md Outdated
Copilot AI balanced review requested due to automatic review settings October 11, 2026 20:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Schema validation and rejected-baseline diagnostic sanitization remain incomplete.

2 open findings
3 resolved since last review

🧠 Review effort: Balanced

Comment thread crates/metabench/src/report.rs Outdated
Comment thread crates/metabench/src/report.rs Outdated
Copilot AI balanced review requested due to automatic review settings October 11, 2026 20:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

User-controlled probe arguments can inject terminal control sequences into the new probe error diagnostic.

1 open finding
2 resolved since last review

🧠 Review effort: Balanced

Comment thread crates/metabench/src/error.rs Outdated
Copilot AI balanced review requested due to automatic review settings October 11, 2026 20:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The broad cross-platform worker, allocator, unsafe, macro-generation, and report-publication changes warrant final human review.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

- Keep benchmark bodies and ordinary adapters free of collection controls by
  generating private allocation, perf, and VTune callers from native cases.
- Gate alloc_tracker behind the same underlying allocator instance and bypass
  tracking in ordinary workers; isolate collectors by validated worker mode.
- Report Criterion slope or mean estimates with confidence metadata, validate
  bootstrap intervals, and compare baselines using the build-time compiler.
- Preserve explicit native case identities, use executable-derived artifact
  roots, and prevent empty, smoke, failed, or partial runs replacing baselines.
- Bound artifact reads using regular-file handles, create worker logs
  exclusively, and encode less-trusted terminal and Markdown output.
- Add structural macro, indexed protocol, lifecycle, cross-thread allocation,
  and collector-failure regressions, including generated-name collision checks.
- Require Linux native listing and smoke contracts in CI and update generated
  crate documentation and native-contract guidance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2cbef463-84aa-4249-91e4-20ba418301f6
Copilot AI balanced review requested due to automatic review settings October 11, 2026 20:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It substantially changes unsafe allocation instrumentation, cross-platform process control, reporting, and generated macro behavior without full-workspace or runtime-analysis validation.

0 open findings

🧠 Review effort: Balanced

@github-actions

Copy link
Copy Markdown

⚠️ SemVer check advisory

Inconclusive comparisons

cargo semver-checks could not complete the following comparisons. These failures are informational because an unbuildable baseline is not evidence of a breaking API change.

metabench (exit 101)

     Cloning cb3ef36dc0942dc93017df934a3fcacfc7f451ad
    Building metabench v0.1.1 (current)
       Built [  18.804s] (current)
     Parsing metabench v0.1.1 (current)
      Parsed [   0.013s] (current)
    Building metabench v0.1.1 (baseline)
error: running cargo-doc on crate 'metabench' failed with output:
-----
   Compiling proc-macro2 v1.0.107
   Compiling quote v1.0.47
   Compiling unicode-ident v1.0.26
   Compiling serde_core v1.0.229
   Compiling serde v1.0.229
   Compiling zmij v1.0.23
   Compiling serde_json v1.0.154
   Compiling libc v0.2.190
    Checking cfg-if v1.0.5
    Checking itoa v1.0.18
   Compiling syn v3.0.8
   Compiling syn v2.0.119
    Checking memchr v2.8.3
   Compiling zerocopy v0.8.62
    Checking bitflags v2.13.2
   Compiling hashbrown v0.17.1
   Compiling shlex v2.0.1
   Compiling rustversion v1.0.23
   Compiling winnow v1.0.4
   Compiling equivalent v1.0.2
   Compiling find-msvc-tools v0.1.14
   Compiling semver v1.0.28
   Compiling rustc_version v0.4.1
   Compiling cc v1.8.0
   Compiling toml_parser v1.1.5+spec-1.1.0
   Compiling serde_derive v1.0.229
   Compiling indexmap v2.14.2
   Compiling zerocopy-derive v0.8.62
   Compiling derive_more-impl v2.1.1
   Compiling rustix v1.1.5
   Compiling toml_datetime v1.1.2+spec-1.1.0
   Compiling autocfg v1.5.1
   Compiling num-traits v0.2.19
   Compiling toml_edit v0.25.17+spec-1.1.0
    Checking either-or-both v0.3.1
   Compiling alloca v0.4.0
   Compiling gungraun-macros v0.9.1
   Compiling proc-macro-error-attr3 v3.1.1
    Checking regex-syntax v0.8.11
   Compiling thiserror v2.0.21
    Checking ciborium-io v0.2.2
   Compiling getrandom v0.4.3
    Checking linux-raw-sys v0.12.1
   Compiling cfg_aliases v0.2.2
    Checking either v1.19.0
   Compiling bincode-next v3.1.1
    Checking anstyle v1.0.14
    Checking clap_lex v1.1.1
    Checking itertools v0.13.0
    Checking clap_builder v4.6.7
    Checking regex-automata v0.4.18
    Checking half v2.7.1
    Checking ciborium-ll v0.2.2
   Compiling nix v0.31.3
   Compiling proc-macro-error3 v3.1.1
    Checking rapidhash v4.5.1
   Compiling proc-macro-crate v3.5.0
   Compiling derive_more v2.1.1
   Compiling thiserror-impl v2.0.21
   Compiling gungraun v0.19.4
    Checking same-file v1.0.6
   Compiling pastey v0.2.3
    Checking cast v0.3.0
    Checking unty-next v0.1.2
    Checking criterion-plot v0.8.2
    Checking walkdir v2.5.0
    Checking regex v1.13.1
   Compiling metabench_macros_impl v0.1.1 (/home/runner/work/oxidizer/oxidizer/target/semver-checks/git-cb3ef36dc0942dc93017df934a3fcacfc7f451ad/5e0aa86380776f0db5a91dc7d93ae494706f69b9/crates/metabench_macros_impl)
    Checking gungraun-runner v0.20.0
    Checking ciborium v0.2.2
    Checking clap v4.6.7
    Checking gungraun-runner v0.19.4
    Checking folo_utils v0.1.14
    Checking tinytemplate v1.2.1
    Checking nix v0.27.1
    Checking page_size v0.6.0
    Checking oorandom v11.1.5
    Checking fastrand v2.5.0
    Checking once_cell v1.21.4
   Compiling metabench v0.1.1 (/home/runner/work/oxidizer/oxidizer/target/semver-checks/git-cb3ef36dc0942dc93017df934a3fcacfc7f451ad/5e0aa86380776f0db5a91dc7d93ae494706f69b9/crates/metabench)
    Checking jiff-core v0.1.1
    Checking anes v0.1.6
    Checking criterion v0.8.2
    Checking gungraun-summary v6.0.0
error[E0432]: unresolved import `gungraun_runner::api::ValgrindTool`
  --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/gungraun-summary-6.0.0/src/v6.rs:10:59
   |
10 |     CachegrindMetric, DhatMetric, ErrorMetric, EventKind, ValgrindTool,
   |                                                           ^^^^^^^^^^^^ no `ValgrindTool` in `api`

error[E0432]: unresolved imports `gungraun_runner::metrics::model::MetricsDiff`, `gungraun_runner::metrics::model::MetricsSummary`
  --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/gungraun-summary-6.0.0/src/v6.rs:12:63
   |
12 | pub use gungraun_runner::metrics::model::{Metric, MetricKind, MetricsDiff, MetricsSummary};
   |                                                               ^^^^^^^^^^^  ^^^^^^^^^^^^^^ no `MetricsSummary` in `metrics::model`
   |                                                               |
   |                                                               no `MetricsDiff` in `metrics::model`

error[E0432]: unresolved imports `gungraun_runner::summary::model::Diffs`, `gungraun_runner::summary::model::FlamegraphSummary`, `gungraun_runner::summary::model::ProfileInfo`, `gungraun_runner::summary::model::SummaryFormat`, `gungraun_runner::summary::model::SummaryOutput`, `gungraun_runner::summary::model::ToolMetricSummary`
  --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/gungraun-summary-6.0.0/src/v6.rs:14:38
   |
14 |     BenchmarkKind, BenchmarkSummary, Diffs, FlamegraphSummary, Profile, ProfileData, ProfileInfo,
   |                                      ^^^^^  ^^^^^^^^^^^^^^^^^                        ^^^^^^^^^^^ no `ProfileInfo` in `summary::model`
   |                                      |      |
   |                                      |      no `FlamegraphSummary` in `summary::model`
   |                                      no `Diffs` in `summary::model`
15 |     ProfilePart, ProfileTotal, Profiles, SCHEMA_VERSION, SummaryFormat, SummaryOutput,
   |                                                          ^^^^^^^^^^^^^  ^^^^^^^^^^^^^ no `SummaryOutput` in `summary::model`
   |                                                          |
   |                                                          no `SummaryFormat` in `summary::model`
16 |     ToolMetricSummary, ToolRegression,
   |     ^^^^^^^^^^^^^^^^^ no `ToolMetricSummary` in `summary::model`

For more information about this error, try `rustc --explain E0432`.
error: could not compile `gungraun-summary` (lib) due to 3 previous errors
warning: build failed, waiting for other jobs to finish...

-----

error: failed to build rustdoc for crate metabench v0.1.1
note: this is usually due to a compilation error in the crate,
      and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/oxidizer/oxidizer/target/semver-checks/git-cb3ef36dc0942dc93017df934a3fcacfc7f451ad/5e0aa86380776f0db5a91dc7d93ae494706f69b9/crates/metabench &&
          cargo check &&
          cargo doc

error: aborting due to failure to build rustdoc for crate metabench v0.1.1

@codecov

codecov Bot commented Oct 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.0%. Comparing base (cb3ef36) to head (f282f88).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #807   +/-   ##
=======================================
  Coverage   100.0%   100.0%           
=======================================
  Files         798      796    -2     
  Lines      119402   119357   -45     
=======================================
- Hits       119402   119357   -45     
Flag Coverage Δ
linux 100.0% <ø> (ø)
linux-arm 100.0% <ø> (ø)
scheduled ?
windows 100.0% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants