Improve Benchmarking by JohnReedV · Pull Request #2798 · RaoFoundation/subtensor

JohnReedV · 2026-06-25T16:23:21Z

Description

This PR

Modifies the benchmark flow so that only extrinsics that have drifted passed the threshold or have incorrect read/writes are updated rather than replacing the entire file and creating conflicts everywhere.
Adds a lint to check for extrinsics with missing benchmarks. This can be hidden with
#[allow(unknown_lints, benchmarked_weight_not_plugged)]
Adds 48 missing benchmarks
Fixes a bug where benchmarks could fail on noisy internal component drift even when the actual benchmark-level weight was still within the configured threshold.
Adds block_step to the benchmark flow so the on_initialize weight is generated instead of hardcoded. The benchmark simulates the current worst case with 2 epochs in a block.
Updates the drift check to include proof size and to compare the actual benchmark-level weight instead of failing on noisy internal component drift.

github-actions · 2026-06-25T16:25:44Z

🛡️ AI Review — Skeptic (security review)

VERDICT: SAFE

Baseline-to-medium scrutiny: established @opentensor-associated contributor with substantial subtensor history; cached repo permission is read; later commits include UnArbosSix; branch apply-benchmark-to-only-bad-benches -> devnet-ready.

The PR does not modify .github/ai-review/* or .github/copilot-instructions.md. The manifest edits only enable features on existing workspace dependencies, and I did not see a Cargo.lock change or new external dependency. The build.rs addition invokes an in-repo source lint that reads Rust files and emits Cargo warnings; I did not find command execution, network access, or secret-handling logic in that path.

The prior register_limit underweight concern remains addressed: the current benchmark uses setup_full_subnet_registration_benchmark, fills the subnet to DefaultMaxAllowedUids, seeds existing neurons, and asserts the subnet is full before measuring the dispatch path.

Findings

No findings.

Prior-comment reconciliation

7af6c073: addressed — The benchmark now uses setup_full_subnet_registration_benchmark, which fills the subnet to DefaultMaxAllowedUids and asserts the subnet is full before invoking register_limit; this covers the prune/replacement path raised in the prior finding.

Conclusion

No malicious behavior or security vulnerability found in the current diff. The previously reported benchmark coverage issue is resolved, so the Skeptic verdict is SAFE.

🔍 AI Review — Auditor (domain review)

VERDICT: 👍

Gittensor UNKNOWN; author is an established opentensor/subtensor contributor with substantial recent PR history, cached repo permission is read.

PR body is substantive and matches the benchmark/tooling scope, so I would leave it unchanged.

Prior register_limit blocker remains addressed: the benchmark now uses setup_full_subnet_registration_benchmark, fills the subnet, and dispatches against the replacement path rather than the cheap append path.

Duplicate-work check found overlapping PRs only through broad shared files such as weights/runtime-adjacent code; none appear to be alternate implementations of this benchmark-flow change.

Validation performed: bash -n scripts/benchmark_action.sh, bash -n scripts/discover_pallets.sh, and git diff --check passed. Targeted cargo test -p subtensor-linting require_extrinsic_benchmarks -- --nocapture could not start because rustup tried to create a temp file under read-only /home/runner/.rustup/tmp. Spec-version auto-fix was not performed because this sandbox could not resolve dev.chain.opentensor.ai; local spec_version is 424, and the diff does not modify runtime/src/lib.rs.

Findings

No findings.

Prior-comment reconciliation

2caec2f2: addressed — register_limit now benchmarks through setup_full_subnet_registration_benchmark, filling the subnet before dispatch so the measured path reaches replacement rather than append.

Conclusion

The prior benchmark coverage issue is fixed, and I did not find a remaining domain issue that should block merge. CI should still perform the normal live spec-version and test checks that this sandbox could not complete.

github-actions · 2026-06-25T16:28:28Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-25T17:53:38Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-25T18:52:27Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-26T16:47:33Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-26T17:17:22Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

…tensor/subtensor into apply-benchmark-to-only-bad-benches

github-actions