Repository navigation
feat(bench)!: interleave --at refs per scenario and read several catalogs - #1439
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
Devin Review found 2 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 48ddc9b5d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
omnigraph-bench run --store … --at <baseline> --at <candidate>measures the baseline's whole selection and then the candidate's, so an hour of machine drift lands on one side of every paired comparison, and every command takes one--config, so a job with two catalogs merges their names outside the harness. The same job hands the measured process an assumed role's temporary keys as static environment, the one credential source object_store never renews, so a run longer than that session fails at its next store call.The harness now interleaves the refs per scenario, in
--atorder with the baseline first, so drift moves both sides alike; records stay one per scenario per ref, so the record, thereport --jsonrow and the store layout are unchanged.--catalogis repeatable and a name resolves in the first catalog that declares it as a suite or a scenario,--configkeeps its one-catalog meaning and conflicts with it, and the catalog'sgroups:key issuites:, the old key refused with a message naming the new one.A release-only integration test runs a two-scenario suite under
--at HEAD --at HEAD~1over a store and reads the records back throughreport: two records per ref under distinct session ids, the rows ordered first scenario at both refs, then the second. A debug test resolvesshowacross two catalogs in both orders and fails an absent name naming both paths, and a unit test pins the eightAWS_*names of the renewing sources in the measured process's environment.For review: the order within a scenario is fixed, the baseline first, so a scenario that gains from what the previous ref left in the page cache shows it as an order effect in the series, and alternating per scenario is the one-line answer then. An unknown name under a single
--catalogkeeps the message--configprints, without the path; two or more catalogs name theirs. The interleave test needs a release build and a product binary, so no check runs it; it passed locally.groups:is the breaking change: a custom catalog moves its key tosuites:. The environment pin is a Rust test because the claim is about the measured process's environment.