Skip to content

feat(bench)!: interleave --at refs per scenario and read several catalogs - #1439

Merged
azimafroozeh merged 3 commits into
ModernRelay:mainfrom
azimafroozeh:0445-bench-raw-surface
Nov 17, 2026
Merged

azimafroozeh merged 3 commits into
ModernRelay:mainfrom
azimafroozeh:0445-bench-raw-surface

Conversation

@azimafroozeh

@azimafroozeh azimafroozeh commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

omnigraph-bench run --store … --at <baseline> --at <candidate> measures the baseline's whole selection and then the candidate's, so an hour of machine drift lands on one side of every paired comparison, and every command takes one --config, so a job with two catalogs merges their names outside the harness. The same job hands the measured process an assumed role's temporary keys as static environment, the one credential source object_store never renews, so a run longer than that session fails at its next store call.

The harness now interleaves the refs per scenario, in --at order with the baseline first, so drift moves both sides alike; records stay one per scenario per ref, so the record, the report --json row and the store layout are unchanged. --catalog is repeatable and a name resolves in the first catalog that declares it as a suite or a scenario, --config keeps its one-catalog meaning and conflicts with it, and the catalog's groups: key is suites:, the old key refused with a message naming the new one.

A release-only integration test runs a two-scenario suite under --at HEAD --at HEAD~1 over a store and reads the records back through report: two records per ref under distinct session ids, the rows ordered first scenario at both refs, then the second. A debug test resolves show across two catalogs in both orders and fails an absent name naming both paths, and a unit test pins the eight AWS_* names of the renewing sources in the measured process's environment.

For review: the order within a scenario is fixed, the baseline first, so a scenario that gains from what the previous ref left in the page cache shows it as an order effect in the series, and alternating per scenario is the one-line answer then. An unknown name under a single --catalog keeps the message --config prints, without the path; two or more catalogs name theirs. The interleave test needs a release build and a product binary, so no check runs it; it passed locally. groups: is the breaking change: a custom catalog moves its key to suites:. The environment pin is a Rust test because the claim is about the measured process's environment.


Devin Review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T12:40:18.628158Z 48ddc9b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 48ddc9b5d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@azimafroozeh
azimafroozeh added this pull request to the merge queue Nov 17, 2026
Merged via the queue into ModernRelay:main with commit 5a1ec45 Nov 17, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant