feat: add sharded Codex security benchmark - #980
Conversation
github-actions
Bot
added
documentation
github-actions
Bot
added
the
review-policy: needs-review
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1fa6adc0e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Adds a new sharded Codex security benchmark execution path (triggered via repository_dispatch) that plans an exact two-packet review, runs up to two parallel shard reviews, validates/“finalizes” incomplete outcomes fail-closed, and deterministically aggregates results without a second model pass. This is benchmark-only and is intended to keep production review behavior unchanged until gated.
Changes:
- Introduces a reusable workflow to run two bounded shard jobs plus trusted finalization and deterministic aggregation.
- Adds trusted Python tooling to plan shards, render a prompt, verify shard plans, validate shard outputs, and aggregate results.
- Extends CI gating to run the new sharding invariants/tests and updates the design doc to reflect accepted limits.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
docs/plans/2026-08-27-sharded-codex-security-review-tdd.md |
Updates TDD status/tracker and revises packet line-limit rationale. |
.github/workflows/pr-gate.yml |
Runs Codex sharding tests in the PR gate workflow. |
.github/workflows/codex-security-review-benchmark.yml |
Adds a sharded benchmark dispatch type and routes to the reusable sharded case workflow. |
.github/workflows/codex-security-review-sharded-benchmark-case.yml |
New reusable workflow that plans, runs, finalizes, and aggregates shard reviews. |
.github/codex-sharded-review-prompt.md |
New trusted prompt template for shard-scoped review runs. |
.github/scripts/plan_codex_review_shards.py |
Builds the trusted shard manifest + packet(s) with hard two-packet limits. |
.github/scripts/render_codex_shard_prompt.py |
Renders the trusted prompt via strict placeholder substitution. |
.github/scripts/verify_codex_shard_plan.py |
Revalidates manifest digest and packet metrics after artifact download. |
.github/scripts/write_codex_shard_result.py |
Validates model output shape/Markdown and writes shard result artifacts. |
.github/scripts/aggregate_codex_shard_results.py |
Validates shard artifacts and deterministically aggregates results (fail-closed). |
.github/scripts/evaluate_review_policy_test.py |
Updates workflow-policy tests for the new dispatch type and PR gate step changes. |
.github/scripts/codex_sharding_test.py |
Adds executable tests covering sharding planner/prompt/verification/aggregation invariants. |
Suppressed comments (1)
.github/workflows/codex-security-review-benchmark.yml:26
- The concurrency group uses
client_payload['context-variant'] || 'all'even for the sharded dispatch path, but the selection logic defaults sharded runs tounified-40. This means two sharded dispatches that omitcontext-variantwill serialize under...-all-..., while a dispatch that explicitly setscontext-variant: unified-40will use a different concurrency group even though it runs the same effective configuration.
group: >-
codex-security-review-benchmark-${{
github.event.action
}}-${{
github.event.client_payload.corpus || 'adjudicated'
}}-${{
github.event.client_payload['context-variant'] || 'all'
}}-${{
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c864e0e59
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🔐 Codex Security Review
Review SummaryOverall Risk: HIGH Findings[HIGH] Automated review incomplete
NotesHuman review is required because the bounded automated review was incomplete. Generated by Codex Security Review | |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 735f9b98b8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ca33d4848
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0f8aa47268
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
0f8aa47 to
4181503
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4181503583
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d9e308712b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0da238dc93
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
be63e67 to
eec0c15
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eec0c15a0a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a22d037de3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cb59cc048d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cd55d915a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b1a6c259c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99e48e973e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99f6d780b7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d5cfffe2e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0bd8bdac53
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
github-actions
Bot
added
review-policy: human-approved
github-actions
Bot
added
review-policy: needs-review
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 55e25c0084
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Reviewable diff: +2319/-44 across 10 files (excludes generated, test, and story files).
Summary
Adds a trusted, benchmark-only sharded Codex review path for replaying fixed historical pull requests within a review-wide two-packet bound. Each case is planned from the exact three-dot diff, reviewed by at most two parallel model jobs, finalized fail-closed, and aggregated deterministically without a second model pass. Production review behavior remains unchanged until the sharded candidate completes and passes the adjudicated and large-PR gates.
How it works
A new typed
repository_dispatchevent selects only checked-in corpus ranges withunified=40,xhigh, and the baseline prompt. Cases run serially to cap global API concurrency. For each case, a secretless trusted preparation job builds at most two packets plus sanitized exact-head snapshots with Git refs and benchmark labels removed; two model-only jobs then run in one parallel wave behind enforceable six-minute job bounds. Trusted finalizers bind results to exact job identities and distinguish active model timeout from post-model or handoff failure. Aggregation preserves artifacts and severities deterministically, then fails incomplete cases so the parent cannot report a green 6/6 run.flowchart LR A["Trusted repository dispatch"] --> B["Select fixed corpus cases"] B --> C["Plan exact review-wide shards"] C --> D{"Fits two bounded packets?"} D -->|"No"| E["Validated oversized-review HIGH"] D -->|"Yes"| P["Prepare isolated ref-free snapshots"] P --> F["Shard 1 model-only job"] P --> G["Shard 2 model-only job"] F --> H["Trusted shard finalizers"] G --> H E --> H H --> I["Deterministic aggregate artifact"]stateDiagram-v2 [*] --> Planned Planned --> Completed: usable model output Planned --> Incomplete: verified timeout or unusable output Planned --> Oversized: review cannot fit two packets Planned --> AutomationFailure: setup or trusted handoff failure Completed --> Aggregated Incomplete --> AggregatedHigh Oversized --> AggregatedHigh AggregatedHigh --> HardFailure: artifacts preserved, case fails AutomationFailure --> HardFailureAreas of the code involved
.github/workflows/codex-security-review-benchmark.yml.github/workflows/codex-security-review-sharded-benchmark-case.yml.github/scripts/plan_codex_review_shards.py.github/scripts/verify_codex_shard_plan.py.github/scripts/write_codex_shard_result.py.github/scripts/aggregate_codex_shard_results.py.github/codex-sharded-review-prompt.md, renderer.github/workflows/pr-gate.yml, test filesdocs/plans/2026-08-27-sharded-codex-security-review-tdd.mdKey technical decisions & trade-offs
max-parallel: 2, rather than giving each architecture domain independent concurrency..githubautomation surface; search two-bin feasibility within a deterministic state budget and emitoversized-reviewinstead of splitting files/hunks or running unbounded.oversized-review.HIGH.Testing & validation
python3 .github/scripts/evaluate_review_policy_test.py— 72 tests passed.python3 .github/scripts/codex_sharding_test.py— 52 tests passed, including contract-parity, isolation, completion-gate, and executable timeout-classification coverage..github/scripts/*.pyfiles.git diff --checkand repository commit/push hooks passed.repository_dispatchintentionally loads only default-branch code, so real adjudication begins only after this benchmark implementation lands onmain. Production sharding is explicitly out of scope.