bot-gated PR review runs can disappear with no review check or surfaced skip reason

[samuelkahessay]

gh-aw version: v0.66.1
Discovered: 2026-04-05
Category: Activation / observability
Severity: Medium

What happens

In a private same-repo pull_request review run observed on 2026-04-05, the pr-review-agent workflow did trigger, but pre_activation denied the actor and set activated=false. That caused activation, agent, detection, safe_outputs, and conclusion to all skip.

The actor on that run was the pipeline bot:

• PR author: app/prd-to-prod-pipeline
• runtime actor in the log: prd-to-prod-pipeline[bot]

The workflow currently has no on.bots: allowlist, so this deny is explainable. The upstream problem is what happens next: from the PR UI, the run effectively disappears. There is no review check row, no neutral/skipped indicator, and no PR comment explaining why review did not happen. The only place the reason is visible is inside the pre_activation job log.

That makes a skipped review indistinguishable from:

• a workflow that never triggered
• a workflow that is still queued
• a workflow that is misconfigured or disabled

What should happen

If gh-aw decides not to activate an agent workflow because the actor failed a role/bot check, the PR should still get a visible negative signal.

At minimum:

1. Emit a neutral/skipped check run for the workflow with the skip reason.
2. Surface check_membership's result / error_message in a job summary or comment, not only in raw logs.
3. Point users to the relevant remediation (on.bots: / on.roles:) in that surfaced message.

The deny itself can be correct. The bug is that the deny is not visible from the PR surface.

Where in the code

• Upstream docs/src/content/docs/reference/frontmatter.md:245-275 documents on.bots: as the intended allowlist mechanism for bot actors.
• Upstream pkg/workflow/role_checks.go:17-39 compiles the membership-check step and injects GH_AW_REQUIRED_ROLES plus GH_AW_ALLOWED_BOTS when configured.
• Upstream actions/setup/js/check_membership.cjs:13-24 only bypasses validation for workflow_dispatch when write is allowed; pull_request events continue through permission checks.
• Upstream actions/setup/js/check_membership.cjs:52-105 checks repository permission first, then optionally falls back to the allowed-bots list, and finally writes is_team_member=false plus a result/error message for unauthorized actors.
• Upstream pkg/workflow/compiler_activation_job.go:445-471 makes the activation job depend solely on needs.pre_activation.outputs.activated == 'true'.
• Local compiled workflow confirms the gate:
  • aurrin-platform/.github/workflows/pr-review-agent.lock.yml:63-64
  • aurrin-platform/.github/workflows/pr-review-agent.lock.yml:1006-1027

Evidence

Observed run

• Private same-repo review run on 2026-04-05
• Event: pull_request
• Jobs:
  • pre_activation: success
  • activation: skipped
  • agent: skipped
  • detection: skipped
  • safe_outputs: skipped
  • conclusion: skipped

Exact pre_activation log excerpt

Checking if user 'prd-to-prod-pipeline[bot]' has required permissions for Aurrin-Ventures/aurrin-platform
Required permissions: admin, maintainer, write
Repository permission level: none
User permission 'none' does not meet requirements: admin, maintainer, write
Evaluate and set job outputs
Set output 'activated'

That is the root cause of the skip. It was not concurrency, batch timing, or a missing trigger. The run fired and then self-suppressed at the membership gate.

Local workflow context

aurrin-platform/.github/workflows/pr-review-agent.md has:

• default role enforcement (no explicit on.roles: override, so v0.66.1 defaults to admin,maintainer,write)
• no on.bots: allowlist for the pipeline bot

So this specific deny is configuration-sensitive. The verified upstream finding is the missing PR-level skip signal after the deny.

Proposed fix

check_membership.cjs already computes the exact reason string the user needs. The problem is that the string stays in job logs.

• actions/setup/js/check_membership.cjs:52-105 already writes is_team_member=false plus a result / error_message describing which permission check failed and which allowlists were consulted.
• That string is the one currently only visible in the pre_activation job log.

So the minimal upstream fix is plumbing, not a new mechanism:

1. When pre_activation ends with activated=false, emit a neutral/skipped check run for the workflow using the result / error_message already computed by check_membership.
2. Bubble the same string into the workflow run summary so operators do not need to open raw logs.
3. Include a short remediation hint for common causes — e.g. "Actor was a bot with no on.bots: allowlist configured."

Optional follow-up: add a docs example for automation-heavy repos that create same-repo PRs through GitHub Apps or bots, showing the on.bots: configuration needed to let those PRs reach the agent.

Because the data already exists, step 1 alone resolves the finding.

Related upstream issues

• #20510 (closed) — bot allowlist was ignored. Different bug: wrong gate decision.
• #21098 (closed) — bot fallback was skipped on the error path. Different bug: wrong gate decision.
• #19467 (open) and #19023 (closed) — concurrency-collapse bugs. Different bug family: disappearing runs due to grouping, not a legitimate pre_activation deny.

I did not find a prior issue covering the specific "legitimate pre_activation deny, no surfaced skip reason, no check run" observability gap described here.

Impact

Medium. The skip may be intentional, but the current UX makes it look like review never happened for unknown reasons. In the observed batch, one PR merged without an automated review verdict, and the only way to diagnose that was to open the Action logs.

For repos that rely on gh-aw reviews for merge confidence, hidden skips create manual audit work and make bot-authored PRs easy to miss.