Fail docs checks without a conclusive outcome

[David Pine (IEvangelist)]

Description

PR Documentation Check could finish green when the agent reported draft_failed and safe outputs created no documentation PR. This adds one authoritative, fail-closed validator job that cross-checks the agent notification against the triggering source PR and the trusted safe-output PR URL.

The validator accepts only a confirmed drafted docs PR or an explicit no-docs-needed result. Missing, malformed, duplicate, mismatched, unsupported, and internally inconsistent outcomes now fail the workflow. The source-comment safe-output job remains responsible only for posting comments and requesting reviewers: it renders warning text for structurally valid but inconsistent outcomes, while malformed or mismatched source identities are logged and intentionally skip posting because their PR association cannot be trusted.

Security considerations

The workflow treats the triggering event's source PR number and the safe-output-created PR URL as trusted boundaries. Agent-provided source identities must match the event before comments are posted, and untrusted validator error data is workflow-command encoded before GitHub Actions annotations are emitted.

This PR is stacked on #19118.

Validation:

• 64 Python workflow tests passed, including direct main(), subprocess exit-code coverage, and a CLI regression proving %, CR, and LF cannot inject a second workflow command or annotation.
• pr-docs-check compiled in strict validation mode with the required gh-aw v0.85.4 compiler and recompilation was byte-for-byte idempotent.
• Recovery run 31158242140 / source PR Improve AI agent observability text #18868 fails validation with draft_failed.
• Successful run 31157424866 / source PR Default to Terminal view for live terminal resources in dashboard console logs #18867 passes with trusted docs PR [docs] Update WithTerminal docs to reflect Terminal-view default for live resources aspire.dev#1447.

Part of #19054

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
    • If yes, did you have an API Review for it?
      • Yes
      • No
    • Did you add <remarks /> and <code /> elements on your triple slash comments?
      • Yes
      • No
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • If yes, have you done a threat model and had a security review?
      • Yes
      • No
  • No

[Copilot]

Pull request overview

Adds fail-closed validation for documentation workflow outcomes.

Changes:

• Validates agent results against the triggering PR and created docs PR.
• Adds unit and CLI exit-code tests.
• Improves invalid-outcome comments and regenerates the workflow.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
.github/workflows/pr-docs-check/validate_outcome.py	Implements outcome validation.
.github/workflows/pr-docs-check/test_validate_outcome.py	Tests validation and exit codes.
.github/workflows/pr-docs-check.md	Adds validation job and comment handling.
.github/workflows/pr-docs-check.lock.yml	Regenerates the compiled workflow.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19155

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19155"

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/pr-docs-check.md:408

• The generic invalid-outcome warning is still bypassed for some duplicate notifications. If the first duplicate reports drafted, the earlier else if (result === 'drafted') runs (even when draftUrl is present) and falsely says the PR could not be confirmed; draft_failed similarly selects its specific branch. Gate those result-specific failure branches on notifications.length === 1 so duplicate outcomes reach this internally-inconsistent warning.

              } else {

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

.github/workflows/pr-docs-check.md:385

• A draft_failed notification can still claim that no docs PR was drafted when created_pr_url is nonempty. That combination is internally inconsistent, so this branch should also require !draftUrl; otherwise the source PR receives the specific “could not be drafted” message instead of the generic invalid-outcome warning. Regenerate the lock workflow after updating the source.

              } else if (notifications.length === 1 && result === 'draft_failed') {

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

.github/workflows/pr-docs-check.md:339

• Invalid or mismatched source_pr_number values return before the comment body is built, so the source PR receives no warning even though the PR description says invalid outcomes render warning text. Since expectedNumber is trusted and is already used as the destination, carry a notification-validity flag into the result branches and let these cases fall through to the warning body instead of returning.

              if (notifications.length === 1) {
                const agentNumber = item.source_pr_number;
                if (!Number.isInteger(agentNumber) || agentNumber <= 0 || agentNumber > 10_000_000) {
                  core.warning(`Invalid source_pr_number from agent: ${item.source_pr_number}; skipping comment.`);
                  return;

.github/workflows/pr-docs-check/validate_outcome.py:74

• When draft_failed is paired with a nonempty trusted PR URL, this reports that no PR was created even though the validator is rejecting an internally inconsistent outcome. Distinguish that case so the failure annotation identifies the actual contradiction.

    if result == "draft_failed":
        raise OutcomeValidationError(
            "Documentation was required, but no docs PR was created."
        )

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[Adam Ratzman (adamint)]

I think the privileged reviewer path and contradictory-result validation above should be fixed before merge. The validator otherwise behaved correctly across the exercised success and failure matrix.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

[github-actions]

Tests selector (audit mode)

The full test matrix and all jobs still run in audit mode. The tests and jobs below are what selective CI would run under enforcement.

1 / 100 test projects · 0 jobs, from 5 changed files.

Selected test projects (1 / 100)

Infrastructure.Tests

Selected jobs (0)

none

---

How these were chosen — grouped by what changed

📄 .github/workflows/pr-docs-check.lock.yml (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/pr-docs-check.md (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/pr-docs-check/test_validate_outcome.py (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/pr-docs-check/validate_outcome.py (changed)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/PrDocsCheckWorkflowTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

Job reasons

none

---

Selection computed for commit becf4d6.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/pr-docs-check/validate_outcome.py:88

• create-pull-request still has fallback-as-issue: true in pr-docs-check.md, so gh-aw can return a fallback issue URL through created_pr_url when PR creation fails. Accepting any nonempty value here would validate result: drafted and render the “Documentation has been drafted” comment even when the URL is /issues/... and no docs PR exists. Disable that fallback and recompile the workflow, or require the trusted URL to identify a microsoft/aspire.dev/pull/<number> before accepting drafted.

    if result == "drafted" and created_pr_url:
        return f"Confirmed drafted documentation PR: {created_pr_url}"

[aspire-repo-bot]

✅ No documentation update needed.

Step 5 branch taken: docs_optional -> build_or_ci_only

Triggered signals: no signals triggered (signal_count = 0; recommendation = docs_optional).

Allowlist justification: all 5 changed files match the build_or_ci_only category - they are entirely within .github/workflows/ (the pr-docs-check GitHub Actions workflow: pr-docs-check.lock.yml, pr-docs-check.md, pr-docs-check/test_validate_outcome.py, pr-docs-check/validate_outcome.py) and tests/Infrastructure.Tests/WorkflowScripts/PrDocsCheckWorkflowTests.cs. This PR hardens the docs-check automation itself (adds a fail-closed validator job so the workflow no longer reports green when no documentation PR was actually created) and does not change any Aspire product surface, CLI, hosting API, dashboard, or integration behavior that ships to microsoft/aspire.dev users.

No docs PR is needed.