[bosconi]

Linear: EDU-101

Motivation

afdocs checks a docs site against the Agent-Friendly Documentation Spec. materialize.com/docs scores 74 today, and #39470 makes the template changes that raise it. This workflow reports the score to #docs every night, so the docs team sees it move and sees regressions.

It was part of #39470, and is split out here so that its secret handling gets its own review, as suggested there.

Description

The "Agent-friendly docs report" workflow runs every day at 06:00 UTC / 02:00 EDT. It scores materialize.com/docs with afdocs on every tenth page listed in llms.txt, and posts to #docs.

• Top-level message: always one line, the same shape at any score, differing only in its emoji. For example: ":x: The docs score 74 / 100 for agent-friendliness on afdocs, over 49 pages: 9 of 28 checks fail or warn. Workflow run." At 100 the emoji is :white_check_mark:.
• Thread reply: the checks that fail or warn, one bullet each, then the full report with every check and the category scores. A run at 100 gets the reply too, so every run has the same shape and the full report.
• Dry run: a manual run is a dry run by default. It prints the message and the reply in the log of the "Score the docs" step, and skips the post job. Pull requests that change the workflow, ci/test/afdocs-report.py or ci/test/afdocs/ always dry-run.

afdocs and its npm dependencies never run where a secret is available:

• Pinned dependency tree. ci/test/afdocs/package.json and package-lock.json pin afdocs 0.22.2 and its 64 dependencies by integrity hash. The workflow installs them with npm ci --ignore-scripts, so no install script runs and npm resolves nothing fresh.
• Scoring without secrets. The score job holds no secrets, and checks out with persist-credentials: false. afdocs-report.py score runs afdocs with only PATH and HOME in its environment. It writes the afdocs JSON, the message and the thread reply, which the job uploads as an artifact.
• Posting on a fresh runner. The post job downloads the artifact and posts it with SLACK_TOKEN. It runs only afdocs-report.py post, which uses the Python standard library, so no npm code runs on that runner.

Slack token: the workflow uses the repo's existing SLACK_TOKEN secret, the one the slack_notify_* workflows use, so no new secret is needed if that bot is a member of #docs. The report goes in a thread reply rather than an attached file, because uploading a file needs the files:write scope, and SLACK_TOKEN's scopes are unknown.

afdocs-report.py reads llms.txt whether it lists the pages itself, as on production today, or links one llms.txt per section, as after #39470. It exits with an error when llms.txt lists no pages, rather than calling afdocs with no URLs. It escapes &, < and > in afdocs' messages for Slack, and sends unfurl_links as a boolean.

Verification

• Dry run against production: afdocs-report.py score https://materialize.com/docs scored 74 over 49 pages, with 9 of 28 checks failing or warning, and wrote the JSON, the message and the reply.
• Message at 100: checked by substituting the score in the saved report; only the emoji and the counts change.
• Scrubbed environment: with SLACK_TOKEN, GITHUB_TOKEN and AWS_SECRET_ACCESS_KEY set, a stand-in afdocs binary saw only HOME, PATH and the variables the shell sets itself.
• Posting: post was run against a stubbed Slack API. It sent the message, then the reply with the message's thread_ts, with unfurl_links as false.
• Lints: ruff 0.16.5, pyright 1.1.407 --warnings, zizmor --persona=pedantic with the repo's config, check-copyright and check-whitespace pass.
• This PR's own run: the workflow runs on this PR as a dry run.

🤖 Generated with Claude Code

[linear-code]

EDU-101

[antiguru]

Thank you!