doc: make the docs friendlier to coding agents (agent docs 1-7) by bosconi · Pull Request #39470 · MaterializeInc/materialize

bosconi · 2026-10-02T04:16:40Z

Motivation

Coding agents read materialize.com/docs through the Markdown tree at /docs/markdown-docs/ and through /docs/llms.txt, and both have defects that cost agents context or send them to dead links. afdocs 0.22.2, the checker for the Agent-Friendly Documentation Spec, scores materialize.com/docs 74. Comparable docs sites score higher: RisingWave 96, Cloudflare 99.

This PR makes the seven agent-friendly docs changes that can be built and checked locally. Each commit names the change it belongs to:

#	Change	Commits
1	Section pages list their children instead of embedding them	`e30cbf6`
2	Links in the Markdown are absolute	`fe2c7b6`, `e4eb365`, `793a9ea`
3	Docs previews report Markdown sizes, broken and relative links, and unrendered shortcodes; docs lint gates on links	`36ac7d5`, `78b9181`, `1b23cc0`, `bab4efe`, `793a9ea`
4	Pages point agents at `llms.txt` and their Markdown	`20862e0`, `1b9563e`
5	Markdown content cleanup: heading attributes, `<no value>`, two wrong descriptions	`c34306f`
6	The docs sitemap has absolute URLs and is served as `application/xml`	`7e0bf94`, `0db7b05`
7	`/docs/llms.txt` splits into per-section indexes, which also list the per-version release notes	`958cc4a`, `7b75276`

The nightly afdocs report to the #docs Slack channel is now its own PR, #39484, as suggested in review. The preview step no longer runs afdocs: it ran npx in the job that holds $GITHUB_TOKEN and the S3 credentials, and #39484 runs afdocs from a lockfile in a job without secrets.

Three more changes are in this repo but need a design decision or the docs owners, and are not in this PR:

8, Render shortcodes nested in YAML data in the Markdown. 29 Markdown pages, including the MySQL and PostgreSQL ingestion guides and the CREATE TABLE pages, carry literal {{< ... >}} text where the HTML shows content. The .skill.md shortcodes print a data field as written, while the HTML ones render shortcodes nested in it.
9, Split the three Markdown pages over 100,000 characters: releases/, observability/appendix-metrics/ and sql/system-catalog/mz_internal/.
10, Give pages distinct descriptions. Eight descriptions are shared by two or three pages.

Three more need the web team or the CDN owners, and are not in this PR:

11, Markdown at the page URL. Publish each page's Markdown at /docs/<path>/index.md, in ci/deploy_website/website.sh, with its alias redirects, and link that form from llms.txt. It can only be tested by deploying.
12, Content before navigation. Move the sidebar after <main> in baseof.html, and render less of the navigation tree on each page. It needs visual and accessibility review.
13, CloudFront answering Accept: text/markdown. This will let agents such as Claude Code, Cursor and OpenCode get Markdown from any page URL. afdocs ceiling is 97 without this :-) The distribution is materialize-com in i2/website.py.

Description

Each commit message explains its change. Decisions the diff does not show:

The website and the skill bundle are told apart by base URL. The Markdown is built twice from the same templates: by website.sh, with --baseURL /docs, and by bin/gen-claude-skill, with none. Change 1's index links and change 2's link rewrite produce absolute URLs in the website build. In the skill bundle they produce bundle-root paths, which is the form the bundle's existing links use. Change 4's llms.txt pointer is website-only, because the bundle has no llms.txt. The markdown-url partial already keyed off the base URL in the same way.
New partials.
- site-origin is extracted from markdown-url, so changes 2, 4 and 6 share one origin.
- markdown-body post-processes .RenderShortcodes for changes 2, 4 and 5, and markdown-link-target maps each link it finds for change 2.
- markdown-link-url, llms-txt-url, has-markdown and absolute-permalink each do one job, described in their header comments.
The HTML pointer in change 4 uses base-relative URLs. htmltest then checks them against the built files, instead of treating them as external links to the lint build's host.
Change 7 and MaterializeInc/agent-skills#84. That PR has the mz-docs skill read /docs/llms.txt to find pages. It also follows per-section indexes, and was tested against this PR's preview, so the two can land in either order.
Not fixed. The sitemap's two future lastmod dates come from the scheduled v26.45 release notes. One absolute link in the v0.27 release notes, to docs/lts/release-notes/, already returns 404.

Verification

Built with Hugo 0.152.2, the version ci/builder installs, and compared with a build of origin/main.

Measurement	origin/main	This PR
Markdown pages over 100,000 characters	14	3
Markdown pages over 50,000 characters	40	11
`sql/index.md`	750,064 bytes	26,135 bytes
Relative links in the Markdown, root-relative and path-relative	9,651 and 2,148	0 and 0
Rewritten links whose target is not a built file	N/A	1 of 8,244, the 404 link above
Markdown pages with unrendered shortcode syntax	35	29
Absolute sitemap `<loc>` values	0 of 565	565 of 565
`/docs/llms.txt`	85,604 characters	1,655 characters, plus 15 section indexes of up to 30,991
Sitemap pages listed in `llms.txt`	488 of 565	564 of 565, all but the docs home
Skill bundle	9,851,663 bytes	5,142,901 bytes

afdocs, scored locally. Both builds were served with Markdown as text/markdown, and scored with --canonical-origin https://materialize.com/docs on the same 49 pages, every tenth page in llms.txt.
- The origin/main build scores 74, the same as production.
- With changes 1 to 6 it scores 81, and with all seven, 83.
- With changes 11 and 13 also simulated in the local server, including llms.txt links of the form /docs/<path>/index.md, it scores 97, and 98 when afdocs chooses its own 50 pages. markdown-link-portability and llms-txt-coverage pass there.
What afdocs still fails. markdown-url-support, content-negotiation and llms-txt-directive-md fail until change 11 or 13 lands. llms-txt-coverage fails until change 11 switches llms.txt to the page-URL form, because afdocs cannot map markdown-docs/<path>/index.md links back to pages. content-start-position and page-size-html need change 12, and page-size-markdown change 9.
HTML. The HTML output is unchanged apart from change 4's pointers and change 5's two descriptions.
The bundle's links are unchanged. Every one still points at the same file under the bundle root, and the bundle differs only by changes 1 and 5.
Lints.
- The hugo and htmltest steps of ci/test/lint-docs.sh pass locally.
- check-copyright, check-whitespace, ruff 0.16.5 and pyright 1.1.407 --warnings pass on the changed files.
Change 3 ran in CI. In build 137194, the preview's Markdown check found 0 broken and 0 relative links, as it did locally.
Link regression gate. lint-docs.sh now runs the Markdown checker in link-only failure mode. Four new fixture tests cover broken Markdown links, relative links, doubled /docs/ prefixes, and warning-only size/shortcode findings. On local production and preview builds, the three link counts were 0; the existing 3 oversized pages and 29 pages with raw shortcodes remained warnings.

🤖 Generated with Claude Code

Agent-friendly docs change 1. list.skill.md appended the full content of every page below a section to the section's own Markdown page, so section pages such as sql/index.md reached several hundred KB. Each of those pages is already published as its own file. List the pages below a section instead, with their titles, links and descriptions. In the website's Markdown tree, pages over 100,000 characters go from 14 to 3 and pages over 50,000 from 40 to 11; sql/index.md goes from 750,064 to 26,135 bytes. The remaining large pages are large in their own content. Links in the list are absolute URLs to the child's Markdown on the website, and bundle-root paths in the bin/gen-claude-skill bundle, via a new markdown-link-url partial. markdown-url now also works from the Markdown pass, where RelPermalink already ends in index.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 2. Source links are written relative to the docs root, such as /sql/create-cluster/. The HTML pass resolves them in _markup/render-link.html, but .RenderShortcodes leaves Markdown links as written, and the website serves the Markdown under /docs/markdown-docs/, where they do not resolve: 139 of 150 sampled targets returned 404. A new markdown-body partial rewrites each root-relative link in the website build: a link to a page becomes the absolute URL of the page's Markdown, keeping its #fragment, and any other link, such as an image, becomes an absolute URL under /docs. In a local build, all 5,927 rewritten links resolve to a built file, and no root-relative links remain. The bin/gen-claude-skill bundle is byte-identical, because its links name directories under the bundle root and stay as written. Extract the origin logic from markdown-url into a site-origin partial so both share it, and build the preview's Markdown with --environment preview so its links point at preview.materialize.com. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 3. After the docs preview deploys, report: - ci/test/check-docs-markdown.py, a new check of the built Markdown: pages over the Agent-Friendly Documentation Spec's 50,000 and 100,000 character thresholds, links to Markdown pages that name no built file, and root-relative links, which do not resolve away from the site. - An afdocs (https://afdocs.dev) 0.22.2 scorecard of the preview, over every tenth page in llms.txt. afdocs cannot map llms.txt's markdown-docs links back to pages on its own. Neither fails the build. The preview status is posted first, so the checks do not delay the preview link. Against production, the afdocs command scores materialize.com/docs 74. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 4. The only per-page pointer to a page's Markdown was the "View as Markdown" button, 82% of the way through the raw HTML, and no page pointed at llms.txt. afdocs found no llms.txt directive on any of 50 sampled pages. - HTML pages get a visually hidden pointer to llms.txt and the page's Markdown at the top of <body>, before the navigation. It is aria-hidden and its links take no focus, so it is invisible on screen and to assistive technology. - HTML pages get <link rel="alternate" type="text/markdown"> in <head>. - Markdown pages get a blockquote pointer to llms.txt after their title, as the Agent-Friendly Documentation Spec recommends. The skill bundle is unchanged, because it has no llms.txt. The 404 page and the self-managed-docs builds, which have no Markdown, get neither HTML pointer. The HTML links are base-relative, so the docs lint checks them against the built files; it passes locally. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 5. - Drop Hugo heading attributes, such as "## Object Introspection (SHOW) { #show }", from the Markdown in both builds. 193 headings carried one. - include-example.skill.md printed "<no value>" before the code block of an example with no description, 29 times across the Markdown. Print the description only when it exists. - Fix two page descriptions copied from other pages. Descriptions are what an agent reads in llms.txt to choose a page: - Reaction Time, Freshness, and Query Latency said "Learn about indexes in Materialize." - bytea type said "Expresses a Unicode string", the text type's description. Eight other descriptions are shared by two or three pages, such as the Cloud and self-managed versions of a page. Those are left for the docs owners. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 6. Every <loc> in /docs/sitemap.xml was a bare path such as /docs/sql/create-source/, because Hugo's built-in sitemap template emits .Permalink, which has no host under --baseURL /docs. The sitemap protocol requires full URLs, and afdocs found no sitemap URLs under /docs to check llms.txt coverage against. Override the built-in template, copied from Hugo v0.152.2, with an absolute-permalink partial that takes the scheme and host from site-origin. A local build lists the same 565 URLs, now absolute. The sitemap's two future lastmod dates, on the docs home and the releases page, come from the scheduled v26.45 release notes and are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 7. /docs/llms.txt listed every page and had grown to 85,604 characters, over the Agent-Friendly Documentation Spec's 50,000, and afdocs warned on llms-txt-size. The spec recommends nested llms.txt files for large sites. /docs/llms.txt now lists the 15 section indexes, at 1,624 characters, and each section's pages move to /docs/<section>/llms.txt, published from the same template with resources.FromString. The largest, for sql, is 31,071 characters. The same 488 page entries are listed as before. The preview's afdocs sample now reads the section indexes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

MaterializeInc/materialize#39470 splits /docs/llms.txt into a root index of per-section llms.txt files, which list the pages. Tell the agent to follow a section index when llms.txt lists those instead of pages, so the skill works before and after that change deploys. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

linear-code · 2026-10-02T05:17:08Z

EDU-101

def- · 2026-10-02T06:08:49Z

QA LLM Review

1. MEDIUM -- llms.txt pointer is glued onto the H1 or description line of 507 of 568 Markdown pages

doc/user/layouts/partials/markdown-body.html:34

The pointer is supposed to be a blockquote after the title, but on every single page it lands on the same line as whatever precedes it. On 86 pages that is the H1 itself, so the page title an agent reads becomes # ROLLBACK> For the complete documentation index, see [llms.txt](https://materialize.com/docs/llms.txt). On 421 more it is glued onto the description line and is not a blockquote at all.

Details

single.skill.md trims all whitespace between the description (or the title, when a page has no description) and the body: {{- end -}} followed by {{- partial "markdown-body.html" ... -}}. That used to be harmless because .RenderShortcodes output starts with the newline after the front matter. markdown-body.html now puts > For the complete documentation index... in front of that newline, so nothing separates it from the line before. In a website build, sql/rollback/index.md, export-data/sink-troubleshooting/index.md and the older releases/v0.* pages show the broken H1. sql/create-cluster/index.md shows the glued description: `CREATE CLUSTER` creates a new cluster.> For the complete documentation index, .... Only the 61 list pages come out right, because list.skill.md has blank lines around the partial call. check-docs-markdown.py checks only link targets and sizes, so it does not catch this.

Fix: start the pointer with a blank line, for example printf "\n\n> For the complete documentation index, see [llms.txt](%s).\n\n%s". With that change, all 568 pages get the pointer at the start of its own line and the H1s come out intact. The skill bundle does not change, because it gets no pointer.

antiguru

Posted by Claude Code on behalf of @antiguru.

I built this branch and its merge-base with --baseURL /docs, both the HTML and the Markdown pass. Several claims hold: the sitemap has 565 of 565 absolute <loc> values, every one of the 569 rel="alternate" type="text/markdown" targets exists, the checker reports 0 broken and 0 relative links, and 16 llms.txt files are published. The Markdown build goes from 0.33 s to 0.47 s, so the regex rewrite costs nothing that matters. zizmor 1.30.1 --pedantic reports nothing on the new workflow.

Blocking

afdocs-report.py runs npx --yes afdocs@0.22.2 with secrets in its environment. The script pins afdocs, but npm resolves its transitive dependencies fresh on each run, without a lockfile. In afdocs_report.yml, SLACK_TOKEN is set on the same step, and the npx subprocess inherits it. In preview-docs.sh, npx runs in the job that holds $GITHUB_TOKEN for the status curl and the S3 credentials for hugo deploy, on every docs PR. A compromised dependency could read all of these. Split the workflow into a scoring step without secrets that writes JSON and a posting step with SLACK_TOKEN, and run npx with a scrubbed environment in both places (env -i PATH="$PATH" HOME="$HOME" npx ...).

Strong suggestions

The fence tracking in check-docs-markdown.py miscounts. prose_lines toggles on any line whose stripped text starts with three backticks. Six generated pages have unbalanced fences, caused by existing generator defects: glued fences such as ``````hcl and ``````nofmt in sql/create-table/kafka, sql/create-table/sql-server and sql/functions/normalize, and fences joined to prose (connection: ```mzsql) in ingest-data/postgres/amazon-aurora, developer-tools/mz-debug and transform-data/dataflow-troubleshooting. After the first broken fence, the checker treats prose as code. With fence filtering off, 31 pages carry raw shortcodes, not 29. The two it misses are ingest-data/postgres/amazon-aurora (raw {{< tabs >}} and {{% include-example %}}) and sql/create-table/index.md (```{{< include-md). The checker should report unbalanced or glued fences as a defect of their own, because they also break the Markdown for agents.
Nothing prevents a regression. The checks run only in the preview step, and their failures are swallowed. lint-docs.sh already builds the Markdown tree, and htmltest does not read .md files. Broken and relative links are now at 0, so lint-docs.sh could fail on them and leave size and shortcodes as warnings. Without that gate, the drop from 9,651 relative links to 0 can come back unnoticed.
Split the PR. The seven template changes, a new checker, and a Slack-posting workflow that uses a secret belong to different reviewers. The nightly workflow and afdocs-report.py would fit in a separate PR, which is also where the secret handling above gets reviewed.

Nits

markdown-link-target.html strips /docs/-prefixed data-file links through printf "%s/" $basePath, which works only when the base path is /docs. In the preview, where it is /materialize/$PR, /docs/sql/... becomes preview.../materialize/$PR/docs/sql/..., and the checker does not count it because the URL is outside markdown-docs/. Stripping a literal /docs/ would fix this.
The partials decide "website build or not" in three ways: ne Path "/" in markdown-link-url, a non-empty trimmed path in markdown-body, and a substring match on "self-managed" in has-markdown and index.llmstxt.txt. One partial per decision would keep them from drifting apart.
Root-level pages go to about/llms.txt through .key | default (urlize .title), but no about section exists, and the built docs/about/ contains only llms.txt.
In post(), the dict[str, str] annotation forces "unfurl_links": "false" to be a string, while Slack documents a boolean. The afdocs check messages reach Slack mrkdwn without escaping &, < and >.
The link rewrite in markdown-body and the heading-attribute strip also apply inside fenced code. I found no affected output, but nothing documents or enforces this.
sample_urls: when llms.txt has no links, all([]) is true, and afdocs is called with --urls "".
sitemap.xml copies the built-in template of Hugo 0.152.2. The header names the version, so check the copy on each Hugo bump.

Once the secret handling is fixed, the template changes (1 to 7) are a clear improvement and could land, preferably without the nightly workflow.

maheshwarip · 2026-10-02T14:32:16Z

I'm in favor, although I'm OOO in a few hrs and won't be able to review in detail. As long as this doesn't affect SEO (i'll let @nisarhassan12 confirm) let's ship it!

Agent-friendly docs change 2, continued. The first pass rewrote inline links that start with "/", and left three kinds of link unresolvable once the Markdown is read away from the site: - path-relative links such as ../create-role, 1,103 in the Markdown; - reference definitions such as "[`SELECT`]: /sql/select/"; - href and src attributes in the HTML some shortcodes emit, such as href="/docs/sql/drop-source/". With Markdown at the page URL simulated locally, afdocs failed markdown-link-portability on 13 to 15 of 50 pages for these. Move the URL mapping into a markdown-link-target partial and apply it to all three forms. A path-relative link resolves against the page's directory, as it does from the HTML page. A reference definition must start with "/" or "." and end its line or precede a title, so that prose such as "[Note]: Materialize ..." is not mistaken for one. In a local build, no relative links remain, and every rewritten target names a built file except one: an absolute link in the v0.27 release notes to docs/lts/release-notes/, which already returns 404. The skill bundle and the HTML are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 3, continued. check-docs-markdown.py counted inline links that start with "/", so it reported no problem links while afdocs failed path-relative ones. Read links from inline links, reference definitions, and href and src attributes, and count root-relative and path-relative links separately. A relative link now makes the check report a problem, as a broken one does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 7, continued. llms.txt left out the per-version release notes to keep the single index small, but the sitemap lists the 77 that still render, so afdocs' llms-txt-coverage found 488 of 565 sitemap pages, under its 95% pass line. With the index split by section, the notes cost 78 lines in releases/llms.txt, 7,386 characters in all. List them like any other page. llms.txt now covers 564 of the 565 sitemap pages, all but the docs home, and every listed page's Markdown exists in a local build. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 6, continued. /docs/sitemap.xml was served as application/rss+xml, on the website and on PR previews alike. hugo deploy sets a file's Content-Type from the first Hugo media type with its suffix, and for ".xml" that is RSS. Add a deployment matcher that sets application/xml for sitemap.xml. The preview's deploy config keeps the matcher, so a docs preview shows the result. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 4, fixed. single.skill.md writes a page's description just before its body without a newline, so on 507 of the 569 Markdown pages the llms.txt pointer joined the description's line, as in "...to Materialize> For the complete documentation index...", and was not a blockquote. Start the pointer with a blank line. In a local build it is now on its own line on 568 pages; the docs home, whose Markdown is index.skill.md, has none. The skill bundle is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 3, continued. Shortcodes nested inside YAML data, such as the include-example and tabs calls in a data file's description, reach the Markdown as literal "{{< ... >}}" text, while the HTML renders them. The skill shortcodes print a data field as written, and the HTML ones pass it through markdownify, which renders nested shortcodes. 29 Markdown pages carry such syntax, including the MySQL and PostgreSQL ingestion guides and the CREATE TABLE pages. Report those pages, outside fenced code blocks, so the count shows on every docs preview until the gap is closed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Agent-friendly docs change 3, revised after review. The preview step ran `npx --yes afdocs@0.22.2` in the job that holds $GITHUB_TOKEN for the status update and the AWS credentials for `hugo deploy`, on every docs PR. npm resolves afdocs' dependencies fresh on each run, so a compromised dependency could read those credentials. Scrubbing the environment would not cover credentials an agent can reach without it, such as an instance profile. Drop the afdocs run. The step keeps check-docs-markdown.py, which uses the Python standard library. The nightly afdocs report on production, with afdocs pinned by a lockfile and run without secrets, is in #39484. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

bosconi · 2026-10-02T17:00:35Z

@maheshwarip in parallel to waiting for confirmation from @nisarhassan12 I will address the other review feedback, including moving the nightly check out into another PR.

nisarhassan12

Approving. The sitemap fix is the big SEO win here: the live /docs/sitemap.xml has relative <loc>s and is served as application/rss+xml, so Google most likely ignores it today. In my local build of this branch, all 565 URLs are full addresses and none of them is noindex.

Two non-blocking notes for follow-ups:

1. Canonical and og:url are relative. In head.html, both print .Permalink, which with --baseURL /docs is a path like /docs/sql/create-source/. Google handles a relative canonical fine, and this predates the PR. og:url is supposed to be absolute, though, and some share scrapers mishandle a path. absolute-permalink.html from this PR would fix both with one line each.

2. Other search engines can index the Markdown copies. Live robots.txt has Disallow: /docs/markdown-docs/ for Googlebot only, and Allow: / for everyone else. That means Bing and other engines can index the .md files as duplicates of the HTML pages, and the new rel="alternate" link and the agent pointer make them easier to find.

A CloudFront X-Robots-Tag: noindex header on /docs/markdown-docs/* would keep other engines from indexing them. It wouldn't change anything for Google while the Disallow is in place, because Googlebot never fetches the files.
While Googlebot is blocked, links to those URLs can still produce bare, URL-only Google results. The only way to prevent that is to remove the Disallow and let the noindex header keep the files out of the index.
/docs/llms.txt and /docs/<section>/llms.txt aren't covered by the Disallow, so Google can index them. Worth deciding whether we want that.

MaterializeInc/materialize#39470 splits /docs/llms.txt into a root index of per-section llms.txt files, which list the pages. Tell the agent to follow a section index when llms.txt lists those instead of pages, so the skill works before and after that change deploys. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

maheshwarip · 2026-10-14T17:37:28Z

@bosconi how are we looking on this? Can we aim to merge it in this week?

bosconi · 2026-10-14T18:49:15Z

@pm yes, I'll get it merged and send out an update when I do. When I do, I'll also send an update on our Anthropic plugin directory submission (based on a branch that already has the changes). Currently on its 7th day waiting to be reviewed. Maybe we should call in a favor from Parker.

def- · 2026-10-15T18:07:09Z

QA LLM Review

1. MEDIUM -- Preview Markdown now has 560 links with a doubled base path, and the new link gate reports 0

doc/user/layouts/partials/markdown-link-target.html:29

The new trim strips a literal /docs/ and no longer strips the build's actual base path. In a preview, shortcode HTML links that relURL has already prefixed (/materialize/<PR>/sql/types/numeric) now become https://preview.materialize.com/materialize/<PR>/materialize/<PR>/sql/types/numeric, which returns 404. Before this commit they pointed at the page's index.md. That gives 560 broken links on 72 preview pages, and neither the lint gate nor the preview report catches them.

Details

The bug is live on the deployed preview: sql/functions/now_and_mz_now/index.md links numeric to .../materialize/39470/materialize/39470/sql/types/numeric. Other affected pages include export-data/index.md, releases/index.md and most of self-managed-deployments/. Neither build at the old head contained a /docs/-prefixed link, and no source under content/, data/ or layouts/ writes one, so the commit fixes nothing that exists today. Production is unaffected because its base path is /docs, where the two trims agree.

The new gate cannot see this. lint-docs.sh builds with base path /docs, where the old and new heads produce byte-identical Markdown. doubled_prefix only matches <root>docs/, but here the repeated segment is the base path. The checker exits 0 on the broken preview tree, with or without --fail-on-links.

Strip the base path as before, and also /docs/. With this change the preview's links match the old head again:

-    {{- $path = strings.TrimPrefix "/docs/" $path | strings.TrimPrefix "/" | printf "/%s" -}}
+    {{- $path = strings.TrimPrefix (printf "%s/" $basePath) $path | strings.TrimPrefix "/docs/" | strings.TrimPrefix "/" | printf "/%s" -}}

To catch this in the checker, flag links whose path repeats the base path itself, for example <root> followed by urlparse(root).path.lstrip("/").

…w base path 793a9ea stripped only the literal /docs/ prefix from root-relative links. In a preview, shortcode links already carry the base path from relURL, so they came out as /materialize/<PR>/materialize/<PR>/... and 404ed: 560 links on 72 pages. Strip the base path and then /docs/, which restores the preview output of bab4efe exactly and leaves the production output byte-identical. The checker now flags a link that repeats the base path itself, not only a repeated docs/ segment. lint-docs.sh also builds the Markdown with a base path other than /docs and gates it, since under /docs the two strips agree and the regression is invisible. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

bosconi · 2026-10-16T23:53:44Z

Thanks, all. 793a9ea and 42d11e2 are pushed with the remaining link fixes. CI is green, so merging.

Review items handled in follow-up PRs

doc: make canonical URLs absolute and list root pages in llms.txt #39913: absolute canonical and og:url (@nisarhassan12), and one partial per build decision, root pages in the root llms.txt, and a Hugo-bump reminder for the copied sitemap template (@antiguru).
doc: close code fences in the Markdown docs and gate on broken fences #39914: fence tracking in the checker, fixes to the generators that glue fences (47 defects on 15 pages, now 0), and the link rewrite and heading strip skipping fenced code (@antiguru).

None of these changes what this PR publishes: the relative canonical and og:url were already on main, the broken fences come from existing generators, and the link rewrite skipping fenced code changes no current page. You marked them non-blocking, so I've kept this PR from another round.

@antiguru

npx with secrets (blocking): resolved. The nightly report moved to ci: post a nightly afdocs report on the docs to #docs #39484, which pins afdocs with a lockfile, scores without secrets, and posts from a separate job. The preview step here no longer runs afdocs. ci: post a nightly afdocs report on the docs to #docs #39484 is where I'd like your review of the secret handling.
Split the PR: done, ci: post a nightly afdocs report on the docs to #docs #39484.
/docs/-prefixed links in previews: fixed. markdown-link-target.html strips the base path, then /docs/.
Regression gate: lint-docs.sh runs the checker with --fail-on-links on the production build and on a build under a non-/docs base path. It fails on broken, relative or doubled-prefix links, all at zero now. Oversized pages and unrendered shortcodes stay warnings.
Slack formatting and empty llms.txt: fixed in ci: post a nightly afdocs report on the docs to #docs #39484.
Fences, partials, root llms.txt, Hugo reminder: in the follow-ups above.

@def-

llms.txt pointer glued to the H1: fixed in 1b9563e.
Doubled base path in previews: thanks, fixed in 42d11e2 as you suggested. The preview Markdown matches bab4efe again, production is byte-identical, and the checker now flags a repeated base path.

@nisarhassan12

Canonical and og:url: in doc: make canonical URLs absolute and list root pages in llms.txt #39913, above.
Markdown copies and search: before going further, a question for you and @maheshwarip. In August you scoped the robots.txt Disallow to Googlebot only, so other engines keep the Markdown findable for coding agents, after Pranshu's point on docs: add nofollow to View as Markdown links. #38411. A noindex header would undo that for every engine. Do you still want the copies findable through search, or out of all indexes? I've filed it as WEB-13 with the options.
Drafts in your area: I had my agent draft these in case they're useful. Take them over, change them, or close them, or give me feedback and I'll get them merged, whichever you prefer.
- MaterializeInc/i2#4261: the noindex header, with the options above in its description.
- doc: publish each page's Markdown at its page URL #39915: each page's Markdown at its page URL (EDU-101 item 11), in website.sh. It waits on the search question.
- MaterializeInc/i2#4263: answer Accept: text/markdown on docs pages (item 13), with @jubrad for the CloudFront side.

@maheshwarip

The SEO review is approved, and the link fixes and CI gate are in. Other EDU-101 follow-ups: doc: give every page a distinct description #39912 (distinct page descriptions, a wording proposal for the docs team), ci: post a nightly afdocs report on the docs to #docs #39484 (nightly score in #docs), and the search question above (WEB-13).
The rest need docs and web owners, so I've filed them under EDU-101: EDU-104 (render shortcodes nested in data files in the Markdown), EDU-105 (split the three Markdown pages over 100,000 characters), and WEB-12 (page content before navigation). I'm happy to implement EDU-104 if you want; it needs your call on the approach.

Linear: [EDU-101](https://linear.app/materializeinc/issue/EDU-101/make-the-materialize-docs-agent-friendly) ### Motivation [afdocs](https://afdocs.dev) checks a docs site against the [Agent-Friendly Documentation Spec](https://agentdocsspec.com/spec/web/). materialize.com/docs scores 74 today, and [#39470](#39470) makes the template changes that raise it. This workflow reports the score to #docs every night, so the docs team sees it move and sees regressions. It was part of #39470, and is split out here so that its secret handling gets its own review, as suggested there. ### Description The "Agent-friendly docs report" workflow runs every day at 06:00 UTC / 02:00 EDT. It scores materialize.com/docs with afdocs on every tenth page listed in `llms.txt`, and posts to #docs. - **Top-level message:** always one line, the same shape at any score, differing only in its emoji. For example: ":x: The docs score 74 / 100 for agent-friendliness on afdocs, over 49 pages: 9 of 28 checks fail or warn. Workflow run." At 100 the emoji is :white_check_mark:. - **Thread reply:** the checks that fail or warn, one bullet each, then the full report with every check and the category scores. A run at 100 gets the reply too, so every run has the same shape and the full report. - **Dry run:** a manual run is a dry run by default. It prints the message and the reply in the log of the "Score the docs" step, and skips the `post` job. Pull requests that change the workflow, `ci/test/afdocs-report.py` or `ci/test/afdocs/` always dry-run. afdocs and its npm dependencies never run where a secret is available: - **Pinned dependency tree.** `ci/test/afdocs/package.json` and `package-lock.json` pin afdocs 0.22.2 and its 64 dependencies by integrity hash. The workflow installs them with `npm ci --ignore-scripts`, so no install script runs and npm resolves nothing fresh. - **Scoring without secrets.** The `score` job holds no secrets, and checks out with `persist-credentials: false`. `afdocs-report.py score` runs afdocs with only `PATH` and `HOME` in its environment. It writes the afdocs JSON, the message and the thread reply, which the job uploads as an artifact. - **Posting on a fresh runner.** The `post` job downloads the artifact and posts it with `SLACK_TOKEN`. It runs only `afdocs-report.py post`, which uses the Python standard library, so no npm code runs on that runner. **Slack token:** the workflow uses the repo's existing `SLACK_TOKEN` secret, the one the `slack_notify_*` workflows use, so no new secret is needed if that bot is a member of #docs. The report goes in a thread reply rather than an attached file, because uploading a file needs the `files:write` scope, and `SLACK_TOKEN`'s scopes are unknown. `afdocs-report.py` reads `llms.txt` whether it lists the pages itself, as on production today, or links one `llms.txt` per section, as after #39470. It exits with an error when `llms.txt` lists no pages, rather than calling afdocs with no URLs. It escapes `&`, `<` and `>` in afdocs' messages for Slack, and sends `unfurl_links` as a boolean. ### Verification - **Dry run against production:** `afdocs-report.py score https://materialize.com/docs` scored 74 over 49 pages, with 9 of 28 checks failing or warning, and wrote the JSON, the message and the reply. - **Message at 100:** checked by substituting the score in the saved report; only the emoji and the counts change. - **Scrubbed environment:** with `SLACK_TOKEN`, `GITHUB_TOKEN` and `AWS_SECRET_ACCESS_KEY` set, a stand-in afdocs binary saw only `HOME`, `PATH` and the variables the shell sets itself. - **Posting:** `post` was run against a stubbed Slack API. It sent the message, then the reply with the message's `thread_ts`, with `unfurl_links` as `false`. - **Lints:** ruff 0.16.5, pyright 1.1.407 `--warnings`, zizmor `--persona=pedantic` with the repo's config, `check-copyright` and `check-whitespace` pass. - **This PR's own run:** the workflow runs on this PR as a dry run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…9913) Follow-ups promised in review of #39470. Part of: [EDU-101](https://linear.app/materializeinc/issue/EDU-101) ### What changed - **Canonical and `og:url` are absolute.** `head.html` printed `.Permalink` for both, which under `--baseURL /docs` is a path such as `/docs/sql/create-source/`. `og:url` must be an absolute URL. Both now use the `absolute-permalink` partial that the sitemap already uses. - **Root-level pages are listed in the root `llms.txt`.** License and Support have no section, so they were published in `about/llms.txt`, a directory with nothing else in it. The root index now lists them under their sidebar header ("About"), and `about/llms.txt` is no longer generated. The root index's orienting note says that pages outside any section follow the section list. - **One partial per "which build is this" decision.** The templates make two different decisions, and checked them three ways: - Is this Markdown published on the website, so that links in it must be absolute URLs, or is it the `bin/gen-claude-skill` bundle, where root-relative links name bundle directories? `markdown-link-url` checked `ne Path "/"`, and `markdown-body` checked for a non-empty trimmed path. `markdown-links-absolute.html` now owns this. The two checks only disagreed for a host-only `--baseURL`, which no build uses. - Does the website publish a Markdown tree for this build at all? `self-managed-docs/*` branches build the HTML alone. `has-markdown` and `index.llmstxt.txt` each matched "self-managed" in the base path. `site-has-markdown.html` now owns this, and `has-markdown` calls it. - **A comment next to the Hugo pin** in `ci/builder/Dockerfile` says to compare `doc/user/layouts/sitemap.xml` with the built-in template on each Hugo bump. This changes the ci-builder image hash, so CI rebuilds the builder image for this PR. ### Verification Built with Hugo 0.152.2 before and after: the production HTML and Markdown passes (`--baseURL /docs`), the skill bundle (no `--baseURL`, as `bin/gen-claude-skill` runs it), the preview passes, the `lint-docs.sh` passes with a host in the base URL, and a self-managed HTML build. Each build was byte-identical across two runs of the same commit, so every difference below comes from this PR. - **Partials refactor alone** (first commit): all six builds byte-identical to the base branch. - **Whole PR:** - Every Markdown tree, production, preview, lint and skill bundle, is byte-identical. - In the production, preview and self-managed HTML, the only changed lines are `og:url` and `canonical`, on all 570 pages, and all 570 of each are now absolute. Production uses `https://materialize.com/docs/...`, self-managed `https://materialize.com/docs/self-managed/<version>/...`, preview `https://preview.materialize.com/materialize/<PR>/...`. The lint build's HTML is unchanged, because its base URL already has a host. - `llms.txt` changes as described above in the production, preview and lint builds. The self-managed build still publishes none. - `check-docs-markdown.py --fail-on-links` passes on the production Markdown, and `test_check_docs_markdown.py` passes. Not changed: the 542 alias pages, rendered by `layouts/alias.html`, keep a path-only canonical. In production, `website.sh` replaces each with an S3 redirect, so their HTML is never served. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

bosconi and others added 7 commits October 2, 2026 00:01

bosconi mentioned this pull request Oct 2, 2026

Fetch mz-docs live from materialize.com instead of bundling a copy MaterializeInc/agent-skills#84

Open

4 of 6 tasks

bosconi marked this pull request as ready for review October 2, 2026 05:05

bosconi requested review from a team as code owners October 2, 2026 05:05

bosconi requested review from antiguru, bobbyiliev, maheshwarip and sjwiesman October 2, 2026 05:17

antiguru reviewed Oct 2, 2026

View reviewed changes

maheshwarip requested a review from nisarhassan12 October 2, 2026 14:32

bosconi mentioned this pull request Oct 2, 2026

ci: post a nightly afdocs report on the docs to #docs #39484

Merged

bosconi and others added 7 commits October 2, 2026 11:09

bosconi force-pushed the jc/mz-docs-optimization branch from 63cb0a7 to bab4efe Compare October 2, 2026 15:10

nisarhassan12 approved these changes Oct 2, 2026

View reviewed changes

doc: fix prefixed Markdown links and gate link regressions

793a9ea

This was referenced Oct 16, 2026

doc: make canonical URLs absolute and list root pages in llms.txt #39913

Merged

doc: close code fences in the Markdown docs and gate on broken fences #39914

Open

doc: publish each page's Markdown at its page URL #39915

Draft

bosconi merged commit 5f2ed97 into main Oct 16, 2026
84 checks passed

bosconi deleted the jc/mz-docs-optimization branch October 16, 2026 23:54

bosconi mentioned this pull request Oct 16, 2026

doc: give every page a distinct description #39912

Open

Navigation Menu

Repository navigation

doc: make the docs friendlier to coding agents (agent docs 1-7) - #39470

doc: make the docs friendlier to coding agents (agent docs 1-7)#39470
bosconi merged 16 commits into
mainfrom
jc/mz-docs-optimization

bosconi commented Oct 2, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Oct 2, 2026

Uh oh!

def- commented Oct 2, 2026

Uh oh!

antiguru left a comment

Uh oh!

maheshwarip commented Oct 2, 2026

Uh oh!

bosconi commented Oct 2, 2026

Uh oh!

nisarhassan12 left a comment

Uh oh!

maheshwarip commented Oct 14, 2026

Uh oh!

bosconi commented Oct 14, 2026

Uh oh!

def- commented Oct 15, 2026

Uh oh!

bosconi commented Oct 16, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

bosconi commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Verification

Uh oh!

linear-code Bot commented Oct 2, 2026

Uh oh!

def- commented Oct 2, 2026

QA LLM Review

1. MEDIUM -- llms.txt pointer is glued onto the H1 or description line of 507 of 568 Markdown pages

Uh oh!

antiguru left a comment

Choose a reason for hiding this comment

Blocking

Strong suggestions

Nits

Uh oh!

maheshwarip commented Oct 2, 2026

Uh oh!

bosconi commented Oct 2, 2026

Uh oh!

nisarhassan12 left a comment

Choose a reason for hiding this comment

Uh oh!

maheshwarip commented Oct 14, 2026

Uh oh!

bosconi commented Oct 14, 2026

Uh oh!

def- commented Oct 15, 2026

QA LLM Review

1. MEDIUM -- Preview Markdown now has 560 links with a doubled base path, and the new link gate reports 0

Uh oh!

bosconi commented Oct 16, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bosconi commented Oct 2, 2026 •

edited

Loading