[bosconi]

This is a draft because it must not merge until WEB-13 decides whether the docs' Markdown copies should be findable through search, and the matching search-indexing change is live.

EDU-101 item 11: serve each docs page's Markdown at its page URL, /docs/<path>/index.md, next to its index.html, and link that form from llms.txt.

Stacked on #39913; retarget to main after it merges.

Part of: EDU-101

Note

Drafted by a coding agent, in case it's useful. website.sh and the docs deploy are @nisarhassan12's area: take it over, change it, or close it, whichever you prefer.

Motivation

Today a page's Markdown is only at /docs/markdown-docs/<path>/index.md, so a coding agent that has a page URL cannot get its Markdown from it, and /docs/sql/select/index.md returns 404. The afdocs measurements in #39470 found that markdown-url-support fails until this lands, and that llms-txt-coverage fails until llms.txt links the page-URL form, because afdocs cannot map markdown-docs/ links back to pages.

What changed

• ci/deploy_website/publish-markdown-at-page-urls.sh copies each markdown-docs/<path>/index.md of a built site to <path>/index.md. website.sh and preview-docs.sh run it after both Hugo passes and before hugo deploy, so the copies are in public/ and hugo deploy uploads them instead of deleting them. lint-docs.sh runs it too, so a docs PR fails if a copy would overwrite a built file.
• The root and section llms.txt link <baseURL>/<path>/index.md, through a new atPage argument of the markdown-url partial. The note in each section index now says to drop the trailing index.md for the HTML page.
• /docs/markdown-docs/ is unchanged, since agents and the mz-docs skill read it today. Links inside the Markdown, the HTML rel="alternate" link and the hidden agent pointer still point there.
• Self-managed branch builds have no Markdown, and are unchanged.

Decisions

• A copy step, not a Hugo output format. Rendering the skill format in the HTML pass would need the Markdown templates to work in a second context, and could drift from markdown-docs/. The copy guarantees the two URLs serve identical bytes.
• Aliases get a copy of their target's Markdown, not a redirect. The alias loop in website.sh turns each alias's index.html into a 301. Doing the same for index.md would double the loop's aws s3 cp calls on every deploy, from 542 to 1,084, so <alias>/index.md serves the target's Markdown with a 200.
• Search indexing is not handled here, and needs @nisarhassan12 and @maheshwarip. In August, www-temporary scoped the /docs/markdown-docs/ Disallow in robots.txt to Googlebot only, on purpose: Google stops spending crawl budget on duplicates, while other engines keep the Markdown crawlable so coding agents can still find it through search (robots.ts). That followed the discussion on docs: add nofollow to View as Markdown links. #38411, where @maheshwarip noted that agents landing on the Markdown from a web search is useful, and @nisarhassan12 agreed. The new /docs/<path>/index.md copies fall outside the Googlebot rule, so Google could crawl them as duplicates of the HTML pages. Two ways to handle that: extend the Googlebot rule in robots.ts to cover them (Google supports wildcards, for example Disallow: /docs/*.md$), which keeps the August policy; or the X-Robots-Tag: noindex header drafted in MaterializeInc/i2#4261, which would also hide the copies from other engines. This PR should land only after that is decided and in place.

Verification

Built the production (--baseURL /docs), preview and lint-docs.sh sites with Hugo 0.152.2, ran the copy script on each, and compared them with builds of #39913:

• Each site has 569 Markdown pages and 569 non-alias HTML pages, the same set of paths. Every <path>/index.md is byte-identical to markdown-docs/<path>/index.md.
• All 542 alias directories got an index.md identical to their target's Markdown, with none skipped. This includes the lint build, where alias targets are full URLs.
• All 578 links in the 15 llms.txt files (root and section indexes) resolve to a built file: 564 page Markdown copies and 14 section indexes. None use the markdown-docs/ form.
• Against doc: make canonical URLs absolute and list root pages in llms.txt #39913, the only changes are the 15 llms.txt files and the 1,111 new index.md files. The markdown-docs/ trees, the HTML, the skill bundle and the self-managed build are byte-identical.
• check-docs-markdown.py --fail-on-links passes on the production and preview Markdown, and test_check_docs_markdown.py passes. A second run of the script on the same tree fails because a copy already exists, which is how a collision with a built file would show up.

Full verification needs a deploy, because only the live site shows what S3 and CloudFront serve. After the deploy:

• curl -sI https://materialize.com/docs/sql/select/index.md returns 200 with content-type: text/markdown, and its body matches https://materialize.com/docs/markdown-docs/sql/select/index.md.
• An alias, https://materialize.com/docs/releases/v0.29.0/index.md, returns 200 with the Markdown of /docs/releases/v0.29/.
• https://materialize.com/docs/markdown-docs/sql/select/index.md still returns 200.
• Every link in https://materialize.com/docs/llms.txt and the section indexes it lists returns 200.

🤖 Generated with Claude Code

[linear-code]

EDU-101