feat(fetch): discover agent resources

[chaliy]

What changed

Regular Markdown fetches now discover agent-facing resources advertised through HTTP Link headers, HTML link/meta declarations, and a bounded set of conventional same-origin paths. Results are returned as structured metadata and appended as compact Markdown navigation links.

The conventional probes include llms.txt, llms-full.txt, auth.md, OAuth/OpenID metadata, API catalogs, MCP server cards, A2A agent cards, and Agent Skills indexes.

Public documentation, the behavioral specification, threat model, changelog, and test coverage are included.

Why

Agents receiving fetched content need a safe, predictable way to find related machine-readable documentation, authentication instructions, APIs, and agent endpoints without independently guessing conventions or recursively crawling a site.

Before / After

Before: a Markdown fetch returned only the requested page; advertised and conventional agent resources were not surfaced.

After: the same fetch includes structured agent_resources metadata and, when resources exist, an appendix such as:

## Agent resources

- [llms-txt](https://example.com/llms.txt) — `llms-txt`; verified via probe
- [auth](https://example.com/auth.md) — `auth`; verified via probe

End-to-end CLI smoke test against a local HTTP server confirmed discovery from HTTP headers, HTML declarations, /llms.txt, and /auth.md.

Risk

• Medium
• Markdown GET requests can make up to ten additional lightweight HEAD requests.
• False-positive catch-all responses are reduced through status/content-type validation; redirected probes are not considered verified.
• Discovery remains same-origin, fixed-path, bounded, non-recursive, and uses existing SSRF/DNS/redirect controls.

Checklist

• Unit tests added
• Unit tests passed
• Smoke tests passed
• Documentation updated
• Specs updated and threat model reviewed

Produced by yolop