Add bots and agents overview page (/docs/bots)

[ulziibay-kernel]

Summary

• Add /docs/bots — an overview of Kernel's Web Bot Auth identities, modeled after OpenAI's crawlers page. Covers both identities:
  • Kernel Agent (existing signed agent, www.kernel.sh) — user-directed, Intermediary.
  • Kernel Search (new crawler, search.bot.kernel.sh) — KernelSearchBot, Direct, Search purpose.
• Purposes are independent so site owners can allow search while blocking agents (or vice versa).
• Document Kernel Search signing config (Signature-Agent, key directory, --signature-agent CLI flag, host-proxy env) on the existing Web Bot Auth page.
• Add bots to the Bot Anti-Detection nav group.

Notes

• This is the public "Bot documentation URL" for the Cloudflare Verified Bot submission.
• Pairs with website PR kernel/website#255 (serves the search.bot.kernel.sh key directory).

Test plan

• /docs/bots renders with both identities and correct directory URLs.
• Nav shows "Bots and agents" under Bot Anti-Detection.

Made with Cursor

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c11afab. Configure here.}

[cursor]

Duplicate bot identity listing

Low Severity

Section 4 repeats Kernel Search’s User-Agent, Signature-Agent, and key directory URLs in prose bullets even though this PR adds bots.mdx as the canonical identity overview with the same fields in a table, increasing the risk the two pages drift apart.

Additional Locations (1)

• bots.mdx#L20-L31

 

^{Triggered by learned rule: Single source of truth — no deep content duplication across pages}

^{Reviewed by Cursor Bugbot for commit c11afab. Configure here.}

[cursor]

CLI reference missing new flags

Medium Severity

This PR documents required build-web-bot-auth flags --url and --signature-agent in the Web Bot Auth guide, but the canonical reference/cli/extensions.mdx entry still lists only --to, --key, and --upload.

 

^{Triggered by learned rule: Single source of truth — no deep content duplication across pages}

^{Reviewed by Cursor Bugbot for commit c11afab. Configure here.}

[cursor]

Risk assessment: Low

The diff is limited to public documentation: one new MDX page, updates to an existing Web Bot Auth guide, and a docs.json navigation entry (82 additions, 5 deletions). It doesn't modify executable code, runtime configuration, infrastructure, authentication enforcement, or production data. The only meaningful exposure is user-facing operational guidance for Web Bot Auth and Kernel Search, so the blast radius remains limited and the change is straightforward to revert.

I found no CODEOWNERS file or requested reviewers, and this PR had no existing approval. The broken-links check also passed.

  

_{Sent by Cursor Automation: Assign PR reviewers}

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
Kernel	🟢 Ready	View Preview	Jul 21, 2026, 4:47 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.