diff --git a/bots.mdx b/bots.mdx new file mode 100644 index 0000000..49b03d9 --- /dev/null +++ b/bots.mdx @@ -0,0 +1,45 @@ +--- +title: "Bots and agents" +description: "Kernel's bots and agents, their purposes, and how to verify them with Web Bot Auth" +--- + +Kernel identifies its bots and agents with [Web Bot Auth](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture). Each identity serves its own key directory from its own authority, so site owners can allow or block each one **independently by purpose** — for example, allow search indexing while blocking user-directed agents (or vice versa). + +Kernel is listed in [Cloudflare's bots and agents directory](https://radar.cloudflare.com/bots/directory/kernel) and [Vercel's public directory](https://bots.fyi/?query=kernel). + +## Kernel Agent + +User-directed browser automation. Kernel Agent visits pages on behalf of an end user's request; it is not an automatic crawler. Requests are signed with Web Bot Auth rather than a dedicated crawler user-agent token. + +| Field | Value | +| --- | --- | +| Purpose | Agent | +| Operator | Intermediary (end-user directed) | +| Signature-Agent | `https://www.kernel.sh` | +| Key directory | `https://www.kernel.sh/.well-known/http-message-signatures-directory` | + +## Kernel Search + +Crawls pages to build search indexes and retrieval databases. Kernel Search identifies itself with the `KernelSearchBot` user-agent token and follows `robots.txt` directives for that token, including crawl-delay preferences. + +| Field | Value | +| --- | --- | +| Purpose | Search | +| Operator | Direct (Kernel-operated) | +| User-Agent | `KernelSearchBot` | +| Signature-Agent | `https://search.bot.kernel.sh` | +| Key directory | `https://search.bot.kernel.sh/.well-known/http-message-signatures-directory` | + +## Verifying Kernel traffic + +Each identity publishes its public key set (JWKS) at its key directory. To verify a request: + +1. Read the `Signature-Agent` header to determine which Kernel identity signed the request. +2. Fetch the public key set from that identity's key directory and cache it per the `Cache-Control` header. +3. Verify the `Signature` and `Signature-Input` headers per [RFC 9421](https://datatracker.ietf.org/doc/html/rfc9421). + +Most major bot-detection services, CDNs, and WAFs verify Web Bot Auth automatically. See [Web Bot Auth](/browsers/bot-detection/web-bot-auth) for how Kernel signs requests. + +## Contact + +For questions about Kernel bot or agent traffic, contact [support@kernel.sh](mailto:support@kernel.sh). diff --git a/browsers/bot-detection/web-bot-auth.mdx b/browsers/bot-detection/web-bot-auth.mdx index 8c83b86..f54f9fb 100644 --- a/browsers/bot-detection/web-bot-auth.mdx +++ b/browsers/bot-detection/web-bot-auth.mdx @@ -145,18 +145,49 @@ The directory should contain your public keys in JWKS format: kernel extensions build-web-bot-auth \ --to ./web-bot-auth-ext \ --key ./my-key.jwk \ - --url https://yourdomain.com/.well-known/http-message-signatures-directory \ + --url https://yourdomain.com \ + --signature-agent https://yourdomain.com \ --upload my-web-bot-auth ``` -### 4. Register with Vercel and other Web Bot Auth-aware directories (optional) +### 4. Kernel Search configuration -If you want Vercel-protected sites to recognize your agent, you can register your key directory with [Vercel](https://bots.fyi/new-bot). Kernel is officially listed in the Vercel directory. +Kernel Search uses a distinct Web Bot Auth identity from Kernel's user-driven +agent traffic: + +- User-Agent: `KernelSearchBot` +- Signature-Agent: `https://search.bot.kernel.sh` +- Key directory: + `https://search.bot.kernel.sh/.well-known/http-message-signatures-directory` + +To build a browser extension for Kernel Search, use the Kernel Search Ed25519 +private key (JWK) and upload it under a distinct extension name: + +```bash +kernel extensions build-web-bot-auth \ + --to ./web-bot-auth-search-ext \ + --key ./kernel-search.jwk \ + --url https://www.kernel.sh \ + --signature-agent https://search.bot.kernel.sh \ + --upload web-bot-auth-search +``` + +For host-proxy based signing, configure the Search crawler's host-proxy +environment with: + +```bash +HOST_PROXY_WEB_BOT_AUTH_ENABLED=true +HOST_PROXY_WEB_BOT_AUTH_KEY_PATH=/path/to/kernel-search-private-key.pem +HOST_PROXY_WEB_BOT_AUTH_DIRECTORY_URL=https://search.bot.kernel.sh +``` +The signing key must correspond to the public key hosted by +`search.bot.kernel.sh`. +### 5. Register with Vercel and other Web Bot Auth-aware directories (optional) +If you want Vercel-protected sites to recognize your agent, you can register your key directory with [Vercel](https://bots.fyi/new-bot). Kernel is officially listed in the Vercel directory. - ## References - [Vercel's Public Directory](https://bots.fyi/?query=kernel) diff --git a/docs.json b/docs.json index fd5ca1a..aa6cb3b 100644 --- a/docs.json +++ b/docs.json @@ -147,7 +147,8 @@ "proxies/datacenter" ] }, - "browsers/bot-detection/web-bot-auth" + "browsers/bot-detection/web-bot-auth", + "bots" ] }, "browsers/extensions",