WebSift — Self-Hosted Web Search & Research Infrastructure for AI Agents

English short version. Bản tiếng Việt có dấu đầy đủ cho intern: docs/README_VI.md. Technical source of truth: docs/SPEC.md · docs/API.md · docs/ARCHITECTURE.md · docs/SECURITY.md · docs/ROADMAP.md.

WebSift is an API-first Modular Monolith (Go, stdlib-only) that gives AI agents ranked web search, clean extraction, site crawl/map, and evidenced research — self-hosted, so you stop paying per-call and stop leaking queries to third parties.

• One binary, production-ready (not a toy MVP): cmd/websift-api serves 7 endpoints (POST /v1/search|extract|crawl|map|research, GET /v1/crawl/{id}, GET /v1/health|version) with auth, rate limits, OTEL-ready structured logs, graceful shutdown, and bounded budgets on every job.
• Deterministic-first: basic search and research work with no LLM. Planner is deterministic (original + keywords + guide/documentation/2026 variants), ranking is BM25 (k1=1.2, b=0.75) + freshness (0.5^(age/180d)) + quality heuristic (weights 0.55/0.25/0.20), fusion is RRF (k=60), dedup is 3-layer (CanonicalURL + sha256 ContentHash + RRF key), diversity cap is 2 per domain.
• Tavily-compatible where verifiable, never claimed 100%: field names mirror Tavily, unknown inbound fields are ignored (lenient decode). Tavily is a reference, not ground truth — ground truth is human judgments in benchmarks/datasets/expected_results.jsonl. No 100% compatible claim until tests/compat/ passes (it does, offline).
• MCP is a thin adapter: mcp/server.go + cmd/websift-mcp expose web_search|web_extract|web_crawl|web_map|web_research as stdio JSON-RPC. The adapter holds no ranking/crawl/synthesis logic (import check enforced in tests/mcp/); it forwards to POST /v1/*.
• Commercial use allowed: Apache-2.0 (see LICENSE). You may run it for a fee, modify it, distribute open/closed builds. Keep the copyright notice, ship the license, mark modified files. No warranty.

Security rules (binding, verbatim)

1. Mọi URL fetch đều phải qua SSRF protection.
2. Web content luôn được coi là untrusted input.
3. Không lưu plaintext API keys.
4. Không log API keys/Authorization headers/sensitive credentials.
5. Không tạo citation nếu không có source/evidence thật.
6. .env chứa secret thật phải luôn bị ignore, không commit.

SSRF is enforced at 4 points (initial URL, redirect, resolved IP, nested resource) with CIDR denylist, obfuscated-IP-literal rejection, resolve-then-connect + IP pinning, max 5 redirects, 5 MiB raw / 10 MiB decompressed caps, and timeouts (15 s direct / 30 s crawl4ai / 8 s searxng; candidate fetches in advanced search are capped at 8 s). Error SSRF_BLOCKED (403) always uses the generic message Fetch blocked by egress policy.

Quickstart (local, Go only, offline)

# 1. Run the API offline (static provider, no Docker, no network)
$env:WEBSIFT_API_ADDR="127.0.0.1:18099"
$env:SEARCH_PROVIDER="static"
$env:CRAWLER_PROVIDER="direct"
$env:WEBSIFT_API_KEYS="dev-key-change-me"
& 'C:\Program Files\Go\bin\go.exe' run ./cmd/websift-api

# 2. Search (another window)
Invoke-RestMethod -Uri http://127.0.0.1:18099/v1/search `
  -Method Post -ContentType 'application/json' `
  -Headers @{Authorization='Bearer dev-key-change-me'} `
  -Body (@{query='raft consensus'; max_results=5} | ConvertTo-Json -Compress)

# 3. Health / version need no auth (for probes)
Invoke-RestMethod http://127.0.0.1:18099/v1/health
Invoke-RestMethod http://127.0.0.1:18099/v1/version

With Postgres 16 local (superuser postgres / postgres123, DB websift on port 5433):

# .env stays local and is ALWAYS ignored (never commit real secrets)
# DATABASE_URL=postgres://websift:websift@127.0.0.1:5433/websift?sslmode=disable
psql "postgres://websift:websift@127.0.0.1:5433/websift?sslmode=disable" -f migrations/001_init.sql

Make targets (for interns)

mingw32-make build              # go build ./...
mingw32-make vet                # go vet ./...
mingw32-make test               # go test ./... -count=1 (no cache)
mingw32-make test-compat        # Tavily compat suite (offline)
mingw32-make test-mcp           # MCP conformance suite (live httptest API)
mingw32-make test-all           # unit + compat + mcp
mingw32-make fmt                # gofmt -w whole repo (run before every commit)
mingw32-make run                # API local offline on :18099
mingw32-make migrate            # load migrations/001_init.sql into local Postgres
mingw32-make benchmark-estimate # dry-run Tavily quota (0 quota spent, MUST run first)
mingw32-make benchmark-concurrency # drill tải §60.21 (tự spawn API, đo CPU/RAM)
mingw32-make benchmark-reliability # drill tắt provider §60.20 (graceful/fatal)
mingw32-make benchmark-full        # full suite 8 drill + report (§60.22)

Commit rule: small commits, detailed messages, commit as you go. Every commit message ends with Co-Authored-By: Claude Code <noreply@anthropic.com>.

Benchmark (budget-aware, 900 requests)

Bucket	Requests
Search basic / advanced / extraction / research / special	300 / 200 / 100 / 150 / 50
Reserve (never planned)	100
Total / usable	900 / 800

Cardinal rule: never hard-code 1 query = 1 request. Every dataset line declares its own estimated_requests (research = max_searches + max_pages). The runner refuses any plan with estimated > usable (REFUSED, exit 1).

# MUST: estimate before any network call (offline, 0 quota)
& 'C:\Program Files\Go\bin\go.exe' run ./benchmarks/runner estimate --dataset benchmarks/datasets/search_queries.jsonl
# Try small against local API (5 queries)
$env:WEBSIFT_API_KEYS="dev-key-change-me"
& 'C:\Program Files\Go\bin\go.exe' run ./benchmarks/runner full --dataset benchmarks/datasets/search_queries.jsonl --endpoint search --sample 5 --base http://127.0.0.1:18099
# Regression gate: recall drop > 5 %, p95 +20 %, error +2 % -> BENCHMARK FAILED
& 'C:\Program Files\Go\bin\go.exe' run ./benchmarks/runner compare --baseline benchmarks/runs/<base> --candidate benchmarks/runs/<new>

Details: benchmarks/README.md · plan: docs/ROADMAP.md §12. Run dirs (benchmarks/runs/) are git-ignored; only datasets + runner are committed.

Repo map

cmd/websift-api/main.go      wiring + serve + graceful shutdown
cmd/websift-mcp/main.go      MCP stdio entrypoint (thin, no logic)
mcp/server.go                5 MCP tools -> POST /v1/* (no ranking/crawl)
internal/api/                router + handlers + middleware + errors
internal/search/             normalize + providers (searxng/tavily/static, comma list) + fusion RRF
internal/ranking/            BM25 + freshness + quality + diversity
internal/crawl/              crawler interface + direct + crawl4ai + jobs + map + URL cache
internal/extract/            NormalizeURL + HashContent + HTMLToMarkdown + Document
internal/research/           deterministic planner + engine + budget + evidence
internal/security/           SSRF validator + safe client + dialer pin IP
internal/auth|cache|config|observability/  keys + TTL cache + env + logger
migrations/001_init.sql      10 Postgres tables
tests/compat/                Tavily Compatibility Test Suite + fixtures
tests/mcp/                   MCP conformance (round-trip + import check)
benchmarks/                  datasets + runner + README (runs/ ignored)
docs/                        SPEC + API + ARCHITECTURE + SECURITY + ROADMAP + ADR + README_VI

License

Apache-2.0 — commercial use allowed. See LICENSE.