Replace client-side lunr search with worker-backed semantic/RAG search

[Copilot]

Replaces the in-browser lunr full-text index with true semantic search powered by the existing Cloudflare Workers AI + Vectorize RAG pipeline. All existing UI components (the ⌘K modal, the React SearchModal, and the CommandPalette) keep their look and behavior — only the underlying search engine changes.

Architecture

graph LR
    subgraph Frontend
        SearchUI[⌘K Search Modal]
        ChatUI[FloatingAIChat]
    end
    subgraph "Cloudflare Worker (ai-assistant)"
        Search["/search endpoint"]
        Chat["/chat endpoint"]
    end
    subgraph Cloudflare
        AI[Workers AI]
        VEC[Vectorize Index]
    end

    SearchUI -->|POST /api/ai-assistant/search| Search
    ChatUI -->|POST /api/ai-assistant/chat| Chat
    Search -->|embed| AI
    Search -->|query| VEC
    Chat -->|embed + generate| AI
    Chat -->|query| VEC

Loading

Both /search and /chat share the same embeddings, Vectorize index and retrieval code — /search is simply the retrieval half of RAG (no LLM call), so it's fast and cheap enough to use for interactive search-as-you-type.

What changed

Worker (workers/ai-assistant)

• New src/search.ts with semanticSearch() — embeds query, calls Vectorize, filters by brand (metadata or URL-prefix fallback), de-duplicates by URL, truncates snippets on sentence boundaries.
• src/index.ts now routes POST /search and GET /search?q=…&brand=…&limit=… with request validation (length cap, limit clamp to 1–25).
• src/types.ts adds SearchRequest / SearchResponse / SearchResultItem types and a brand field on vector metadata.

Indexer (scripts/index-docs.ts)

• Chunks are now tagged with brand metadata and namespaced chunk IDs (eh-… / wc-…) so the worker can filter cleanly and both brands can coexist in the same Vectorize index.

Frontend

• themes/mieweb-docs/assets/js/main.js (the primary ⌘K search modal) — replaced all lunr index loading/building/searching with debounced fetch calls to the worker, using AbortController to cancel stale requests. Shows a friendly "Search is temporarily unavailable" fallback on error. The same HTML templates, skeletons, keyboard navigation, and ?q= deep-linking behavior are preserved.
• src/components/SearchModal.tsx — same swap on the React side; now shows snippets under each result.
• src/components/DocumentationApp.tsx (CommandPalette) — same worker-backed fetch with cancellation.
• themes/mieweb-docs/layouts/_default/baseof.html — injects window.SearchApiUrl from the new Hugo param and drops the lunr.min.js script tag.

Config

• config-eh.toml + config-wc.toml — new [params.search] block with apiUrl = "/api/ai-assistant/search", and removed the lunr.min.js module mount.
• package.json — removed lunr and @types/lunr.
• eslint.config.js — removed the lunr browser global.
• Removed themes/mieweb-docs/assets/js/vendor/lunr.min.js and the now-empty vendor/ directory.

Docs

• workers/README.md — updated title, architecture diagram, API reference, and curl examples for the new /search endpoint.
• themes/mieweb-docs/README.md — removed the vendor/lunr.min.js line from the file tree.

API — POST /search

Request

{ "query": "schedule an appointment", "brand": "eh", "limit": 10 }

Response

{
  "query": "schedule an appointment",
  "results": [
    {
      "id": "eh-eh-features-scheduling-chunk-0",
      "title": "Scheduling",
      "url": "/eh/features/scheduling/",
      "section": "features",
      "snippet": "Scheduling allows you to manage appointments…",
      "score": 0.87
    }
  ]
}

GET /search?q=…&brand=…&limit=… is also supported.

Deployment notes

1. Deploy the updated worker: cd workers/ai-assistant && npx wrangler deploy.
2. Re-index docs (to pick up the new brand metadata): build the site for each brand (./build.sh eh / ./build.sh wc) then npm run index:docs -- --brand eh and npm run index:docs -- --brand wc.
3. Legacy vectors without brand metadata continue to work — the worker falls back to filtering by URL prefix (/eh/… or /wc/…) until they're re-indexed.
4. The Pages binding already routes /api/* to the worker, so /api/ai-assistant/search is reachable with no Cloudflare config changes.

Verification

• npm run typecheck — clean
• npm run lint — 0 errors (73 pre-existing warnings, same as baseline)
• npm run build:components — succeeds, bundle contains no references to lunr
• ./build.sh eh — succeeds, grep -r lunr public/eh/ returns nothing, window.SearchApiUrl is correctly injected
• Worker tsc --noEmit — only 3 pre-existing errors from BaseAiTextEmbeddingsModels / BaseAiTextGenerationModels types (unrelated to this PR)

[cloudflare-workers-and-pages]

Deploying wc-docs with    Cloudflare Pages

Latest commit:	4df709e
Status:	✅  Deploy successful!
Preview URL:	https://2f79b7a1.wc-docs.pages.dev
Branch Preview URL:	https://copilot-replace-client-side.wc-docs.pages.dev

View logs

[cloudflare-workers-and-pages]

Deploying eh-docs with    Cloudflare Pages

Latest commit:	4df709e
Status:	✅  Deploy successful!
Preview URL:	https://41edfe00.eh-docs.pages.dev
Branch Preview URL:	https://copilot-replace-client-side.eh-docs.pages.dev

View logs

[github-actions]

👋 @Copilot, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit 38a7fd6

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit ff7043f

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit 79553df

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit 2b002b7

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit a9c5f33

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit a7ff77f

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit 8761a84

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit bec542f

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit 2c811ae

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit e475962

[Copilot]

Pull request overview

This PR replaces the docs site’s client-side Lunr search with a worker/Pages-function backed semantic (Vectorize + Workers AI embeddings) search pipeline, and adds an inline “Ask AI” (RAG answer) option in the ⌘K search UI.

Changes:

• Add semantic search (/search) + versioning and inline RAG answer (/search/answer) endpoints, plus prompt-injection hardening.
• Update indexer to embed heading-aware chunks with brand + anchor metadata and publish a content-addressed index version to KV.
• Remove Lunr assets/dependencies and update frontend search UIs to call the new APIs (with cancellation and richer snippets).

Reviewed changes

Copilot reviewed 29 out of 33 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
wrangler.toml	Adds DOCS_CACHE KV binding for search cache/version support.
workers/ai-assistant/src/types.ts	Adds search-related types + brand/anchor metadata fields.
workers/ai-assistant/src/search.ts	Implements semantic retrieval + brand filtering + URL de-dupe + snippet building.
workers/ai-assistant/src/rag.ts	Adds prompt-guard rules + injection short-circuit for chat RAG.
workers/ai-assistant/src/prompt-guard.ts	Adds worker-side prompt injection guard (mirrors Pages Functions copy).
workers/ai-assistant/src/index.ts	Routes /search (GET/POST) to semantic search handler.
workers/README.md	Documents new search endpoints and usage.
wikigdrive.toml	Removes lunr module mount.
themes/mieweb-docs/layouts/partials/search-modal.html	Adds Ask-AI CTA container + answer card region.
themes/mieweb-docs/layouts/_default/search.json	Adds rawContent to support heading/anchor chunking.
themes/mieweb-docs/layouts/_default/home.searchindex.json	Updates comment to reflect Vectorize indexing consumption.
themes/mieweb-docs/layouts/_default/baseof.html	Injects window.SearchApiUrl; removes Lunr script include.
themes/mieweb-docs/assets/js/vendor/lunr.min.js	Deletes Lunr vendored bundle.
themes/mieweb-docs/assets/js/main.js	Switches ⌘K modal to semantic search + adds inline Ask-AI.
themes/mieweb-docs/README.md	Updates theme feature list/tree for semantic search.
src/components/SearchModal.tsx	React search modal now calls semantic search + adds inline RAG answer + client caching.
src/components/DocumentationApp.tsx	CommandPalette search swaps to worker-backed search.
scripts/index-docs.ts	Adds brand/anchor metadata, content hashing + KV version publishing, and change-skip logic.
package.json	Removes lunr and @types/lunr.
package-lock.json	Removes Lunr packages from lockfile.
functions/api/ai-assistant/version.ts	Adds KV-backed index version reader.
functions/api/ai-assistant/types.ts	Adds search + version response types and metadata fields.
functions/api/ai-assistant/search/version.ts	Adds /search/version endpoint for client cache pinning.
functions/api/ai-assistant/search/answer.ts	Adds /search/answer RAG answer endpoint with injection defenses + caching.
functions/api/ai-assistant/search.ts	Adds semantic /search endpoint with version-aware caching headers.
functions/api/ai-assistant/rag.ts	Adds prompt-guard rules + injection short-circuit for chat.
functions/api/ai-assistant/prompt-guard.ts	Adds shared injection heuristics + prompt fragment for LLM calls.
eslint.config.js	Removes lunr global.
config-wc.toml / config-eh.toml	Adds [params.search] and removes Lunr mount.
build.sh	Rebuilds React bundle during builds; optionally refreshes Vectorize index when env vars are present.

Comments suppressed due to low confidence (1)

src/components/SearchModal.tsx:680

• ArrowRight uses group-hover:opacity-100, but the parent button doesn’t have the group class, so the hover style will never activate and the icon will remain opacity-0. Add group to the button className (or remove the group-hover styling) so the indicator behaves as intended.

              <button
                key={result.id}
                data-index={index}
                onClick={() => handleSelect(result)}
                className={cn(
                  "hover:bg-muted w-full px-4 py-3 text-left transition-colors focus:outline-none",
                  index === selectedIndex && "bg-muted"
                )}
              >
                <div className="flex items-center gap-3">
                  <FileText className="text-muted-foreground h-4 w-4 flex-shrink-0" />
                  <div className="min-w-0 flex-1">
                    <div className="text-foreground truncate font-medium">
                      {result.title}
                    </div>
                    {(result.heading || result.section) && (
                      <div className="text-muted-foreground flex items-center gap-1 text-xs">
                        <Hash className="h-3 w-3" />
                        {result.heading || result.section}
                      </div>
                    )}
                    {result.snippet && (
                      <div className="text-muted-foreground mt-1 line-clamp-2 text-xs">
                        {result.snippet}
                      </div>
                    )}
                  </div>
                  <ArrowRight className="text-muted-foreground h-4 w-4 flex-shrink-0 opacity-0 group-hover:opacity-100" />
                </div>

[github-actions]

👋 @wreiske, Your documentation has been pushed to https://docs-qa.med-web.com/148-merge/ for commit c0ccdcf