WIP: Apocalypse Mode - optional offline Wikipedia via Kiwix/ZIM

[alectimison-maker]

Direction and status

Draft — product direction changed. Do not merge the current implementation as-is.

The current patch proves that local Wikipedia fallback is feasible, but its bounded corpus of 999 English vital-article introductions is not the intended final product. Please rework this PR toward the fully optional Apocalypse Mode described below and in #2735. The proposer should own the implementation; this update changes the specification only.

Product goal

Add an opt-in Settings → Advanced → Apocalypse Mode entry for worst-case offline or severely disconnected use. Selecting that entry must open a dedicated, full management page—at the same product level as Traces or Chat History—where the user installs, imports, monitors, updates, and removes offline knowledge archives. Archive management should not be squeezed into the Settings panel itself.

When explicitly enabled, WebBrain should let the user download or import a Kiwix-compatible Wikipedia archive and search it locally.

This is offline knowledge retrieval, not a promise that the entire agent works without a network. Cloud-hosted LLMs will still require connectivity; fully offline agent operation also requires a local model.

Why the current 999-article English cache is insufficient

• English-only behavior is an implementation choice, not an offline requirement.
• The most prominent 999 topics are also the knowledge most likely to be represented in modern model weights, so they provide relatively little additional coverage.
• The stronger product benefit is access to long-tail knowledge, deterministic source text, citations, and revision/license metadata.
• Downloading introductions one API batch at a time recreates part of an archive system that Kiwix/ZIM already provides.

A tiny collection may remain available as a clearly named optional starter pack, but it must not be presented as full “offline Wikipedia.�

Information architecture and user experience

• Settings → Advanced contains an Apocalypse Mode entry with a compact status summary, such as Off or the number and total size of installed archives.
• Clicking the entry navigates to a separate Apocalypse Mode management page, comparable to the existing Traces and Chat History destinations.
• That page is the control center for enabling the feature, browsing available archives, importing local archives, viewing storage use and download progress, pausing/resuming work, choosing update behavior, and deleting installations.
• Apocalypse Mode must be off by default and cause no archive download or material storage use until the user explicitly opts in.

When enabling or installing content, let the user choose:

• Wikipedia language independently of the UI language, with a sensible locale-based suggestion.
• Archive scope/tier: topic or starter packs, all article introductions, full text without images, or the largest available archive.
• Downloading an archive or importing an existing local .zim file.
• Storage location where platform APIs allow it.
• Manual or automatic archive updates.

Before downloading, show the current exact download size, available-space implications, archive date, language, content tier, and source/license. Require explicit confirmation. Provide visible progress, pause/resume, failure recovery, update, and complete deletion controls.

For scale, the official Kiwix archive index currently lists English Wikipedia at roughly 12 GB for all introductions, 49 GB for all articles without images, and 115 GB for the largest archive. These values are examples only; the UI must obtain and display current metadata rather than hard-code them.

• Kiwix archive index: https://download.kiwix.org/zim/wikipedia/
• Kiwix browser archive guidance: https://browser-extension.kiwix.org/current/www/index.html
• Kiwix Reader/ZIM overview: https://get.kiwix.org/en/solutions/applications/kiwix-reader/

Technical direction

• Prefer Kiwix-compatible ZIM archives over crawling MediaWiki article-by-article.
• Design an archive/provider boundary so languages, subject packs, and future offline knowledge sources are not hard-coded into agent logic.
• Use the ZIM full-text index or an equivalent local index; return only relevant snippets/passages to the model.
• Preserve canonical source, archive/revision date, language, and license/attribution metadata in results.
• Keep all retrieved archive text on the existing untrusted-content path.
• Support resumable, integrity-checked downloads and recovery after service-worker/browser restarts.
• Do not bundle a Wikipedia archive with the extension.
• Avoid treating extension IndexedDB as the only home for multi-gigabyte archives. Prefer an imported/user-selected local file or another platform-appropriate durable store, while maintaining Chrome/Firefox parity.
• Degrade clearly when a browser cannot provide equivalent persistent-file access.

Reliability and review requirements

The reworked implementation must also address these issues identified during review:

• Disabling/removing the feature during an in-flight fetch must not allow the fetch to repopulate storage or reschedule itself afterward. Use cancellation/generation checks and test this race.
• Persistent failures need bounded or exponential backoff rather than indefinite five-minute retries.
• Test real storage lifecycle and concurrency behavior, not only mocked stores.
• Removing an archive must cancel background work and verify that its data is gone.
• Rebase on current main and rerun Chrome, Firefox, security, and relevant lifecycle tests.

Acceptance criteria

• Apocalypse Mode appears under Settings → Advanced, defaults to off, and opens a dedicated management page when selected.
• Archive installation and lifecycle management happen on that full page, not inside the Settings panel.
• No archive network or storage activity occurs before explicit opt-in.
• The user can choose a language and archive tier and see its exact size before confirming.
• Existing .zim archives can be imported where supported.
• Downloads are resumable and expose progress, pause, retry, update, and delete controls.
• Local searches return relevant passages with attribution and remain treated as untrusted content.
• The feature works after restart and fails safely during removal, interruption, corruption, or insufficient disk space.
• Documentation distinguishes offline knowledge retrieval from fully offline model inference.
• Chrome and Firefox behavior is tested and documented.

Current patch (to be replaced or substantially reworked)

The existing commits download a pinned English catalog of 999 vital-article introductions into IndexedDB in alarm-driven batches, opportunistically cache live Wikipedia responses, and use lexical fallback when Wikipedia is unavailable. That work can inform the provider/search interface, provenance handling, and interruption tests, but it should not set the final corpus, language, storage, download, or UI architecture.

Closes #2735

[vercel]

@alectimison-maker is attempting to deploy a commit to the esokullu's projects Team on Vercel.

A member of the Team first needs to authorize it.

[esokullu]

Thanks for the initial implementation. The product direction is now documented in #2735 and in this PR description: a fully optional Settings > Advanced > Apocalypse Mode entry that opens a dedicated management page (like Traces or Chat History) for user-selected Kiwix/ZIM language and archive installations. Please do not merge the current 999-English-introduction design as-is; rework the implementation toward that specification, including safe cancellation, retry backoff, durable storage/import behavior, progress and lifecycle UI, and Chrome/Firefox tests. I marked the PR as draft. No maintainer code changes were made to your branch.

[esokullu]

Review follow-up on current head 0a38092539695a486885105aa6d774074d776d0e: five additional lifecycle/consent issues should be addressed before merge.

1. OPFS target collision on repeat installs — handle('install') derives the OPFS key from the catalog item id and filename, while manager.install() creates a new random archive-record id. Installing or updating the same catalog entry can therefore create multiple records that share one file; concurrent work or deleting either record can corrupt/remove the other record's bytes. (src/{chrome,firefox}/src/agent/apocalypse-mode.js)

2. Catalog network access before opt-in — the catalog action fetches the Kiwix catalog without first requiring config.enabled === true. This violates the acceptance criterion that Apocalypse Mode cause no archive network activity before explicit opt-in. The same enabled-state boundary should cover other archive-network actions such as resolution. (src/{chrome,firefox}/src/agent/apocalypse-mode.js)

3. Interrupted-import recovery races an active import — every status snapshot invokes recoverInterruptedImports(), while the management page polls status every two seconds and the actual import runs in the page context. The stale scan does not atomically claim the record before deleting bytes and writing an error state, so it can erase or overwrite an import that resumed or completed after the scan. (src/{chrome,firefox}/src/agent/apocalypse-mode.js, src/{chrome,firefox}/src/ui/apocalypse-mode.js)

4. External-file downloads are not restart-safe — a save-picker FileSystemFileHandle is persisted to IndexedDB and later used by the MV3 background worker. A restored handle can require renewed read/write permission, but permission prompting requires a window with user activation; the worker cannot repair that state. The UI needs to preflight/renew permission in the management page and surface a reauthorization state rather than silently retrying an unusable target. (src/{chrome,firefox}/src/ui/apocalypse-mode.js, storage lifecycle)

5. Unexpected worker failures still retry forever — the alarm listener catches any rejected processNext() call and unconditionally creates another alarm five minutes later. The manager's per-record retry cap does not bound this outer path, so persistent store/storage failures can reintroduce indefinite retries. (src/chrome/src/background.js, mirrored Firefox listener)

I am preparing mirrored fixes and regression coverage for these five cases.