test: adversarial robustness suite for the crawl pipeline

[vringar]

Adversarial robustness suite for the crawl pipeline

Generates adversarial circumstances for the crawl pipeline and asserts it
degrades gracefully: forward progress (every started visit reaches a terminal
state), no silent data loss, no hang, proper incomplete-visit recording, and
the crawl continues via the watchdog / BrowserManager recovery path.

What's added

• test/storage/test_adversarial_storage_controller.py — drives the real
  StorageController subprocess in-process (real socket via DataSocket),
  injecting provider faults. Covers write/flush faults (transient + permanent),
  a raising store_record task, and malformed/hostile records (the latter
  against the real SQLite provider).
• test/storage/test_adversarial_socket.py — raw-socket wire-protocol
  hostility against the controller's asyncio server (truncated / garbage /
  oversized / wrong-arity frames, non-sequence body, mid-message disconnect).
• test/test_adversarial_pipeline.py — full TaskManager → BrowserManager
  → Firefox recovery for a crashing custom command, a hanging custom command,
  and a browser killed mid-visit. Browser-required; skips cleanly where no
  launchable Firefox is present, runs for real in CI.
• test/storage/conftest.py — adversarial_mp_logger fixture (captures logs
  like mp_logger but does not assert log-cleanliness, since these tests
  intentionally provoke ERROR logging).
• docs/developers/Adversarial-Robustness.md — catalogue of scenarios,
  survival vs. defect status, and the prioritized gap list.

What the pipeline survives (passing tests)

• All socket hostility: the server stays up after every hostile frame
  (including a 2 GiB length-prefix), a subsequent good visit still completes,
  and shutdown never hangs.
• A transient store failure: the controller recovers and a later visit on a
  fresh connection completes.
• Malformed records (huge values, injection-y strings, missing visit_id)
  against the real SQLite path.

Confirmed defects (captured as xfail(strict=True) — the failing assertion is the finding; CI flags an XPASS once fixed)

• G1 — a raising store_record task strands the visit. The controller
  fires un-awaited asyncio store tasks; the exception only surfaces when
  finalize_visit_id awaits them, after popping them and before recording the
  completion token — so the visit is never enqueued to the completion queue, on
  finalize or on shutdown (browser-crash case). A callback-bearing
  CommandSequence would hang forever and the visit is silently lost.
• G1b — the same raise tears down the whole shared connection (BrokenPipe
  for subsequent records). The TaskManager keeps one long-lived DataSocket
  for site_visits/crawl_history/finalize across all visits, so one bad
  record can break the socket for the rest of the crawl. (The controller itself
  recovers on a fresh connection — proven by the transient-recovery test.)
• G2 — a permanent write_table fault raises out of flush_cache during
  shutdown, crashing the controller before the completion queue is drained —
  completed visits lost.

Pre-existing and related: G3 — SQLite silently drops unknown-table/column
records (and PRIMARY-KEY conflicts), logged only as "Unsupported record"
(crosslink #28/#30). A crashing extension modification (S2) is documented as a
design item; its recovery shape is covered by the mid-visit browser-kill test.

The G1/G1b/G2 fixes are design calls (count-and-continue vs. fail-loud when the
provider itself fails, and whether a per-record failure should ever close the
connection) and are left for a maintainer; the deliverable here is the
adversarial tests + findings.

Relates to #28 / #30 (silent drop) and the socket-reliability thread.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 3.63636% with 159 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.09%. Comparing base (25a7827) to head (b1a2e03).

Files with missing lines	Patch %	Lines
...est/storage/test_adversarial_storage_controller.py	3.63%	159 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1198      +/-   ##
==========================================
- Coverage   62.36%   60.09%   -2.27%     
==========================================
  Files          40       41       +1     
  Lines        3930     4095     +165     
==========================================
+ Hits         2451     2461      +10     
- Misses       1479     1634     +155     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.