fix(retain): bound the within-batch link passes so a delta cannot OOM the worker (#3848)

[nicoloboschi]

Fixes #3848.

What broke, and why now

Both passes that link a batch's new units to each other are quadratic in the batch, and both were sized by an assumption that stopped holding in v0.9.2: that a batch is a streaming sub-batch of ~1.7K facts.

Delta retain is not. Since #3660 an oversized replacement diffs the complete body — correctly; that is the fix for re-extracting and tombstoning the parts that had not changed — so slice 1 now owns every changed chunk of the document and hands ~36K facts to a single _insert_facts_and_links. The slice-local diff #3660 replaced had been the only thing bounding this, and it bounded it by being wrong (unchanged=1 changed=0 new=0 removed=19 on a 20-chunk document, from that commit's own measurement). The streaming path's real bounds — retain_chunk_batch_size, RetainMemoryBudget from #3763 — live in retain_batch, and the delta path returns at orchestrator.py:1784, before reaching them.

So this is not "delta retain was always unbounded". It is a bound that was load-bearing and accidental, removed along with the bug that provided it.

The fix

Temporal. Every same-fact_type pair inside the 24h window was appended before _cap_links_per_unit ran, so keeping 20 links per unit cost n*(n-1) tuples. The cap is recoverable from a bounded candidate set: weight is a non-increasing function of the gap, so the top 20 by weight are the 20 nearest in time. Sorting each fact_type group by event_date and walking the next 20 entries hands every unit its nearest successors, and its nearest predecessors through the reverse link each earlier unit writes — the 40 nearest overall, which contains the 20 the cap will choose.

This is the idiom memory_engine.py:10201 already uses for entity-inferred links, down to the same 2*K bound, rather than a new one.

Semantic. normalized[other_indices] is advanced indexing, so the per-unit loop allocated and filled an (n-1, dim) copy on every one of its n iterations. At 36K facts that is a 110 MB memcpy done 36,000 times, inside a synchronous call with the event loop blocked behind it. Same O(n^2 * dim) dot products, now one BLAS call per 256 rows instead of one per unit.

Measured

One fact_type inside the window. tracemalloc for bytes, no tracemalloc for time:

	peak MB	seconds
temporal n=2000	402.5 → 7.9	0.85 → 0.013
temporal n=4000	—	3.60 → 0.027
temporal n=8000	—	14.73 → 0.054
semantic n=8000	unchanged	11.12 → 0.130

Temporal was 4.1× per doubling and is now 2.0×. Extrapolated to the ~36K facts of the document in #3848: ~130 GB, against the 6 GiB limit that issue reports. It crosses that seconds in, which is why the climb is never visible at 1-minute resolution — only the OOM.

Semantic's peak was never the problem; the copies were freed. Its ~4 minutes of blocked event loop was, and it is the wall the temporal fix alone would have exposed next: a worker that stops answering its liveness probe gets restarted into the same requeue loop, just with a different signature.

What changes functionally

Nothing is lost relative to v0.9.1, and per-unit the multiset of link weights is identical to the all-pairs sweep — that is asserted exactly, ties included. Two deliberate differences:

• The per-unit budget is spent on candidates, not enforced during generation. Counting links as they are appended looks like a tighter bound and is a worse one: the reverse links land first, so a unit hits the cap before its own turn comes and keeps only predecessors — a graph biased backwards, where the cap picks the nearest on both sides. test_a_unit_keeps_neighbours_on_both_sides pins that.
• Weight ties resolve differently. Any gap past ~16.8h clamps to 0.3, so a unit with more than 20 distant neighbours has more tied candidates than places. Which ones survived was decided by dict order; it is now decided by proximity.

Under 21 same-type units in a batch the output is literally identical — the window covers everything, so nothing can differ.

Tests

test_within_batch_link_bounds.py pins both rewrites against verbatim copies of what they replaced. The copies must stay verbatim: the moment a reference shares code with the live implementation it stops proving anything (same discipline as test_chunking_streams.py in #3763).

Verified non-vacuous by mutation:

mutation	tests failed
per-unit budget enforced during generation	16
candidate window halved	23
self-exclusion dropped from the blocked semantic pass	12

Scope

This removes the two quadratic terms. It does not make an arbitrarily large delta safe — the remaining per-delta state is linear in delta size, which is headroom rather than a bound. Routing the delta path through the streaming pipeline's budget is filed as the follow-up in #3848 and is the change that actually closes it.

Supersedes #3853, whose approach to the temporal loop this follows — thanks @Sword-Saint69 for the diagnosis and the sliding-window shape.

https://claude.ai/code/session_017ufCz6qrNxn36Stug7ek8A

[strix-security]

Strix Security Review

Warning

This pull request has 1 commit after the last Strix review (627ce7f). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

No security issues found.

_{Updated for 627ce7f.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}