perf(retain): give chunks extraction mode its own sub-batch budget

[nicoloboschi]

The problem

retain_batch_tokens (10k) exists to keep a fact-extraction prompt inside a context window. retain_extraction_mode="chunks" never builds that prompt — it stores each chunk as-is and calls no LLM — but it was still cut into sub-batches by the same ceiling.

The consequences compound:

• ~13–29 chunks per sub-batch at the default retain_chunk_size, and sub-batches run sequentially (_run_retain_execution), each with its own document tracking, recovery chunk-hash scan and final ANN pass.
• The streaming producer's per-chunk fan-out has only those ~26 chunks to work with, so the CoalescingEmbedder added in fix(retain): batch a document's chunk embeddings instead of one request per chunk #3785 never had more than one backend request in flight.
• retain_chunk_batch_size (default 100) was unreachable: the sub-batch slice, not the config, ended every consumer batch.

Follow-up to the ceiling flagged but not fixed in #3784 / #3785.

The change

New static config retain_chunks_mode_batch_tokens / HINDSIGHT_API_RETAIN_CHUNKS_MODE_BATCH_TOKENS, default 500_000 (~2MB of text), selected by sub_batch_token_budget() once the retain config is resolved. Every LLM mode keeps the 10k budget unchanged. What still needs bounding in chunks mode is memory, and that is what the new default is sized for.

_resolve_retain_config now also applies the provider == "none" → chunks-mode override. It advertises itself as a mirror of what _retain_batch_async_internal resolves, and without that half a server with no LLM reported whatever mode the bank had on paper while every sub-batch actually ran in chunks mode — which made this whole change a silent no-op on exactly the deployment it was written for.

Deliberately not changed: _split_contents_into_async_children. That site packs distinct documents into child operations and never fragments a single item, so a larger budget there would reduce cross-worker parallelism rather than increase it.

Measurements

In-process retain, LLM_PROVIDER=none (forces chunks mode), local embeddings, isolated pg0, 2MB document, 3 repeats per arm. The structural columns were identical on every repeat; only wall time varied (±0.6s).

	budget 10k	budget 500k
wall	32.2s	19.8s
throughput	26.4 ch/s	43.0 ch/s
streaming passes	32	1
embed backend requests	32	14
peak embed in-flight	1	4
avg embed concurrency	0.83	3.70
DB commits	32 (max 29)	9 (max 100)
tracemalloc peak	10.6 MB	17.3 MB

At 4MB: 64 passes → 2, 16.8 → 36.4 chunks/s.

peak embed in-flight = 1 is the headline: the fan-out existed, but a pass never held enough chunks for a second request to overlap.

The db_commits row is what settles whether retain_chunk_batch_size is dead config — max 29 against a configured 100 means it never bound once. After the change max 100: it becomes the real bound. It is dormant, not dead; it should not be removed.

Behaviour is unchanged

The same document through both budgets produces byte-identical stored state — 427 chunks, contiguous chunk_index, identical text per index (same SHA-256 over the index:text sequence).

Tests

hindsight-api-slim/tests/test_chunks_mode_batch_tokens.py (9 tests, no LLM, no DB):

• budget selection per mode, and a guard that the shipped chunks default stays larger than the LLM default — equal or smaller would silently restore the serialised behaviour with nothing else noticing;
• the splitter reaching retain_chunk_batch_size under the new budget but not the old one, with the same total chunk count either way;
• the provider == "none" override, verified to fail on the unfixed method ('concise' != 'chunks') rather than passing for its own reasons.

hindsight-dev/benchmarks/perf/chunks_mode_parallelism.py reproduces the table above (passes / embed requests / peak in-flight / DB commits / tracemalloc peak). It measures through two monkeypatches, so it refuses to print numbers it never actually recorded.

What has not been verified

The DB-backed retain suites were not run locally — they need a real LLM provider and the benchmark worktree pins LLM_PROVIDER=none. CI covers them. Local lint.sh eslint could not run in a fresh worktree (no node_modules); ruff, ruff format, ty and the touched test suites all pass, and no TypeScript is in the diff.

The gains were measured with local embeddings, where the win comes from feeding the model larger batches. With a remote backend (TEI/OpenAI) the 32 → 14 request drop is HTTP round-trips, so the improvement should be larger.

https://claude.ai/code/session_01HVZ94d143NPSsZjbnvwqba