MDM batches never fail on size at the end of a run

[paulananth]

MDM batches never fail on size at the end of a run

The installed Company proof failed at the GLEIF merge: a match proposal of 19.7 MB, over mdm.record_match_proposal's 16 MB bound. The same failure happens on main with the old landed bundles, so it is not caused by the raw-capture work (#905). Size failures like this have recurred; this fixes the cause.

Why proposals were too big

How one SEC batch grows on its way through the Stage (919 records):

Stage	Size	Growth
Records file	1.4 MB	x1
Save request	6.1 MB	x4.4
Match proposal	9.6 MB	x7

A GLEIF batch of 1,000 records (about 1.9 MB) made a 19.7 MB proposal.

• The proposal held the save request twice: as command and as effects. effects is read only by save_batch's equality check.
• Batches were cut by count (1,000), never by bytes.
• The bounds were copied by hand into about 10 places and met only at the end, in the database.

What changes

• Migration 013:
  • A proposal stores effects_sha256 in place of effects, and save_batch compares the request's hash to it.
  • A proposal stored before keeps its effects and is checked as before.
  • The 16 MB bound applies to the stored body; a proposal as sent may be 64 MB.
• mdm.prepare cuts a batch at 1 MiB of records as well as at 1,000 records. Batch ids now count batches, so a re-prepared input that is byte-cut gets new ids.
• The Merge Stage measures a batch's request before any SQL and refuses one over the bound, with its size and the remedy.
• edgar_warehouse/mdm/clean/limits.py holds the bounds once; every Python site reads it, and a test holds the SQL functions to the same numbers.

Company preparation's own one-batch bundle gets the same byte cut on the L4 branch, which rewrites that file.

Evidence

• New tests:
  • 6 Postgres proposal tests: the hash is stored and applies; an old-form proposal stored before 013 applies, and a changed request against one is refused; changed effects are refused; the bound applies to the stored body; an oversized request is refused before SQL.
  • 4 limits tests.
• Locally:
  • 218 MDM Postgres integration tests (217 passed, 1 xfailed);
  • 822 MDM and engine tests;
  • 80 core Postgres tests after the review fixes.
• Mastering time measured on chunk 1: 87% database, 13% Python. So the fix is structural; a Rust rewrite would change neither the time nor the failure.
• Full installed proof on this branch: passed (2026-10-11 00:55 ET, 26 min). 6,414 Companies; GLEIF 3,716 records, 3,052 bound; bindings 9216cc18…, equal to the checkout replay; a killed worker's item verified on its next attempt; the second pass changed nothing. Report .planning/workstreams/mdm-batch-size/evidence/installed-company-proof-20261011.json.
  • Two earlier runs failed in the GLEIF read because the Mac slept mid-read and the worker's lease lapsed. The passing run kept it awake (caffeinate).

Review

Three axes:

• GoF: leave it; plain constants plus a test, no pattern.
• Standards: 013 correct against 001; names and messages fixed.
• Spec: the early request check, the deferred-bound test and the old-form refusal test added.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk