materialize: consistency suite verifying exactly-once against a real runtime - #3294
Open
mdibaiee wants to merge 87 commits into
Open
materialize: consistency suite verifying exactly-once against a real runtime#3294mdibaiee wants to merge 87 commits into
mdibaiee wants to merge 87 commits into
Conversation
A shard whose processing loop fails is marked FAILED and stays that way — the allocator will not reschedule it. Recovery means unassigning it, which `activate` already does for every shard it upserts, so a publication clears failures as a side effect. That side effect is not always what you want. Recovering a crashed connector by republishing bumps the task's version, drives an Apply, and opens a new session; the materialization consistency suite needs the shard back without perturbing the specification it is testing. So this does only the unassigning. It sits beside `split-shards`, which is where the data-plane admin authorization it needs already lives. Like that command, it authorizes at Admin capability and so requires an `estuary_support/` grant.
…runtime A materialization connector is expected to uphold exactly-once delivery, and there was no mechanical way to find out whether it does. The Apply-drains-pending-work contract was tested by hand-calling RPCs with the runtime simulated by the test; fencing was tested by installing fence rows and checking a stale nonce, never a real race; and the integration harness could not crash a connector at a chosen point in a transaction or run two instances concurrently. So the invariants that matter most were the ones nobody could test — most recently the Snowpipe Streaming v2 write path, whose exactly-once claim was verified by hand in production. This runs a connector as a real task on a real data plane, breaks it at precise points, and checks that the destination still holds exactly the right data. mise run ci:consistency [--filter <scenario>] [--debug] Three pieces: - A shim named as the catalog's `local:` command, with the real connector as its argument. It decodes the protocol stream in flight, writes a trace the runner waits on, and injects faults keyed on protocol events — crash, stall, replay an Acknowledge, or run a frozen second instance whose stale commit races. No change to Flow and no change to any connector. - A reference materialization over SQLite implementing each of the four connector classes, with seven switchable defects. It costs no credentials, so scenario development is local, and it makes the document-counter class executable before any production connector adopts it. - Invariant checkers over the workload's own oracle, so the right answer is computed rather than recorded. The expectation is read from the collection with `flowctl collections read`, which the connector under test had no hand in — that is what makes "loses nothing" checkable, since a tail-truncated materialization is internally consistent. The suite proves itself: every scenario names a defect it must catch, and each runs twice — clean, where it must pass, and defective, where it must fail. Unit tests assert that every scenario is paired and every defect is reached, so coverage cannot quietly erode. The runner adds two guards of its own: a scenario whose fault never fired fails rather than passing on an unperturbed run, and the task must commit further transactions after the fault, so a shard left permanently down cannot masquerade as a clean destination. Compliance is default-strict. Every connector is held to every invariant and anything weaker is an exemption carrying a written justification, because the alternative — declare your class, run only that class's checks — makes downgrading the claim the cheapest way to make a failing test pass. Deviations from the spec, and the four scenario families deferred with their reasons, are recorded in docs/materialize/consistency-testing.md.
…sorb publish contention Two corrections found by running the suite: A quiescent read has to wait for the *collection* to stop growing, not for the capture's shards to disappear. Disabling a capture does not promptly remove its shards, and a task that still has shards may or may not still be producing — so waiting on shard lifecycle timed out where reading the collection until its count repeats does not. What matters is only that no new document is arriving; an expectation read one document early reports the materialization as having duplicated something it merely delivered on time. A publication builds against the whole control plane, so it can fail from contention with unrelated work — several scenarios publishing at once, or the stack's own ops-catalog maintenance. A catalog that is genuinely wrong fails the same way every time, so the publish retries a bounded number of times and reports the last error in full.
…rashed shard Gazette declines to unassign a shard under `--only-failed` when it is the *primary* that failed, which is exactly the case a crash fault produces: it reports zero shards unassigned and the task stays down until the scenario's deadline. Dropping the filter recovers it immediately — FAILED to PRIMARY in seconds. Unassigning a healthy shard costs a brief reassignment, and the harness only calls this while already waiting on a task that is not making progress. With this, crash-between-commits passes both ways: clean over 1068 documents with its fault injected, and 507 violations against the non-idempotent-acknowledge defect it is paired with.
Joining a task's shards had no tooling: the procedure was to hand-edit `gazctl shards list -o yaml` — widen the surviving shard's key range to cover its partner's, mark the partner `delete: true` — and `gazctl shards apply`. This is that, as a subcommand. `activate` gains `map_shards_to_join` beside its existing `map_shard_to_split`. Shards are paired and each pair merges into the one with the lower range, which matters for two reasons: - A shard's ID derives from its range *begin*, so merging upward leaves the survivor's ID untouched — it keeps its recovery log and its accumulated state, and only its `end` widens. - `apply_changes` already orders shard upserts before shard deletions, and those before journal deletions, so the survivor owns the widened range before its partner goes away and no key is ever unowned. A pair is joined only where it is genuinely adjacent on exactly one axis — two shards from the same split. A gap would silently drop the keys inside it and an overlap would deliver them twice, so anything else is refused rather than guessed at. An odd shard out is left alone, which makes the command idempotent once a task is down to one shard. Like `split-shards` it requires the V2 runtime, and Admin capability on the task.
`join-after-split` splits a task, lets both children commit, then joins them back and checks the destination. Scaling down is not scaling up backwards: one shard absorbs another's key range and the other is deleted, so work the departing shard still owed has to be picked up by the survivor — whose destination state was accumulated under a narrower range than it now owns. It asserts only on the destination, never on which checkpoint the connector chose, because the asymmetry there is real: a split child inherits its checkpoint from the range that contained it, but two ranges collapsing into one leave no single range that contained the result, so a join falls back to the recovery log. Paired with `ignore-key-range`, the defect under which two shards fence each other out of a single fixed range.
… journal mode Every shard-split scenario failed, and the reactor blamed the runtime: the leader reported `receiving Opened fan-in / unexpected EOF` from the new child. The connector's own log had the real cause — `Error code 5: The database file is locked`. Switching journal mode takes a brief exclusive lock, so `PRAGMA journal_mode = WAL` fails outright while a sibling shard holds the destination open, unless the connection has already been told to wait. The busy timeout was being set *after* that pragma, so the second child of a split died during `Open` and took the whole task down with it. Worth noting for its own sake: the failure presented as a leader-protocol error two layers above the actual fault, which is a good argument for the suite reading a task's connector logs before believing where an error points.
…ot already in it The busy timeout does not help here, which is what the previous commit assumed: changing journal mode needs a brief exclusive lock and SQLite fails that outright rather than consulting the busy handler. So an unconditional `PRAGMA journal_mode = WAL` still died with "database is locked" whenever a sibling shard had the destination open, and every shard-split scenario still failed. Journal mode is a durable property of the file, so the first opener sets it — uncontended, since it is also the one creating the file — and every opener after reads `wal` and leaves it alone. Reading takes no exclusive lock.
…not on upgrade The busy timeout still did not apply, and this is why: every transaction in the reference destination reads and then writes, and a DEFERRED transaction takes only a read lock at the SELECT. In WAL mode, upgrading that to a write lock after another connection has written returns SQLITE_BUSY_SNAPSHOT *immediately* — the busy handler is never consulted. So the loser of the race simply failed, and the second child of a split died fencing during `Open`, which the leader reported as an unexpected EOF from its fan-in. `BEGIN IMMEDIATE` takes the write lock up front, so contention waits out the busy timeout instead. That is what a destination shared between two shards of a split task — and between a live instance and its zombie — actually needs.
…oint The first real finding from the shard-split scenarios, once they could run at all. Under V2 the non-zero shards of a leaderful task are stateless: no recovery log, everything acquired through the leader protocol. The leader therefore refuses an `Opened` from a non-zero shard that carries a runtime checkpoint, and the task fails with `expected Opened` during its fan-in. The reference connector's `remoteAuthoritative` class returned its stored checkpoint from every shard. That is correct-looking and works perfectly on a single-shard task, and takes the task down the moment it is split — exactly the latent defect this suite exists to surface, found first in the suite's own reference connector. So however authoritative a destination is for the *data*, a connector has to gate its checkpoint on being shard zero. Tested against the range the runtime sent rather than the one `ignore-key-range` may have substituted: which shard this is, is not the connector's to decide. It is also an argument for keeping the shard-reconfiguration scenarios in the default set — no single-shard run can reveal this.
…a membership change The second finding from the shard-split scenarios, and the one that shaped the invariant set: a membership change preserves exactly-once delivery of the *set*, but not delivery *order* at the sink. A split child resumes from its inherited checkpoint and may deliver a sequence the departing parent had already raced past, so an id's rows can land out of order while remaining exactly one row per document. That is what the run showed: 1513 documents, no loss, no duplicates, conservation intact, oracle agreement intact — and 78 monotonicity complaints. So the three reconfiguration scenarios declare a monotonicity exemption, with the set-based checks explicitly not exempt. Those four carry the exactly-once claim and are the ones a split has to keep. The document-counter class needed the same exemption for a different reason — rows of an uncommitted transaction stay visible until recovery skips past them — which suggests sink ordering is simply not a property the runtime offers whenever sessions can overlap.
…scenarios `split-during-store` now passes both ways over ~1550 documents. The two that do not are recorded as leads rather than chores, because both deserve investigation before anything is changed: - `split-during-commit` reports oracle-agreement failures in both directions at once — over- and under-counting together, which is the signature of a torn reduction rather than a replay. It is precisely the rule the scenario is named for. Whether the fault is the runtime's or the reference connector's is not established, and `split-during-store` passing narrows it to the mid-commit window. - `join-after-split` applies its join correctly but the task does not resume committing afterwards. The join was verified by hand: two shards collapse to one covering the full range and the survivor keeps its ID.
…ky, not torn Two corrections to the earlier note. The mixed direction of the oracle-agreement failures is not evidence of a torn reduction. A balance is signed, so duplicating a debit and duplicating a credit move the sum opposite ways; there is a unit test making exactly that point, and I read the same signal wrongly here. What actually implicates the harness is the timing: passing runs finish in ~75s and every failing run took 199-240s, spending its time against deadlines. `drain` stops when the destination has gone unchanged for three polls or the deadline passes, and then judges whatever it has — but a task recovering from a membership change can plateau for longer than that, so the runner can call an incomplete destination settled and report the shortfall as a violation. Until settling is conditional on the task being healthy and idle rather than merely unchanged, this scenario cannot tell a real defect from its own impatience.
…rely quiet
`split-during-commit` failed roughly one run in three, and the cause was the harness
rather than the connector. Its own warning sat immediately before the verdict:
WARN the destination stopped short of the collections log="722/740" merged="740/738"
71 violation(s) over 1478 documents
Eighteen documents were still undelivered, and the run took 41s — it gave up early on
the three-quiet-polls heuristic rather than exhausting any deadline. A task restarting
after a membership change stops writing for longer than three polls, so an incomplete
destination looked settled and the runner reported its own impatience as data loss.
So a plateau now counts only while the task has a primary on every shard, and five
consecutive quiet polls are required rather than three. The deadline remains the
backstop, which keeps the property that a genuine shortfall is reported as a
violation naming the missing documents rather than as a timeout — that behaviour is
what makes the defective half of these scenarios meaningful, and it is unchanged.
…f the split flake `split-during-commit` failed intermittently with oracle-agreement violations, each account off by roughly one transaction's worth, in either direction. The root cause is in the reference connector's post-commit-apply class, and it is a real requirement on that class of connector rather than an artifact of the suite. A connector must answer `Load` consistently with everything it has been asked to `Store` in a committed transaction, whether or not it has physically applied it yet. Post-commit-apply stages during `Store` and applies during `Acknowledge`; between those two points its rows are durable and *invisible*. `load()` read only the destination table, so a Load landing in that window returned a document missing that transaction's contribution. The runtime then reduced from a stale base and stored an incorrect reduction. Three things about it explain the symptoms exactly: - **Why it needs a split.** The runtime re-uses documents it cached from prior transactions, so a long-lived session rarely issues a real Load for a key it just stored. A split gives its children cold caches; they issue real Loads. - **Why it is intermittent.** It needs a Load to land inside the staged-but-unapplied window. The scenario's 4s `StartCommit` stall widens that window, which is why this scenario and not `split-during-store` exposed it. - **Why it looked like a torn reduction.** A balance is signed, so a contribution missed once and re-applied once move the sum opposite ways. Mixed over- and under-counting is the signature of a wrong base, not of tearing. So `load()` now consults `_flow_staged` first, newest row wins, with a staged deletion acting as a tombstone rather than falling through to the stale table row. Staged rows are consulted across all shards, not just the caller's: a key belongs to one shard at a time so the newest staged row is unambiguous, and after a split the parent's staging outlives it while a child owns some of those keys.
…covered The root cause of `split-during-commit` generalises past the reference connector, so it belongs in the design record rather than only in a commit message: if a connector stages writes, its Load path has to see that staging. Any connector deferring application — staged files, a post-commit merge, a queued batch — owes the runtime a read-your-writes view over committed transactions. A single-shard test cannot detect its absence, because the runtime's document cache means it rarely issues a real Load for a key it just stored; only a reconfiguration, whose children start with cold caches, makes the gap reachable.
The previous commit made `load` consult staged rows across every shard. That fixed the clean runs — two in a row upheld every invariant over ~1500 documents, where before it failed one in three — and then broke the other half of the scenario: the defective build started *passing*, because shards sharing one fixed range could now see each other's staging, which is precisely the loss `ignore-key-range` exists to cause. A reference connector that quietly repairs the defect it is meant to exhibit is worse than one with a narrow residual gap, so the lookup reads only the caller's own staging. The residual gap is the window after a split where a parent's staged rows sit under the parent's range while a child already owns some of those keys; the low child adopts and applies them, so the window is short. Worth noting what caught this: the paired-defect requirement, doing exactly the job it was written for. A suite that only checked the clean direction would have accepted the over-broad fix and lost a defect's coverage silently. Also here, all in service of diagnosing this class of failure faster: - The drain's health check is a single listing rather than `await_primary`, which blocked for its whole timeout inside each poll and so spent the runner's patience on the check instead of on giving the task time to finish. - The "stopped short" warning now says whether the wait ended because the destination went quiet — a finding — or because the deadline expired, which is not. - A failing run writes `evidence.json` beside its trace, holding the expectation and the delivered rows. Diagnosing one of these from the violation list alone means guessing which side is wrong, and the run's tasks are deleted on the way out.
A shard's `Load` now consults staged rows from every range that *contains* its own — its own and its ancestors' — rather than only its own. Own-shard-only was wrong, and measurably so: `split-during-commit`'s clean half failed 5 runs out of 5. A split child inherits keys whose staged rows still sit under the parent's wider range, so scoping to its own range leaves exactly those invisible. Containment fixes that while keeping siblings blind to each other, since sibling ranges never contain one another. With this, the clean half upholds every invariant over 1500-1800 documents where the Load path is concerned. Two things remain open and are recorded in the design document rather than hidden here: - The `log` binding — delta-updates, never loaded — still loses roughly 2-3% of its documents in about three runs of five, with the task healthy and quiescent. No Load is involved, so this is a separate defect from the one fixed here. - `IgnoreKeyRange` is no longer reliably caught by this scenario. That defect makes both shards claim the *identical* full range, so containment admits them to each other's staging and repairs part of the damage it is supposed to cause. The scenario needs a paired defect that containment cannot repair.
One root cause found and fixed with proof — a Load must see staged writes, including an ancestor's — and two things left open: a ~2-3% loss on the append-only binding that no Load is involved in, and the paired defect no longer biting because containment repairs part of what it exists to cause. The three measured scopes are tabulated, because the wrong ones are the evidence: own shard only failed 5 of 5, which is what refuted the reasoning that the window was narrow.
…ion on recovery Second root cause of `split-during-commit`, and this one is measured shut. `acknowledge` applied only `committed_txn`, the single newest, while `discard_staged_after` removes only transactions *after* it. A transaction that is staged and log-committed but never acknowledged, with a newer one behind it, is therefore neither applied nor discarded: it leaks, permanently. A split fences the parent mid-flight, which is exactly how two of them pile up. What located it was counting rather than reasoning. The shim's trace records documents Stored per binding; comparing that against the destination gave, across three failing runs: 640 stored / 582 delivered / 600 in the collection, 790 / 738 / 760, and 780 / 724 / 740. The connector was handed *more* than the collection holds — the split replays input — and applied fewer. That eliminated the runtime, the harness and the expectation in one step. Every symptom follows: one transaction's worth of documents, ~20, spread thinly across half the accounts at *early* sequences, with zero duplicates. With the fix the append-only binding is exactly right — 730 expected, 730 delivered, none missing, none duplicated — where before it lost 18-23 in three runs of five. The rule for connector authors: on recovery, apply every committed-but-unapplied transaction, not just the most recent. `Apply` already looped over `staged_txns` for exactly this reason; the session path did not.
The leaked-transaction bug is measured shut — the append-only binding now delivers 730 of 730 with no duplicates, where it lost 18-23 in three runs of five. Also records the measurement that found it, because it generalises: compare what the connector was asked to Store, from the shim's trace, against what the destination holds. That eliminated the runtime, the harness and the expectation in one step, after several turns of theorising had not. What remains is narrower and named: a child can see an ancestor's staged rows but cannot tell which belong to an aborted transaction, nor discard them. The underlying flaw is that staging is keyed by a per-shard counter that restarts for a new key range, so keying it by something stable across a membership change is the direction — not another containment refinement.
…shortfall It did not. That claim rested on a single evidence.json showing 730 of 730; the next four runs had two still 18 documents short, healthy and quiescent, against a pre-fix rate of three in five. Indistinguishable. The leaked-transaction bug is real and the fix is correct by inspection, with a unit test pinning it. But it does not explain the symptom, and treating one observation as a result is the third time in this investigation that has gone wrong.
Staging was keyed by `key_begin` alone, and after a two-way split the low child shares its begin with the departed parent — so "shard 0's staging" named both an ancestor's leftovers and a live sibling's in-flight work. Nothing could act on that safely: the discard path would have deleted a sibling's uncommitted transaction, which is the very loss this suite exists to catch. The insight is that a *per-range* counter is sound once the range identity is unambiguous; the counter never needed to become global, only better keyed. So the connector state map, `_flow_applied_txn` and `_flow_counter` are now keyed by `(key_begin, key_end)`, matching `_flow_staged`, which already carried both bounds for the Load lookup. That makes containment the single rule for both staged *visibility* and staged *ownership*: strictly containing is an ancestor, equal is oneself, and two live shards never contain each other. On Open, post-commit-apply now settles what an ancestor left behind — discarding its uncommitted transactions and applying its committed ones — since the ancestor is gone and nobody else will. Both children do this for the same ancestor, which is harmless: applying is idempotent per range and transaction, and discarding is a delete. Status, stated plainly: one run in five is now fully correct in *both* directions for the first time — the clean build upheld every invariant over 1913 documents and the defective build was caught with 129 violations. `IgnoreKeyRange` bites again because that defect makes both shards claim the *identical* range, which strict containment excludes. The other four runs fail with a new signature, recorded rather than guessed at: the merged standard and delta bindings are wrong *identically* (so not a Load problem — the delta binding never loads) while the append-only binding gains a few duplicates. That points at delivery or at the ancestor repay path, and it needs an evidence.json read before anything else is changed.
…class limit The evidence from a fast failure: every bad account has repeated (id, seq) rows in the merged delta binding and the running sum diverges exactly at the first repeat, with final oracle and final sequence correct everywhere. Nothing is lost; early transactions are applied twice. The append-only binding agrees — 783 delivered against 779 expected, three duplicates, none missing. The cause is the ancestor repair, and it is not fixable by keying. A split child finding an ancestor's staged-but-unapplied transaction cannot answer whether its own resume point precedes it: if after, applying is the only way not to lose it; if before, the runtime is about to replay the same input and applying duplicates. A non-zero V2 shard is stateless and takes its progress from the leader, so its starting point can legitimately predate the ancestor's last committed transaction. There is no third option available to the connector. So the limit belongs to the class: post-commit-apply staging is safe across a membership change only if re-application is idempotent per *document*, which a keyed merged destination has for free and an append-only one cannot without a dedup key. That is why the Snowpipe Streaming v2 path uses a counted channel — the document-counter class — rather than post-commit staging. This scenario derived that reason the long way round.
…e Streaming v2 Three corrections to the document-counter class, all from the same misunderstanding: the channel offset belongs to the *destination*, and the connector's only job is to compare its last committed offset against it. **The committed offset was a mirror.** `session.appended` was seeded from the destination at Open and then incremented by the connector as it wrote, and that count went into the checkpoint. Keeping a second copy of the one number that matters is wrong in itself, and worse here: its drift is invisible in exactly the situation the class exists to survive, a process dying between the destination accepting a row and the connector noting that it had. The checkpoint now reads each channel's offset back from the destination, and the mirror is gone. **A merge binding was being run through the counted channel.** An offset counts rows the destination *accepted*, which says nothing about an upsert, and Snowpipe Streaming v2 handles delta-updates bindings only — so a third of the class's behaviour emulated something no such connector does. The binding set is now per scenario: `standard_binding` defaults off for this class, a unit test enforces that no counter scenario is handed a merge binding, and the checkers take an optional standard binding so only the two checks that need a reduced row are skipped. Per-document cardinality, running-sum-against-oracle and monotonicity all still apply. **Nothing exercised the class across a split**, so one channel per binding *per shard* — several appending to one destination table, each with its own offset — was never tested. `counter-survives-a-split` does that now, and it matters beyond coverage: a counted channel resumes by asking the destination how far it got, so a newly created shard needs no inherited state at all. That is exactly the property post-commit-apply staging cannot have, and the reason Snowpipe uses a channel rather than staged files. Also adds a guard for how the split scenario was found missing: a scenario in the table with no test in `tests/scenarios.rs` never runs, and the table then claims coverage that does not exist. `every_scenario_is_reached_by_a_test` fails instead.
…vives a split The offset belongs to the destination, one channel per (binding, shard), several channels of a binding appending to one table. The connector keeps its own copy only to compare against the destination's on restart, and it must read the destination rather than mirror it — a mirror's drift is invisible in exactly the case the class exists for. Also records the conclusion the two classes together produce: a counted channel resumes by asking the destination how far it got, so a new shard needs no inherited state, whereas a post-commit-apply child inheriting staged work must either duplicate or lose. That is the reason the Snowpipe path uses a channel rather than staged files, and this suite now tests the claim directly.
…tion Two scenarios — `split-during-commit` and `counter-survives-a-split` — fail for a reason that is neither a connector defect nor a harness defect. The runtime does not yet guarantee that a prepared transaction is finished under the same shard split it was prepared under, through to the commit of the driver checkpoint. A split or join landing inside that window corrupts any strategy that reconciles against per-shard destination state, and no connector can work around it. Both scenarios are kept, and marked `blocked_on_runtime`: they run, they report, and they do not fail the suite, because there is nothing a connector author could change to make them green. The marker carries a justification long enough to be audited, and the reported line is the signal for when the limitation is fixed. The doc records both directions of the failure (scaling down duplicates, scaling up can lose a prefix) and one property worth carrying elsewhere: keying a channel by the shard's whole range rather than by `key_begin` alone means a new child never inherits an offset that isn't its own, so it can only duplicate, never silently lose. The measured run shows exactly that — 40 duplicates, all x2, nothing missing.
mise resolves tasks from the executable bit, so without it the task is invisible: `mise run ci:consistency` reports "no task found" and suggests near-misses.
Crashing a shard that a split produced was one scenario doing two incomparable things. The two shards fail differently and are now tested apart: - `counter-crash-in-split-leader` crashes the child that is also shard zero. It owns half the keyspace and holds the recovery log, so the runtime replays it and what is under test is the connector: its own channel, its own offset, crashing with appends the checkpoint does not know about. - `counter-crash-in-split-non-leader` crashes a non-zero child, which in a V2 task is stateless — no recovery log, state arriving by leader broadcast — so it is rebuilt from nothing and has to rediscover from the destination how far its channel got. Three things had to be got wrong first, and all three are recorded in the doc because each was a scenario that looked like coverage and was not. A split alone perturbs nothing this class can get wrong: it lands at a transaction boundary, so nothing replays, so no channel has anything to skip — and skipping is the whole of the behaviour. A split-only scenario was paired with `drop-document-counter` and then with `ignore-key-range`, and each time the clean run passed AND the defective run passed too, three runs each. The paired-defect rule is what caught it. A fault cannot be aimed after a membership change by occurrence count. `arm_after` counts a session's own commits and a split child starts at zero, so any threshold a child can reach the parent reaches first: the crash fired mid-split, killed the shard, and the split never landed. Hence `ShardTarget`, selecting by range rather than by when a session got there. Its `admits` tests both bounds, because the upper child of a split owns `[mid, MAX]` and its `key_end` alone is indistinguishable from an unsplit shard's — a bug the non-leader test caught. Either shard's death fails the whole task, not just its own. Whichever dies, the survivor reports `expected leader message ... unexpected EOF`. Measured in both directions, including a crash confirmed in `00000000-7fffffff`. Unassigning the failed shards on a five-second loop for three minutes recovered the task about two runs in three, so both scenarios now set `restart_after_fault`, and `harness::restart_task` disables the materialization to tear its shards down and republishes to build them again — a restart rather than a reschedule. Clean runs verified over 3440 and 4827 documents run singly, and 3587 and 4460 run concurrently.
`join-after-split` never resumed committing, and the cause was in the reference
connector, not the join itself. The reactor showed 33 restarts and 26 copies of:
connector_checkpoint has clock Clock(1785498707s) which doesn't match
Recover's committed_close (Clock(1785498720s)) or hinted_close (...)
which `runtime-next/src/leader/materialize/startup.rs` raises when a connector
hands back a close-clock from outside the shard's own history.
The destination keys `_flow_fence` by range, and a join widens the survivor back
to `(0, MAX)` — the *same key the pre-split parent used*. So the survivor adopted
the parent's checkpoint from before the split, whose clock predates everything in
its recovery log, and the runtime refused it. The fence lookup filtered on
`kb <= key_begin && ke >= key_end`, which matches the shard's own range key, so a
stale row for that exact range was indistinguishable from a true ancestor.
A strictly narrower overlapping row now identifies a join, and a survivor adopts
nothing: the recovery log is the correct resume point, because two ranges
collapsing into one leaves no single range that contained the result. A split's
child still inherits from the narrowest range that strictly contains it. Both
paths have a unit test, the new one asserting the survivor gets `None` even though
a row for its exact range exists.
Also gates the join on every child having committed *for itself*, via
`commits_per_split_shard`. The previous gate counted commits task-wide, which two
commits from one child could satisfy while the other had none. That was a real
hazard on the same mechanism — a survivor whose log still held the parent's
checkpoint — though not what was failing here.
Clean runs verified over 3988 and 2932 documents.
The suite existed to be pointed at real connectors and could not be: the subject was the reference connector in all but name. Endpoint and resource configs were synthesized in its schema, destinations were read through a subcommand only it implements, and the defect pairing doubled the cost of every scenario to compare against a defective build that no real connector has. `materialize-databricks` now passes `baseline` — every invariant upheld over 2764 documents, its destination read back and checked — in 179s. Three parts. `materialize-boilerplate` grows a `read` subcommand and an optional `DestinationReader`, which `materialize-sql` implements for every SQL connector via an optional `RowReader`; databricks and sqlite adopt it in one line each. Resource config shape is asked of the connector through `spec` and read from the schema's annotations — `x-collection-name` names the table, `x-delta-updates` flags delta — because the field is `delta_updates` in one connector and `delta` in another and only the annotation is contractual. And a named subject runs once: the second pass proves the harness can tell a good subject from a bad one, which needs switchable defects. Everything below was found by running it, and none of it was visible while the only subject was the reference connector: The shim never told its child which codec to use, so it spoke JSON to a connector defaulting to protobuf. Flow's Rust JSON encoding of a `bytes` field is not what Go's jsonpb reads, so `Load.key_packed` arrived empty — a real connector is driven with protobuf, which is also the runtime's default. The harness injected its own SQLite `path` into a stranger's strictly-parsed config. Table names were fixed constants on the assumption every run owns its destination, which collides in a shared catalog. `recover` unassigned shards on every poll rather than when a task had stalled, yanking a connector that needed tens of seconds per transaction and livelocking it into 46 restarts. Its patience, and the gates', are now proportional to the subject's transaction pace rather than constants sized for a local destination. And three invariant checks shared one latent assumption: that rows reach them in the order they were stored. `SELECT *` on a table has no order — it returned one account's rows as [10, 12, 4, 9, 26, ...] — so the running sums, and "the latest delta row", were computed over an arbitrary permutation. That reported 600 violations against a connector which was exactly correct, which is the worst thing this suite can do. The checks now establish order from `seq`, and the contract is written where the next one will read it. Monotonicity is the exception and is exempted for a subject read as a table. It is *about* arrival order, and the order rows are returned in is not guaranteed to be the order they were stored in — a distributed destination may return rows from any partition or file in any order. There is nothing to recover: a commit timestamp ties every row of a transaction, and building a total order from that would manufacture violations. The set-based checks carry the exactly-once claim. The crate README gains instructions, including that decrypting a connector's test config is two steps — `sops -d` and stripping the `encrypted_suffix` its own sops block declares, as `unseal` does.
… subject A real connector's class is now declared through FLOW_CONSISTENCY_SUBJECT_CLASS, because how a connector divides durability with the runtime is a property of its implementation that `spec` does not report. Scenarios of a class the subject does not implement used to run anyway, and two of them spent a thousand seconds each against materialize-databricks before failing for that reason alone. But class is a poor filter, and the first attempt at one -- run a scenario only against its own class -- threw away most of the suite's value. A fault a connector must survive is rarely a property of its class: a crash mid-Store must lose nothing whether the connector fences a remote checkpoint, stages queries for a post-commit merge, or counts the rows a channel accepted. Exemptions are permissive, so a scenario written against one class holds another to a weaker or differently-shaped property rather than an impossible one. `Scenario::applies_to` therefore defaults to every class claiming exactly-once, and the four counter-named scenarios are renamed after the perturbation they inject rather than the mechanism one class uses to survive it -- the convention the rest of the suite already followed. Against materialize-databricks this took the number of scenarios verified from 3 to 12. A gap in the runtime is scoped to the classes it exposes rather than failing the whole scenario, so `split-lands-on-prepared-transaction` -- previously `counter-split-during-commit` -- runs for every exactly-once class and is an expected failure only for the counted channel, which writes during Store and so cannot take back the rows of a prepared transaction. It passes for materialize-databricks, and that is the evidence the gap is the runtime's rather than an impossible ask. `zombie-at-start-commit` is the one scenario narrowed to a single class, and for a harness reason rather than a correctness one: the two racing instances are ordered by their Open fences, and a class that does not fence leaves nothing to order them by, so they proceed as two live writers for the whole run. Two scenarios replaying Acknowledge inside a live session are removed along with `Action::Replay`. The runtime sends exactly one Acknowledge per transaction, so the perturbation was protocol-illegal, and a real connector rightly exits on it -- `crash-between-commits` already asks the same question legitimately by crashing after Acknowledged, before the state update commits. NonIdempotentAcknowledge stays paired there. External deadlines are sized to a remote destination: every scenario that failed at 900s failed by running out of budget mid-phase, one having committed 2 of 3 transactions and another never reaching a fault keyed on the second post-split commit. This turned up two defects in materialize-databricks, both filed: estuary/connectors#4986 (a multi-shard task crash-loops on a retryable concurrent COPY INTO error) and estuary/connectors#4987 (joining shards silently drops documents a departing shard had staged).
…rnals The suite already purged the ops partitions a finished scenario leaves behind, because they saturate a four-broker data plane's assignment slots at around fourteen hundred and every later scenario then fails waiting for a capture that can never append. Task shards leak the same way and are worse. A run killed mid-flight -- a CI timeout, a wedged stack -- can delete its specs while leaving its shards in etcd, and those look harmless right up until the build artifact they reference is collected. Then gazette's `completeRecovery` nil-dereferences rather than erroring, the reactor segfaults on every promotion attempt, and it crash-loops: 7,581 restarts, with the whole data plane down behind it, because validating a derivation needs a reactor to connect to. Nothing in the suite's own output says why. The partition leak degrades a stack; this one wedges it. They also live in a second etcd root. Journals and recovery logs are under `/gazette/<cluster>/items`, task shards under `/flow/<cluster>/items`, so a purge of the first alone leaves the shards -- which is exactly what happened while diagnosing this: 67 items removed and 39 shards still there. Matching is by live-spec substring rather than by parsing a catalog name out of the key. Key shapes differ by kind -- `<kind>/<name>/<hex>/<range>` for a shard, `recovery/<kind>/<name>/<hex>/<range>` for its log, `.../name=<url-encoded>/...` for an ops partition -- and a parser that mishandled one shape would delete a live task's state. Substring matching can only err toward keeping debris. Every candidate must already name the suite's own tenant prefix, which bounds the function so it cannot reach another tenant or the data plane's own tasks whatever the matching decides. Verified against real state rather than a fixture: 39 orphaned shards removed, 0 remaining, and the 12 non-suite items -- the ops rollups and the soak task -- untouched.
A review pass over the crate. Three kinds of finding, and the third is the one that mattered most. **Dead code.** `Store::staged_txns` queried columns `_flow_staged` no longer has, so it would have failed at prepare had anything called it. `record_applied_spec` and its `_flow_spec` table were write-only. The file-based exemption machinery -- `Exemption::load`, the `Scope` enum and its tests -- had no caller, and `partition_exempt` filters by invariant alone, so `scope` was parsed, stored and never consulted; the README promised exemption files "live in the connectors repository" and nothing read them. `read_destination` took a `delta` it discarded. **Duplication.** `scenarios::Subject` and `catalog::Subject` were field-for-field identical and copied between; now one type. The resource-config construction existed twice, in the catalog and in the verification read that addresses the same tables; now one helper. Two monotonicity justifications appeared verbatim three times each, which is how wording drifts; now two consts, and `scenarios.rs` is 59 lines shorter. The mise task's `--concurrency` default of 2 silently overrode the nextest profile's `test-threads = 4`, so one of the two settings was always dead and the toml's argument for four was never in effect. The default is gone; the profile governs unless asked. **Comments that said the opposite of the code**, which is worse than no comment because it is trusted. The zombie's nonce ordering was backwards -- the instance that fences *second* holds the newer nonce, and the zombie opens first precisely so it ends up stale. Flow's key ranges are inclusive, not half-open. A stale block above `merge_peer_patches` described peer work running at `Open`, the opposite of the current design. `ENV_SUBJECT_CLASS`'s doc still argued the abandoned only-own-class model, contradicting `Scenario::applies_to`. The `unassign` rationale misattributed gazctl: it does remove a FAILED assignment under `--failed`; what that filter skips is a wedged-but-not-yet-FAILED primary, which is precisely the state `recover`'s stall detection fires on. Eleven stale doc-comment stacks -- an old function's doc left sitting above the one that replaced it -- are removed. The `spec` defect list, which offered a defect that no longer existed and omitted one that did, is now generated from `Defect::ALL` so it cannot drift again. The nextest group is renamed `capped-shard-reconfiguration`, since it caps rather than serialises, and its guard test and message follow. Four design-doc passages were wrong: recovery does escalate to a republish, `ShardTarget` is implemented rather than deferred, a paragraph was duplicated, and "no unit seams below the runner" describes a crate that has several. Four connector claims were overstated. Staging load keys is what databricks, snowflake, bigquery and postgres do -- but clickhouse, elasticsearch, bigtable and google-sheets call `WaitForAcknowledged` inside the loop and read per key, equally correct because the wait still precedes the read. The reference's "primary" test adds `r_clock_begin == 0` where databricks tests `keyBegin == 0` alone, and databricks' coordinator behaviour is gated behind its `scale_out` flag; both divergences are now stated rather than implied away. The document-counter class models the *production* Snowpipe Streaming v2 design, not the in-repo snowflake path, which stages blobs and registers them at `Acknowledge`. And `read` is scoped to connectors implementing `sql.RowReader` -- two of them today, behind an unmerged PR. **Two trust fixes.** `Outcome.exempted` was populated and dropped, so an exemption that never suppressed anything was indistinguishable from one carrying real weight; each now prints its count, and the monotonicity exemptions turn out to suppress 6-263 violations apiece. And any harness error in a scenario's defective half counted as the defect being caught, including a publish that failed from stack contention -- so a flaky control plane could silently vacate a pairing. `stack::PublishFailed` is a typed marker in the error chain, and the defective half now panics on it instead. The README gains the requirement that produced two retracted issues today: a scenario which splits or joins shards needs the subject configured for multi-shard operation, and where that is behind a feature flag the harness cannot know its name. `materialize-databricks` gates its coordinator on `advanced.feature_flags: scale_out`, off by default; without it two shards contend over one table and the suite reports defects that are not there.
A run's tables carry its run id, so nothing reused them and nothing removed them: every scenario left three behind in the destination, holding data no one would read again. The reference connector was never affected — its destination is a file inside the run directory, deleted with it — so this only shows up against a real subject, where it had accumulated well over a hundred tables during development. Three properties of the call site, all deliberate: It is best-effort and never fatal. A warehouse refusing a DROP says nothing about the connector's consistency, and failing a scenario that passed would trade a real signal for a housekeeping one. It runs whatever the outcome, unlike the run directory kept on failure for its trace. A failing scenario's evidence is already in its violation report, so the table itself is not needed afterwards. It skips the standard binding for a scenario that never materializes one, because a warning on every counter-class run is how people learn to ignore the warnings that matter. Verified end to end against a real warehouse: baseline upheld every invariant over 2,953 documents, logged no cleanup warnings, and all three of its tables were confirmed absent afterwards by name rather than inferred from the absence of an error. The *invocation* is provisional and expected to change. It currently shells out to a `drop-resource` subcommand, and connectors should not grow subcommands for a harness's benefit: `Materializer.SnapshotTestResource` and `DeleteResource` already exist for exactly this and the integration tests already use them. Reaching them from outside needs the driver in an importable package -- which `materialize-iceberg`, `-google-pubsub` and `-sns` already do -- so the subcommand goes away once those migrations land. What does not change is everything above: the call site, the best-effort semantics, and the resource-config plumbing are settled regardless of how the destination is finally reached.
…erify The one path left by which a run could pass while checking far less than it claims. `ResourceShape::delta` was optional, and `resource()` silently omitted the flag when a connector had no `x-delta-updates` property. Both delta bindings then became merge bindings, and a duplicate applied to a merge binding is an idempotent upsert — invisible. The append-only binding survived by accident, its key including `seq`, but `merged_delta` degenerated to one row per account, which `check_merged_delta` satisfies trivially. The run passed, with its sharpest detector switched off and nothing saying so. The field is now required, so the state cannot be constructed: a connector without the annotation is refused at `spec` with a message saying why it cannot be verified rather than what field is missing. Two more places where a value described the reference connector and was reported as though it described the subject: `suppressed_rows` is read from the reference connector's own `_flow_suppressed` table, in a SQLite file a real subject does not have, so every external run reported "0 re-delivered rows absorbed". By this suite's own rule a scenario that absorbed nothing demonstrated nothing — so a hard zero there was not merely wrong, it asserted the opposite of the truth, which is that nothing was measured. It is now `Option`, printed as unmeasurable. `standard_binding` came from the scenario's class rather than the subject's, so a subject declared `documentCounter` — delta-only by that class's definition — would still be handed a merge binding by any scenario written against another class. The unit test enforcing this covers only the reference path. It now derives from the subject's class where there is one. Also: the destination-size guard polled a path that never exists for a real subject, and the zombie was respawned on any later `Open` without consulting its fired marker, so a live instance dying after the zombie had already raced would bring up a second runtime-fed connector for the rest of the run. Fencing makes that survivable for the one class the scenario applies to, which is exactly why it would have gone unnoticed. Six comments that no longer matched the code: the crash-between-commits rationale sat on `Acknowledge` when the fault is on `Acknowledged` and its own comment says why; `crash_at` claimed "the first transaction" when `nth` is session-scoped and `arm_after` is what that meant; `load`'s doc still described `Open` applying pending work, which it deliberately no longer does; the materialize proto path; and a scenario function still carrying its pre-rename name. Simplifications: the defect-pairing test iterates `Defect::ALL` instead of re-listing all seven, so a future defect is covered automatically; two auto-discovered `[[bin]]` sections; a one-line delegating wrapper; an assertion the `applies_to` builder already makes; and `--concurrency` was silently ignored when `--filter` was given. Finally, several comments narrated what went wrong during development rather than the constraint a reader needs. The rules are kept and the stories dropped — they belong in the commit messages that introduced them, where they already are.
Per this repository's convention a README is "ONLY a roadmap for expert developers, orienting them where to look next". This one had become a second design document, arguing the same points as `docs/materialize/consistency-testing.md` in its own words: the two rules, how a violation is detected, reproducibility from journals, the connector classes, and the compliance model were each stated in both places. Removed from the README, where the design document already says it: the two rules (which it states as four, including the two nothing can enforce mechanically), how a violation is detected, reproducibility, and the compliance model. The README now points at where each lives rather than restating it. Moved rather than removed: the class table, with the post-commit-apply details that follow `materialize-databricks`. The design document discussed the classes but had no table, so deleting it would have lost something. Kept, because it is roadmap material and not argument: the diagram, the entry-point table, the run commands, running against a real connector, and reading a failure. That last one gains the concrete gate messages worth recognising on sight, where the run directory is, and what purges the debris a killed run leaves. README 292 to 221 lines; the design document 470 to 497, having absorbed the table.
`destination-ahead-of-checkpoint` and `recovery-reconciles-with-destination` each declared a monotonicity exemption, and neither was needed. Nothing in those scenarios reorders delivery: there is no membership change, the replayed input is byte-identical journal order, and the recovery skip is a per-binding prefix count, so the order rows arrive in is preserved. Measured rather than argued. Both suppressed 0 violations across three runs, and both still pass with the exemptions removed. This matters because of what the compliance model claims for itself: the set of exemptions is supposed to read as a map of where the fleet is actually weak. An exemption that suppresses nothing makes that map worse, and it is indistinguishable from a load-bearing one by inspection — which is why these two survived review until the per-run suppression count made the difference visible.
… subcommands estuary/connectors#4981 landed on a better mechanism than the one this suite asked for. Rather than `read` and `drop-resource` subcommands on every connector, `tests/materialize/testctl` is a program outside the connector that calls `Materializer.SnapshotTestResource` and `DeleteResource` — the same functions the connectors' own integration tests call. That is the right shape, and not only because a production CLI should not grow surface for a test harness. Both capabilities already existed on the `Materializer` interface, so the subcommands were a second implementation of them: the parallel abstraction is gone, and there is now one account of what a resource holds and one way to drop it. The reference connector keeps its own `read` subcommand. It lives in this repository and nothing but this suite runs it, so a subcommand there costs nothing — and it is Rust, which `testctl` cannot drive. `ReadVia` makes the two paths explicit at the call site rather than implicit in which arguments happen to be set. Two more variables, because there are now two artifacts: the connector binary the shim `exec`s and the runtime drives, and `testctl`. `FLOW_CONSISTENCY_SUBJECT_NAME` is the name `testctl` knows the connector by, which is not derivable from a binary's file name. Table names now carry `_flow_test_<unix>` as well as the run id, which is the convention `testctl -mode sweep` requires. This matters beyond tidiness: dropping by name can only remove what the caller knows it created, so a run killed before its own cleanup leaves tables nothing can find. Sweeping enumerates what is actually present, and the timestamp is how it leaves a concurrent run's tables alone. Not yet exercised end to end: #4981 is still open, so nothing has run against a `testctl` built from it. The invocation is written to its documented interface — flags, modes, and the newline-delimited-JSON-per-row output the parser already expects — and the reference path, which is what CI runs, is unaffected.
…what the docs claimed The README said "all five are required together; setting some alone is an error rather than a silent fall back". That was false, and the code is what was wrong: `external()` tested only the SUBJECT/SUBJECT_CONFIG pair, so setting just _CLASS, _TOOL or _NAME returned None and quietly ran the reference connector -- and a green reference run looks exactly like a green real one in the summary. It now requires all five or none, naming which are unset. The design document still described the abandoned design, in a section that directly contradicted the README and this crate's own comments: "materialize-boilerplate exposes what it already has as a read subcommand ... One code path serves both". Rewritten to record both designs and why the second is better -- the subcommands were a second implementation of methods already on `Materializer`, so the integration tests and the harness had two accounts of what a resource holds -- and to state that there are deliberately two read paths, which is what `ReadVia` exists to make visible. Nine comments no longer matched the code. Three were verified false rather than merely stale: `documentCounter` was described as fenced by a nonce table, when `append_counted` takes no nonce and checks no fence; the `published` deque's justification claimed the runtime can deliver `Acknowledge(N)` after `StartedCommit(N+1)`, which it cannot, since `Flush(N+1)` waits on every shard's `Acknowledged(N)` -- the defence is kept, but for the protocol's allowance rather than an ordering the runtime never emits; and `Spec`/`Apply` were described as separate sessions when only `Validate` is. The rest: a stall/replay/zombie list naming an action that no longer exists, an arming count that disagreed with its own rule, a claim that `check_merged_delta` was the only order-dependent check when `check_log` is a second, a dangling cross-reference, and an overstated claim about what `reduce.jsonl` records. A gap in the applicability model, which nothing recorded a decision about: `split-during-commit`, `split-during-store` and `join-after-split` all default to every exactly-once class, so a counted-channel subject would run them -- and each puts a membership change on a live transaction, which is precisely the runtime gap `split-lands-on-prepared-transaction` names. They are narrowed away from that class. Narrowed rather than marked `blocked_on_runtime`, because exposure here is a *race*: a split lands mid-transaction nearly always rather than always, and only once a batch has been appended. A gap marker asserts the class must fail, which would make a lucky pass a test failure. `split-lands-on-prepared-transaction` is the scenario that asks the question deterministically, by stalling a live prepared transaction, and it keeps the marker. `crash-in-split-leader` and `-non-leader` are deliberately not narrowed: they crash after the split has settled, so the replay happens under stable membership, and a counted channel handles it. One exemption deleted: `StandardDeltaAgreement` on `at-least-once-never-loses`. A crash at `StartedCommit` replays the whole transaction symmetrically to every binding, so both views inflate identically and the checker agrees -- the justification's premise, a duplicate applied to one binding and not the other, is unreachable under this fault. Verified: the scenario still passes without it. Five other exemptions were audited for the same treatment and **kept**, because measuring them refuted the argument for deleting them. The claim was that reorder-without-duplicate is unreachable, so any monotonicity violation would ride with a non-exempt duplicate and fail the run regardless. Over three runs all five suppressed between 13 and 295 violations apiece, with no zero -- and those runs passed, so the reordering happens without duplication. What separates them from the two deleted in 5525ae9 is a membership change: a split child resumes from an inherited checkpoint and can deliver a sequence the departing parent had already raced past, which is what these justifications say and what the numbers confirm. Deleting them would have turned five passing scenarios into failures. Finally, the suppression counts are only trustworthy on a reference run, and now say so: a real subject also gets the blanket monotonicity exemption, which matches the same violations, so a scenario-level one is credited for work the blanket one would have done anyway.
…rounds to delete `at-least-once-never-loses`'s conservation exemption measures zero on most runs, which under the audit discipline added earlier would read as paperwork. It is not, and the reason is in the workload: it is double-entry, so every transfer is a matched pair of legs. Replaying a whole transaction re-applies both legs, the sum still balances, and nothing fires — which is why the count is usually zero. It fires when a pair straddles the replayed transaction's boundary, duplicating one leg without its partner, and transaction boundaries are time-based, so that is uncommon rather than impossible. Removing the exemption would make the scenario fail intermittently. So the discipline needed correcting, not just this justification. A zero count is corroboration for deleting an exemption, never the argument: that has to be a reason the violation cannot occur at all. The two deleted in 5525ae9 had both — no membership change, byte-identical replay, a pure per-binding prefix skip — and the five kept afterwards had neither.
… cannot run Two of these I introduced yesterday, and both misled in the direction that matters most — what a reader believes ran against their connector. `README` said "only two things are excluded" and the `ENV_SUBJECT_CLASS` doc said much the same, both written before `MEMBERSHIP_CHANGE_FAIRLY_ASKED` added a third exclusion in the same session. A `documentCounter` subject skips five scenarios, not one. Both now list the exclusions and defer to `Scenario::applies_to` and a run's own `not-applicable` lines rather than to prose that has already drifted once. The exemption comment on `split-lands-on-prepared-transaction` claimed "a class the gap does not expose is skipped by `applies_to`". That scenario never narrows `applies_to` — every exactly-once class runs it and the unexposed ones must pass, which the adjacent comment says is the entire point. The conclusion held for a different reason, which is now the stated one. Two comments still described the retired boilerplate-`read` design, claiming one code path serves the reference connector and real subjects alike. Since testctl there are deliberately two, and the boilerplate ships no read subcommand at all. `Apply`'s table-drop comment said "reaching here having drained means its staged work has already landed", contradicting the same function's own doc that `Apply` drains nothing. The shim's Spec/Validate/Apply comment asserted how the runtime groups those into sessions. It has been wrong twice, in opposite directions, so the claim is gone rather than guessed a third time: the pass-through does not depend on it, and the reason it gets no zombie — a second process re-running someone's DDL buys nothing — never did. Finally, a pairing that cannot execute. `split-lands-on-prepared-transaction` is an expected failure for its own class, so `both_ways` panics before the defective half ever runs, making its `catches` a claim nothing tests. `every_defect_is_paired_with_a_scenario` now inventories such pairings explicitly and asserts the set, so a defect cannot come to look covered by a pairing that can never run.
… three stories Five trace gates in the harness were the same loop written five times — read the trace, decide from it, sleep, or fail on a deadline. They are now four calls to one `poll_trace`, whose argument is the only thing that genuinely differed: what a timeout *means*. `count_commits` takes the trace it is given rather than re-reading the file, so a poll reads once. Three spawn-with-deadline blocks in `stack.rs` were likewise one shape, and had already drifted into three: two `ensure!` and one `bail!`, two appending stderr raw and one trimming it, three phrasings of the same timeout. One `bounded_output` now serves `flowctl`, the shard scripts and the reader. The testctl story existed in three long copies — the README, the design document, and the `ENV_SUBJECT_TOOL` doc. The class table existed twice, byte-identical. Per this repository's convention the design document is the home for reasoning, so the other copies are pointers now. Drift had already happened once here, which is the argument that matters: the README and the design document contradicted each other for a day. Also gone: `--failed` and `--dry-run` from `shard-tools.sh`, which nothing passed — and the `--failed` comment actively recommended the filter that `unassign_shards` documents as wrong, since it skips a wedged-but-not-FAILED primary, which is the state the stall detection fires on. And `Row.binding`, written and never read. Two items from the same review are deliberately not here. Collapsing the reference's `published` deque to the single map `materialize-databricks` keeps would make the reference more faithful and remove real code, but it changes what the connector does under replay and so belongs in its own change, measured. And `Trigger::Load` stays: no scenario faults on it, but removing it changes whether Loads are offered to the zombie, which is a behavioural decision rather than a tidy-up.
…ould not The reference kept a ledger keyed on (table, key, doc) across every batch, so a row re-staged under a new batch was refused rather than appended. Its comment claimed this modelled "a real destination recognising a staged file it has already loaded". It did not: `materialize-databricks` dedupes at *file* granularity with fresh-UUID names, so the same row in a new staged file loads a second time. The reference was more forgiving than the connector it models, which is the one thing this crate must not be. Narrowing it to same-batch dedup turns out to mean deleting it. Same-batch idempotency is already provided by the batch's own last statement retiring its staged rows — re-running an entry finds nothing staged — which is also why the `non-idempotent-acknowledge` defect works by *not* retiring. A ledger keyed by batch would claim every row for its own batch and append all of them, which is what no ledger does. Two measurements say this was right, and the second is the interesting one: Absorption never fired. Every scenario in every recorded run reported "0 re-delivered rows absorbed", so the mechanism was untested as well as inaccurate — and the metric reporting it could never vary, which is why it goes too. And the defect it was masking is now caught with 754 violations where it previously showed 110. `crash-between-commits` pairs `NonIdempotentAcknowledge`, whose damage is duplicate appends — most of which the ledger was silently absorbing. The pairing passed while the majority of the evidence for it was being suppressed. The full reference suite is unchanged at 14 of 15 with 12 pairings caught, the one failure being the declared runtime-gap scenario. Two store tests now assert what actually holds: a new batch appends, and the same batch applied twice appends once.
…pending estuary/connectors#4981 landed, which makes the README's "not yet merged" caveat wrong and the question it hedged — which connectors can be a subject — a simple one: whichever `testctl` can drive, meaning any whose package is importable. The connectors repository's own `tests/materialize/testctl/README.md` is the current list, so this points there rather than carrying a copy that goes stale the next time one is converted. Also drops a duplicate: the `package main` prerequisite was stated twice.
…oint recovery `fence` identified a join by the presence of a strictly-narrower overlapping row, and refused to return a checkpoint when it saw one. Nothing ever deleted those rows, so `joined` stayed true for that range forever: every later open — including a plain crash-restart long after the join — refused its own checkpoint and fell back to the recovery log. For `remoteAuthoritative`, whose whole claim is that the destination holds the authoritative checkpoint, that silently reopens the crash-after-destination-commit window the class exists to close, and a crash there would double-apply. The absorbed children's rows are now retired once the survivor has adopted nothing from them, after the nonce bump so a departed child's zombie is still fenced off. The existing test only covered the first post-join open, which is why this was green; a second test covers the open after that, which is the one that was broken. No scenario crashes after a join, so nothing in the suite would have caught it. Two more things this review found in yesterday's work, both mine: `every_defect_is_paired_with_a_scenario` could not detect what its own comment claimed. It collected `catches` from *every* scenario including those blocked on a runtime gap for their own class, then asserted against that same inclusive list — so if a defect's only executable pairing were deleted, leaving a blocked one, both assertions would still pass. It now builds the paired set from executable pairings only. And removing `suppressed_rows` left two doc blocks behind, one of which had become the leading paragraph of `run_dir`'s documentation while describing a field that no longer exists. Separately, `gazctl` was a hard prerequisite nothing provisioned. `mise run local:stack` built only the broker, the suite recovers every crashed shard through `gazctl unassign`, and the resolver did not even look in the stack's own GOBIN — so on a fresh checkout the first crash scenario died at its first recovery, while the README claimed the suite needs "a running local stack and nothing else". The stack task now builds it from the same module, and the resolver checks GOBIN first.
`await_commits_each_shard` was the one gate still hand-rolling the read-decide-sleep loop that `poll_trace` exists for — and `poll_trace`'s own comment claimed "every gate below is this shape". `commits_per_split_shard` now takes the trace it is given, like every other reader, so the gate collapses onto it and the comment becomes true. `Session::pending` was a struct field used as a local: initialised empty, populated and fully drained inside one call. It is a local now, which also removes a field that looked like cross-call state and was not. `published` was a `VecDeque` that can hold at most one entry — its own doc says so — because the interleaving it guards against is one the runtime cannot produce: `Flush(N+1)` waits on every shard's `Acknowledged(N)`, so `Acknowledge(N)` always precedes `StartCommit(N+1)`. It is an `Option`, with a `debug_assert!` where the second entry would have gone, since silently overwriting one would surface later as a duplicate nobody could explain. `read_trace` tolerated an unparseable line *anywhere* while its comment excused only "a partial final line". Those are different: a torn line mid-file is two shims appending at once, it will never be complete on a later poll, and swallowing it turns a lost event into a gate that times out for no visible reason. Only the final line is now excused; anything else warns and names the line. Smaller: `--filter` dropped `--no-fail-fast`, so a multi-scenario filter stopped at the first failure. `Stack::shards` was public with one caller inside the same file. And the "subject names no connector binary" error path is a panic now — a `Subject` with no argv cannot be constructed, so per the project guidelines that is an impossible state, not a condition to report up the stack. Left alone: converting six `scenario.split_shards = true; scenario.settle_commits = N;` tails to a builder. A first attempt at it spanned two functions and produced something that only looked right, and the payoff is cosmetic — not a trade worth making at the end of a long change.
… code or the proto does
Eleven claims, each verified against the source rather than reasoned about.
`StartedCommit` said "the connector has committed". The proto says the driver "has started to
commit its transaction (if it has one)", and a Go connector's `StartCommitFunc` may still be
running when it returns. The scenarios using it are unaffected — the window is a superset of
the instant they want — but it is not a guarantee the protocol offers.
`Acknowledged` was "the only point at which the connector has applied a transaction and the
shim can still kill it". It is the *earliest*: a crash anywhere up to the next recovery-log
commit replays the same `Acknowledge`, which is what `crash-mid-store` does too. Earliest is
still the right choice, because least is happening around it.
`Apply` was said to be "handed no connector state, so it cannot know which staged work
committed". `Apply.state_json` exists and runtime-next populates it; the classic runtime sends
`{}`. Draining nothing there is a *choice* — a second reconciliation path exercised only by
some runtimes, where `Acknowledge` is the path every transaction takes — and the comment now
says so rather than claiming a protocol impossibility.
`crash_at` said `nth` "counts within the session, not within a transaction", contradicting the
`nth` field's own two-regime doc and `crash-mid-store`, which depends on `Store` being counted
per transaction. That one was mine, from correcting a different error on the same line.
The crash path claimed "a stopped process does not act on SIGKILL until it is continued".
SIGKILL is the documented exception in signal(7). The `SIGCONT` is kept for a different reason
— so the kill lands on a running process and its exit status is observable — and says so.
The reference connector's structured-logging hooks claimed the reactor "discards" unstructured
stderr, and that this produced "connector exited with no log output". Neither: the decoder wraps
every plain line as a warning, so a multi-line `anyhow` chain arrives shredded with its level
lost, which is the real reason to emit one JSON object. And that message fires only when stderr
carried nothing at all — a SIGKILL — so a scenario reporting it was killed, most likely by this
suite's own fault, rather than having logged something that was thrown away.
A `documentCounter` subject skips **four** scenarios, not five. Counted rather than asserted
this time; the README's own list did not sum to five either.
The design document still carried the account of splits that `scenarios.rs` explicitly retracts
— "it lands at a transaction boundary, so nothing is replayed" — while contradicting it twenty
lines later. And the narrowing guard's comment claimed no scenario needs narrowing except one,
when three are narrowed for a second, different reason; it now states both reasons and its own
blind spot, that a two-class narrowing is invisible to it.
Smaller: a module doc's colon led into a paragraph that had moved; `to_delete` claimed a
garbage-collection role it does not have (retirement is each batch's own trailing `DELETE`);
the trace's concurrent writers are a split task's shims, not the zombie, which writes no trace;
and `stack.rs` reaches the stack through `gazctl` as well as `flowctl`.
The request and response pumps each walked `shim.matched(..)`, logged the fault, and matched on the action — identically, except for the `Zombie` arm. `fire_faults` now does the walking, the logging, and the crash and stall, and hands a matched zombie back: it is meaningful only against a request, because the request pump owns the instance whose stdin it freezes. Two shapes the review also flagged in these pumps are deliberately left alone. The `buffer.reserve(1)` before each `read_buf` may well be redundant, since `BufMut for Vec` grows itself. But if it is not, removing it turns a full buffer into a zero-length read, which both pumps treat as EOF — a silent, intermittent truncation of the protocol stream. That is a bad trade for two lines, and verifying it properly is a separate piece of work from a tidy-up. Extracting the read loop itself buys little: the two differ in what EOF means — the request pump shuts the connector's stdin and the zombie's, the response pump just returns — so what would be shared is the three lines that are already obvious.
`check_log` and `check_merged_delta` scored the same event differently, in two ways that both inflated the log checker: It used `<=`, so a *repeated* seq counted as a regression of order. It is not — it is a duplicate, which `NoDuplicates` owns. Counting it in both reported one fault as two, under two invariants, and one of them is frequently exempt: the same event was therefore suppressed or held depending on which checker happened to see it. And it held a high-water mark rather than tracking the last delivered seq, so a replay of `8, 9, 10` after `10` scored three violations. Only the 8 is a regression; the 9 and 10 that follow it are in order. One replay is one violation now. `check_merged_delta`'s semantics were the principled ones, so they are the reference, and a test pins both checkers to the same score for the same event. The disagreement is the defect — either behaviour is arguable, two behaviours is not, because a run's suppression counts are read as evidence about which exemptions carry weight and two scoring rules make that evidence incomparable. Which mattered, because those counts were the evidence. Re-measured over the six scenarios that carry monotonicity exemptions, the appends-during-store counts fall from 256-295 to 40-54, and the membership-change counts from 13-66 to 9-33 — the double-counting was inflating them roughly fivefold. All six still fire, and none reaches zero, so every exemption remains load-bearing and the earlier decision to keep them stands on numbers that are now comparable. What this does not resolve: a reading of the store says the interleaving these justifications describe cannot occur, and measurement says something reorders delivery without duplicating it. Both cannot be right, and after this correction the numbers are no longer the suspect part. The justifications are, and they should say that reordering is observed rather than assert a mechanism nobody has constructed.
…appens Both monotonicity justifications asserted a mechanism. Review could not construct either, and the objection is sound: on the delta paths these classes use, a parent write past a child's resume point is either refused by the fence or lands as a duplicate — and `NoDuplicates` is not exempt, so such a run fails regardless of this exemption. The counted channel's version had a subtler version of the same flaw: rows of an uncommitted transaction being visible explains why *uncommitted* rows appear, not why committed ones arrive out of order. Yet the exemptions suppress 9-33 violations per run on the reconfiguration scenarios and 40-54 on the counted channel's, measured after the two checkers were made to score a regression the same way. So the reordering is real, and only the explanation was invented. The justifications now record what is established — the observation, its magnitude, and that the set-based checks pass alongside it — and state plainly that the cause is unknown. That is worth more than a plausible story: a reader who believes the stated mechanism will not look for the real one, and will mis-predict which scenarios are affected. The design document gets the same treatment, including that it used to claim the mechanism itself. What justifies tolerating an unexplained deviation is unchanged and is now the load-bearing sentence: no-loss, no-duplicates, conservation and oracle agreement are NOT exempt, so whatever causes the reordering demonstrably neither loses nor duplicates a document. Naming the cause is open work. The two exemptions deleted earlier in review were removed for describing mechanisms that could not fire *and* suppressing nothing; these suppress a great deal, which is why they stay.
`Binding` was a struct wrapping one `Table`, so the `Table` is the binding now. `Invariant` spells each name twice — serde's `kebab-case` rename and a hand-written `Display` — and an exemption is *written* with one and *reported* with the other, so a drift between them would silently stop an exemption matching the violations it names. Unified by pinning rather than restructuring: `Invariant::ALL` plus a test that every printed name deserializes back to the variant that printed it. `trace_apply` and `trace_reduce` each gated on the same variable, resolved the same directory and opened the same file in the same mode. One `trace_line` does that. `trace_reduce` also parsed the same document twice to read two fields from it. `delete_prefix` passed a prefix with no trailing `/` into what flowctl matches as a plain string prefix, so a run whose hex id extended another's could have deleted that run's tasks. Vanishingly unlikely and a one-character guarantee. `all_primary(..).unwrap_or(false)` folded a *listing error* into "unhealthy", so a persistently failing listing reported as "stuck unhealthy" — a real state with a different cause, which sends the reader looking in the wrong place. It warns now. The agent authorization warm-up existed twice, in `local:test-tenant` and `ci:consistency`, for different grants: `read` on the tenant and the support-level `admin` that shard surgery needs. The capability differed, the mechanism did not, so `warm_authorization` moves into the lib both already source. Identifier quoting was inconsistent: `apply_statements` doubled embedded quotes and seven other sites interpolated the name raw. Harness-generated names cannot contain a quote, so nothing was exploitable — but the mixture read as if the seven had been audited and found safe, when they had not been. `Table::ident` is now the only spelling. And the six `scenario.split_shards = true; scenario.settle_commits = N;` tails are three builders — `splitting`, `splitting_after_fault`, `splitting_then_joining`. A regex attempt at this earlier spanned two functions and produced something that only looked right; done by hand per site this time. Not changed, by decision: `fence` still runs for all four classes even though two never consult the nonce. Two classes not reading a fence is a property of those classes, not a reason for the destination to behave differently under them — one `Open` path means a scenario cannot pass because its class skipped the bookkeeping. Recorded at the function so it is not "optimised" later.
…stop repeating three shapes The defective half accepted the *absence* of a clean result as evidence that its defect was caught. Only a failed publish panicked; a warmup gate that timed out, a split that never landed, a collection read that failed, or a fault that never fired all printed "the task could not run" and passed. So a stack degrading between the clean and defective halves silently vacated the pairing — the exact regression the pairing exists to detect. Clean-half-first ordering only guards against failures deterministic across both. The line is now drawn at the fault: a run failing *before* its fault fires carries a `BeforeFault` marker, attached at every gate preceding one, and the defective half panics on that as it already did on `PublishFailed`. A failure after the fault is still accepted, because that is what a wedging defect looks like — `ignore-key-range` leaves two shards fencing each other so neither commits — but the perturbation must have happened. Three claims about real connectors were wrong, and all three were mine. `Apply` drains nothing here, and the comment said `materialize-databricks` does the same. It does not: it runs through `boilerplate.RunApply`, whose `drainPendingState` invokes the connector's own `Acknowledge` for the affected state keys when a resource is altered in place and `Apply.state_json` is non-empty — the use the proto documents for that field. So this is a declared divergence, recorded as one, with the consequence stated: no scenario faults a crash during `Apply`, and a real connector's drain *does* run during perturbed scenarios, because `recover` escalates to republishing the task. Dropping a removed binding's table was described as "what the runtime asked for". Nothing asks for it — `RunApply` never deletes a removed binding's resource, and the connectors repository calls doing so indefensible, language this PR quotes approvingly elsewhere. It is a harness convenience over a destination that belongs to nobody else. And `materialize-clickhouse` was listed among connectors that read per key after an up-front wait. It belongs with the stagers: it waits because `Acknowledge` creates its load tables, then stages inside the loop and joins afterwards. Also the `fence` comment from yesterday claiming `documentCounter` consults the nonce. It does not — `append_counted` takes none — and `atLeastOnce` passes one with checking hardwired off. Only `remoteAuthoritative` reads or writes the fence checkpoint, so someone auditing fence coverage for the counted channel would have concluded it was fenced when it is not. Three repeated shapes are gone. Fourteen identical test bodies and a separately-maintained list of the same names come from one `scenario_tests!` declaration — which also closes a verification gap: the list could previously claim a scenario was covered while no test existed to run it, and the guard compared the list against the scenario table without ever looking for the test functions. The monotonicity check was duplicated verbatim between the two delta checkers, so their agreement was maintained by a test written because they had drifted twice; sharing `check_monotonic` makes it structural, and the test now pins the scoring *rule* instead. And two verified no-ops: an `await_commits(after)` immediately after `recover`, which returns only once the task has reached `after`, and a seven-line `FaultRule` literal that was exactly `crash_at(Trigger::StartCommit, 4)`.
…ix stale claims `the_counter_class_never_takes_a_standard_binding` asserted that the counted-channel class never gets a merge binding — but `standard_binding` was assigned exactly once, from the class, so the test asserted that one line of `Scenario::new` does what it says. It is a derived method now, and the property holds by construction. Likewise `a_join_scenario_splits_first`: `splitting_then_joining` is the only way to set `join_shards` and it sets `split_shards` too. Both read as real guards. A test that cannot fail is worse than no test, because it occupies the place where a real one would go and reports green either way. Six claims corrected, most of them mine: `stack.rs` opened with "everything the harness needs from the local stack, all of it through flowctl", which the same file contradicts twice: shard surgery goes through `gazctl` via the scripts, and reading a destination goes through connector code. Both exceptions are now named. `catalog.rs` said the reduction trace records "each recovery decision". It records applies, not recovery decisions — the counted channel's skip, decided in `open_counters`, is traced nowhere. `protocol.rs` had already been corrected on this; this copy had not. `subject.rs` still said "both variables are required together" above a paragraph explaining that there are five. The README claimed a unit test enforces fault-arming "for every `Crash` rule", naming one exemption. There are two: a rule aimed at a split shard also cannot fire before the warmup, because the shard it names does not exist until the split. A doc comment introducing "the one membership-change scenario" preceded text describing two. And `trace_line` carried two merged doc comments, the first belonging to `trace_apply` — which was left undocumented as a result, and now has it back.
…a membership change `split-during-commit` was named for a state it never reaches, and the state it named had no coverage at all. The shim fires a fault *before* forwarding the request that triggered it, so a `Crash` on a request trigger kills the connector before it receives that request. "Crash at `StartCommit` #4" therefore means the connector never sees `StartCommit` #4: whole 64-document batches of that transaction are staged, the remainder died in memory, no statements were rendered and no state patch was published. Nothing in any checkpoint names those rows. So the scenario's claim — "staged work *committed* by one shard is applied exactly once by the larger set that replaces it" — described something that had not happened. What it does verify is worth keeping, and is now what it says: staging whose transaction never committed must never be applied, and the replay must be delivered exactly once by shards that did not stage it. That is the hazard which produced a real bug earlier in this work, when an earlier reference connector decided what to apply by inspecting the destination and applied abandoned work — landing on exactly this recovery. So this adds rather than re-keys. `split-after-commit-before-apply` crashes at the `Acknowledge` *request*, and because the runtime's cycle is `Acknowledge → Flush → Store → StartCommit → Persist`, an `Acknowledge` opens each transaction and confirms the one before it. Crashing at `Acknowledge` #4 therefore leaves transaction *3* committed-but-unapplied: statements rendered, state patch in the recovery log, apply never run, and the connector's in-memory record of it dead with the process — so recovery has only the checkpoint, which is the point. After the split, that entry is filed under the departed parent's range key, so each child sees it as a peer's rather than its own and only the primary may run it. This is the first scenario to exercise `peers` recovery at `Open`, `merge_peer_patches`, and `apply_pending` over a range that no longer exists. Verified both ways: clean upheld every invariant over 1,437 documents, and `ignore-key-range` is caught — with every shard claiming the whole keyspace, both children compute themselves primary, both find the entry under their own range key, and the task stops short (log 345/570, 17 of 40 accounts behind). And the asymmetry that caused the mislabelling is now documented on `Trigger`, where it will be read before someone keys a new fault: a request-side crash means *instead of*, a response-side crash means *after, unrecorded*. Both windows are wanted; conflating them is what named a scenario for a state it could not reach.
Seven findings, each of which let a run report a pass it had not earned. **A corruption check behind a duplication exemption.** `check_log` compared a delivered document against the one the collection holds and filed a mismatch under `OracleAgreement` — which `at-least-once-never-loses` exempts, for duplication. So a connector altering documents in transit passed a scenario about re-delivery. Content integrity is now its own invariant, and `partition_exempt` asserts nothing exempts it rather than leaving that to review. Its neighbour was mis-filed the other way: a reduced seq *behind* the collection is loss, and reporting it as an oracle disagreement put it behind the same exemption, so a connector losing merged-path documents passed a scenario named "never loses" as long as no account vanished outright. **Exemptions with no ceiling.** An exemption states a cause, and a cause implies a volume: "one replayed transaction" is tens of documents, measured at 40 to 76. Uncapped, the same justification also absorbs a connector that re-delivered the whole workload. `at-least-once-never-loses` now caps its three at 500 — an order of magnitude above measurement, a tenth of the workload — and an exemption over its ceiling stops absorbing anything, so the run fails with the violations themselves rather than a count. Ceilings are per invariant, not per exemption: a real subject also carries the blanket monotonicity exemption, and the broadest claim has to govern or a narrower ceiling would fail a subject nobody made that claim about. **A zombie that raced nothing.** The freeze was keyed at `Store` #10 of the second transaction, which reads as letting the zombie work first and was in fact freezing whatever was left of it: a fenced instance does not survive being run, its first commit is refused, and the process was long gone. The freeze suspended a corpse and the thaw resumed nothing, so the clean half verified that a stale writer dies on its own — while the defective half, with fencing off, raced properly and caught the defect, which is why the pairing looked sound. Frozen at `Open` the zombie has taken its fence and done nothing else, which is the one point it is certainly alive; it replays the transaction whole against a destination two commits ahead. Clean passes at 717 documents, and `skip-fence-check` is caught with 125 violations. **A split racing a stall it was meant to land inside.** Four seconds of stall against a split issued before the fault left the overlap to chance, and a publication takes longer than that. The split is now issued when the stall begins. What the harness still cannot impose is the runtime finishing the stalled transaction before handing over — but that is the guarantee the scenario declares missing, and `split-during-commit` reaches the same state deterministically by crashing instead. **A drain that looked at the first moment it could pass.** The gate is "at least as many as the collection holds", so it was met the instant the last expected document landed, handing the checkers a destination a duplicate in flight had yet to reach. One quiet poll now confirms it. A complete destination that never settles is handed over at the deadline rather than erroring, since "stopped short" is the wrong thing to say about it. **A precondition nothing acted on.** A repeated `(id, seq)` in the collection makes the comparison unsound — the expectation folds it to one, a reducing binding counts it twice — and was recorded in a field. It now fails the run, as an unsound workload rather than a violation, because the fault is in what the harness was given. **A gap declaration that could not be wrong.** The `RuntimeGap` panic was unconditional, so the day the runtime closes a gap the suite would keep printing the old diagnosis of a run that no longer matched it. A pass now fails too, with the opposite message. Two of the review's suggestions are declined in the design doc's Deferred section with the reasoning: a summed `docs: 1` would change the soak fixture this suite deliberately reuses unmodified, and checking `oracle.set` compares an order-dependent value that would need a reordering exemption as broad as monotonicity's — trading an exact check for a suppressed one. The surfaces no scenario perturbs are recorded there too: a crash during `Apply`, a backfill bump, a binding disabled and re-enabled, and a second crash during a replay.
…has stopped growing `join-after-split` failed with 131 violations against a materialization that was right, and the evidence said so plainly: the destination held 1310 log rows where the expectation held 1236 documents. Nothing had been duplicated — the expectation was short. Disabling the captures is published, and a publication returns once the spec is stored. Activation carries it to the data plane afterwards, so the captures were still writing when the collections were read. The only guard was `read_collection_when_final`'s plateau — two equal reads three seconds apart — and a capture pausing across that window reads as a finished one. Nothing here was new; the drain's new confirming poll just left more time for the extra documents to land, which is what turned an occasional wrong answer into a reproducible one. So the run now waits for the captures to have *stopped*, and that is checked rather than assumed. A disabled shard is not deleted — its spec stays listed with `disable: true`, and an earlier version of this waited for an empty listing, which never comes: three scenarios sat at 480s against a 600s timeout. What is observable is the pair — every shard's spec carries `disable`, and no shard still reports primary. The first says the runtime has been told; the second says the transaction it was in is behind us. After that nothing can append and the plateau only confirms. `split-lands-on-prepared-transaction` then reported an unexpected pass, which the new assertion for a stale gap declaration caught on its first outing. Two things came out of chasing it, and both are recorded where the next reader will need them. The gap is real, and this is not the same artifact. A caught run delivered 2072 log rows against 2070 documents: two rows delivered twice, both of them documents the expectation holds, nothing ahead of it. That is the counted channel re-appending a transaction that never committed, exactly as declared. But it is *intermittent*, and forcing it does not work. The hazard needs the runtime to hand the range over mid-transaction and it usually does not — it finishes the transaction and hands over at a quiet point, which is no hazard. Two attempts to force the overlap both made the scenario pass: issuing the split only once the stall had begun, and lengthening the stall to twenty seconds. Given a shard that will hold still, the runtime takes the quiet point; asking it to hand over when the harness likes is asking for the guarantee under test. So this reverts to four seconds and an unordered split, the configuration with observed hits, and the declaration now says the failure is intermittent so a pass is read as evidence about one run rather than about the runtime.
…p it reads on The README and the class variable both said a counted-channel subject skips four scenarios. It skips five: `split-after-commit-before-apply` joined the four that land a membership change on a live transaction, and neither count was updated with it. And the README read a pass of `split-lands-on-prepared-transaction` as evidence about the runtime. It is evidence about that run: the hazard needs the runtime to hand a range over mid-transaction and it usually does not, so the scenario is red either way — with a violation count when it lands, and as an unexpected pass when it does not.
… and draw the setup line at the perturbation Four findings from review, one of them answered differently than proposed. **Loss on the merged-delta binding.** A shortfall there is reported as an oracle disagreement, so a scenario exempting that invariant for duplication absorbs loss along with it — the same mis-filing already corrected in `check_standard`. The proposed fix does not transfer, though: reading a total below the collection's as loss and one above it as duplication assumes the quantity is monotone, and `balanceDelta` is signed and mixed-sign within an account, since every document is one leg of a transfer and an account is both sender and receiver across its history. Omitting a subset moves the total by whichever sign that subset carries. A measured account shows both directions at once: rows missing, the account total -703 against the collection's -671, *below* it, while the running-sum check on the same account reported -319 against an oracle's -358, *above* it. What is sound on this binding is sequence coverage, because a sequence only advances — and it was not checked at all. An account whose delta rows stop short of the collection's latest sequence now files `NoLoss`. That is not covered by the drain gate, which reads the *standard* binding when the subject has one, so a delta binding can be short behind a complete standard one. It catches a missing tail and not a missing middle; the middle stays ambiguous on this binding, and the log binding's row-per-document is what settles it. Both facts are now in the design doc's Deferred section rather than implied by a checker's choice of invariant. **`BeforeFault` sat on the wrong side of a fault-less scenario's perturbation.** `split-during-store` and `join-after-split` inject no fault, so their perturbation is the split — and the gate waiting for both children to commit for themselves carried the marker anyway. Both pair `ignore-key-range`, whose signature is children fencing each other off, which is exactly the state that stops a shard committing: the defective half would have reported a caught defect as "the run failed before its fault fired", i.e. as the environment's doing. The marker comes off that gate. The apparent inconsistency around it — the calls that *issue* a split or join are marked while the gates awaiting their consequences are not — is the rule rather than an oversight, and is now stated on `BeforeFault`: a perturbation that never happened is setup failing, and everything after one is the subject's until shown otherwise. **The zombie's freeze point is now enforced by a test.** The shim logs a dead zombie to its trace, which is read only when a gate times out — so on a vacuous pass nobody sees it, which is the very failure the guard exists to prevent. A unit test asserts every `Action::Zombie` rule triggers on `Open`, where a fenced instance is certainly still alive. **A misquoted runtime error.** Two comments quoted "doesn't match Recover's committed_close"; the message has no "Recover's", so grepping the runtime for the quoted string found nothing.
The rule this was meant to record said "the calls that issue a split or a join carry the marker", and only the split does. The code is right and the sentence overclaimed: the join comes after the split, which is already `join-after-split`' perturbation, so a join that cannot be issued may be the defect wedging its children rather than the environment — marking it would recreate the misclassification the previous commit removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
A test suite that verifies a materialization's exactly-once guarantees against a real
runtime, rather than against a mock. It publishes a workload to a running local stack,
perturbs the materialization at chosen points in the protocol, and then checks invariants
over what the destination actually holds. Addresses estuary/connectors#4956 (steps 1 and 2).
The subject is an input: a connector binary, its endpoint config, and its consistency class.
The reference connector exists to prove the harness can tell a correct connector from a
broken one; any connector can take its place.
Four pieces:
consistency-shim) named as the catalog'slocal:connector, with the realconnector as its argument. From that position it sees every protocol message, traces it,
and injects faults — crash, stall, or a frozen "zombie" second instance whose stale commit
races. No change to Flow and no change to any connector under test.
materialize-reference) over SQLite, implementing fourconsistency classes — remote-authoritative, post-commit-apply, document-counter (the
Snowpipe Streaming v2 shape), at-least-once — with seven switchable defects.
no loss, no duplicates, monotonicity, document integrity, and agreement between a merge
binding and a delta binding over the same collection. The workload carries an oracle in
each document, so the correct answer is computable rather than asserted from a snapshot.
Two rules make the suite trustworthy, and both are enforced by unit tests:
boundaries are shaped by the runtime's duration policy and a rate-paced capture, so a
fault says "the 4th
Acknowledge", never "the document for account 7".clean, where it must pass, and against its defect, where it must fail. A checker that
goes blind through refactoring becomes a test failure rather than a green result that
means nothing.
Workflow steps:
Needs a running local stack (
mise run local:stack). The task provisions the test tenant,grants the support access that shard administration requires, and purges the gazette state of
finished scenario tasks — ops partitions, task shards and their recovery logs.
That purge is not tidiness. Ops partitions saturate a four-broker data plane's assignment
slots at around fourteen hundred, after which nothing new gets a primary broker. Leftover
shards are worse: once the build artifact they reference is collected, gazette's
completeRecoverynil-dereferences rather than erroring and the reactor crash-loops — 7,581restarts in one case, taking the whole data plane down, since validating a derivation needs a
reactor to connect to. Both leak because the control plane does not delete a task's shards or
ops partitions when it deletes the task. The suite is excluded from the default
nextestprofilebecause each scenario publishes tasks to a live stack.
Against a real connector, five variables name the subject, required all-or-none so a partial
naming fails rather than silently running the reference connector instead:
The class is required rather than inferred:
specdoes not report it, and how a connectordivides durability with the runtime is a property of its implementation rather than of its
configuration schema. The tool and name are what
testctlneeds — the name being the one itknows the connector by, not the subject binary's filename, which is whatever the person
building it chose.
A subject driven through a shard split must be configured for multi-shard operation,
and where that sits behind a feature flag the harness cannot know its name — see "What it
found" below for what happens otherwise. For
materialize-databricksthat isadvanced.feature_flags: scale_out.The resource-config shape is discovered by calling
specand reading thex-collection-name/x-delta-updatesannotations, so it works whatever the connectorspells them. A connector with no
x-delta-updatesproperty is refused: it cannot take adelta binding, and a duplicate applied to a merge binding is an idempotent upsert, so
accepting one would leave every scenario passing with its sharpest check silently disabled.
The run is single-pass: the clean/defective pairing needs switchable defects, which only the
reference connector has.
Reading the destination and dropping it afterwards both go through
tests/materialize/testctl(estuary/connectors#4981, merged), a program outside the connectorthat calls
Materializer.SnapshotTestResourceandDeleteResource— the same functions theconnectors' own integration tests use. Neither belongs in the materialization protocol: it has
no request that reads a destination back, and removing a binding deliberately leaves its table
in place, since destroying a user's data as a side effect of a catalog edit would be
indefensible. A harness has the opposite problem, naming resources per-run and otherwise
accumulating them forever in a shared warehouse.
Which connectors can be a subject follows from that:
testctldrives one whose package isimportable, so converting a connector still in
package mainis the prerequisite.A run removes the tables it created. They carry the run id, so nothing reused them and
nothing removed them — every scenario left three behind. The drop is best-effort and never
fatal, because a warehouse refusing a
DROPsays nothing about the connector's consistency,and it runs whatever the outcome, since a failing scenario's evidence is already in its
violation report. Names follow the connectors repository's
_flow_test_<unix>convention, sotestctl -mode sweepcan also clear what a killed run left behind.Scenarios that reconfigure shards use two scripts rather than new
flowctlsubcommands:They drive
gazctl, authorized by the existingflowctl raw gazctl-env --name <task> --admin.flowctlandactivateare untouched by this PR.Documentation links affected:
None — no user-facing documentation changes. Two internal documents are added or updated:
docs/materialize/consistency-testing.md— the design record: decisions and thealternatives rejected, the counted-channel model, and the runtime gap the suite measures,
with the signature by which to recognise it.
crates/materialize-consistency/README.md— roadmap and entry points, including how torun against a real connector and a "Reading a failure" section covering what each gate
means and which symptoms are the environment rather than the connector.
Notes for reviewers:
Expected suite result is 15 of 16 passing (15 scenarios plus a guard that every scenario is
reached by a test). The sixteenth is
split-lands-on-prepared-transaction, which declares aruntime gap and is therefore red either way: a run that hits the gap fails with its violation
count, and a run that misses it — the gap is intermittent, see below — fails as an unexpected
pass rather than going quietly green. Every other scenario passes both ways — clean
upheld, and its paired defect caught.
A scenario applies to nearly every class, and that took two attempts to get right. The
first version ran a scenario only against the class it was written for. That threw away most
of the suite's value: a fault a connector must survive is rarely a property of how it divides
durability with the runtime — a crash mid-
Storemust lose nothing whether the connectorfences a remote checkpoint, stages queries for a post-commit merge, or counts the rows a
channel accepted — and exemptions are permissive, so a scenario written against one class
holds another to a weaker or differently-shaped property rather than an impossible one.
Scenario::applies_tonow defaults to every class claiming exactly-once. Againstmaterialize-databricksthis took the number of scenarios verified from 3 to 12.Three exclusions remain, and
Scenario::applies_tois the authority for them: an at-least-oncesubject skips the exactly-once scenarios;
zombie-at-start-commitneeds a class that fences atOpen, so the shim can order the two racing instances; and the four membership-change scenarios—
split-during-store,split-during-commit,split-after-commit-before-applyandjoin-after-split— skip the counted channel, because each lands a membership change on a livetransaction and whether that reaches the counted channel's exposure is a race. A
RuntimeGapasserts a class must fail, which would make a lucky pass a test failure, so those four ask the
question without the marker and
split-lands-on-prepared-transactioncarries it.Consequently the four scenarios once named
counter-*are renamed after the perturbationthey inject rather than the mechanism one class uses to survive it, which is the convention
the rest of the suite already followed:
destination-ahead-of-checkpoint,recovery-reconciles-with-destination,crash-in-split-leader,crash-in-split-non-leader.Their
verifiesstrings are rewritten to the outcome owed by any class, since those stringsare what a failure reports.
One scenario was added rather than re-keyed, because
split-during-commitwas named for a stateit never reaches. The shim fires a fault before forwarding the request that triggered it, so a
crash on a request trigger kills the connector before it receives that request: "crash at
StartCommit#4" means the connector never sees it, so nothing was committed and no checkpointnames the staged rows. What that scenario verifies is worth keeping and is now what it says —
staging whose transaction never committed must never be applied. The state it was named for gets
its own scenario.
split-after-commit-before-applycrashes at theAcknowledgerequest, andsince the runtime's cycle is
Acknowledge → Flush → Store → StartCommit → Persist, anAcknowledgeopens each transaction and confirms the one before it — so crashing at #4 leavestransaction 3 committed-but-unapplied, with only the checkpoint to recover from. It is the first
scenario to exercise peer recovery at
Openover a range that no longer exists. Clean upholdsevery invariant at 1561 documents;
ignore-key-rangeis caught, both children computingthemselves primary and applying the same entry. The asymmetry that caused the mislabelling is now
documented on
Trigger, where it will be read before someone keys a new fault.The reference connector is modelled on the real ones. Post-commit-apply follows
materialize-databricksand remote-authoritative followsmaterialize-postgres, whichmatters because an earlier version diverged in a way that produced a genuine bug: it decided
what staged work to apply by inspecting the destination rather than its own checkpoint.
Leftover staging cannot distinguish work awaiting application from work abandoned mid-commit.
So the checkpoint now carries the statements themselves, keyed by binding; only the primary
shard runs them, learning of peers' work from the aggregated state patches delivered with
Acknowledge; non-primary shards recover nothing; executed entries clear via null mergepatches; and
Applydrains nothing. The fence is a single conditionalUPDATEwhoseaffected-row count is the verdict, rather than a read beside the write.
One runtime gap, scoped to the class it reaches, and intermittent. A prepared transaction
must outlive a membership change — the capability
discussion 2581 names. It reaches the counted
channel, which writes during
Store, so rows of an uncommitted transaction are already in thedestination when a split lands and cannot be taken back. It does not reach a class that only
stages: the children inherit staged work and a merge that runs again is a no-op. So
split-lands-on-prepared-transactioncarries aRuntimeGapnamingDocumentCounterand runsfor every exactly-once class.
The failure is per-run, not per-suite, and the attempt to make it deterministic is worth
reading before anyone tries again. The hazard needs the runtime to hand the range over
mid-transaction, and it usually does not — it finishes the transaction it is in and hands over
at a quiet point, which is a committed transaction and no hazard. When it does land it lands
narrowly: a caught run delivered 2072 log rows against 2070 documents, two rows twice, both of
them documents the expectation holds, nothing ahead of it. That is the counted channel
re-appending an uncommitted transaction, exactly as declared.
How often it lands is now measured, and the number is low: one run in seven. That is a change
from earlier in this work, when the scenario failed on nearly every run, and the reason is worth
saying plainly rather than presenting the new rate as the old one. The expectation used to be read
from a collection that was still growing, and that produced duplicate-shaped violations too — so
some of those earlier failures were the harness rather than the gap, and which ones cannot be
recovered now that their evidence is gone. The single run examined in detail has the clean
signature above and is a genuine hit. Read the gap as real but rare, and read a passing run as
saying nothing either way.
Two attempts to force the overlap both made the scenario pass: issuing the split only once
the connector's stall had begun, and lengthening the stall from four seconds to twenty. Given a
shard that will hold still, the runtime takes the quiet point — so asking it to hand over at a
moment of the harness's choosing is asking for the guarantee under test. The overlap is left
unsynchronized on purpose, and a pass is to be read as evidence about that run.
split-during-commitreaches the same destination state deterministically by crashing ratherthan stalling, so coverage of the prepared-but-uncommitted state does not rest on this race.
One scenario is narrowed to a single class, for a harness reason rather than a correctness
one.
zombie-at-start-commitorders its two racing instances by theirOpenfences — thelive instance waits for the zombie to fence first, so that it holds the newer nonce. A class
that does not fence leaves nothing to order them by, and they proceed as two live writers for
the whole run. Idempotency is the other classes' answer to a zombie, and
crash-between-commitsis where they are held to it.The zombie is frozen at
Open, and that is load-bearing rather than incidental. It wasfrozen at
Store#10 of the second transaction, which reads as letting the zombie work for awhile and is in fact freezing whatever is left of it: a fenced instance does not survive being
run, because its first commit is refused and the process exits. The freeze suspended a corpse and
the thaw resumed nothing, so the clean half verified only that a stale writer dies on its own —
while the defective half, with fencing off, raced properly and caught the defect, which is why
the pairing looked sound. Frozen at
Openthe zombie has taken its fence and done nothing else,which is the one point it is certainly alive; on thaw it replays the transaction whole against a
destination two commits ahead. Clean passes at 717 documents and
skip-fence-checkis caughtwith 125 violations.
A connector rule worth knowing, because getting it wrong looks like a runtime bug. A
coordinating connector has one shard apply staged work for its peers, so the shard that
loads a key and the shard that applies it differ. Nothing orders them directly: the
leader emits
Action::Loadon its extend path andtail_donegates onlymay_close, so aload phase can start while the previous transaction is still being acknowledged.
Flushcloses the window — it is sent only once the Tail reaches
Done, which requires everyshard's
Acknowledged. So load keys must be staged as they arrive and the destination readonly at
Flush, which is whatmaterialize-databricks,-snowflake,-bigqueryand-postgresall already do, and what the boilerplate enforces by panicking if aLoadedresponse precedes
WaitForAcknowledged. Reading perLoadrequest instead corrupts mergedbindings only, and
split-during-commitis the scenario that catches it.Replaying
Acknowledgein-session was removed as a perturbation. The runtime sendsexactly one
Acknowledgeper transaction, so re-sending it inside a live session isprotocol-illegal and a real connector rightly exits on it — which is what
materialize-databricksdoes.crash-between-commitsalready asks the same questionlegitimately, crashing after
Acknowledgedwhen the connector has applied the transactionand the state update has not yet committed, so the restart replays that same
Acknowledge.NonIdempotentAcknowledgestays paired there (110 violations in the defective half).Ways this suite could have lied to itself, all now closed. A review pass looking specifically
for passes that had not been earned found seven; each is a mechanism rather than a one-off fix.
A corruption check behind a duplication exemption. Comparing a delivered document against the
one the collection holds filed a mismatch under
OracleAgreement, whichat-least-once-never-losesexempts — for duplication. So a connector altering documents intransit passed a scenario about re-delivery. Content integrity is now its own invariant, and the
partition asserts nothing exempts it rather than leaving that to review. Its neighbour was
mis-filed the other way: a reduced seq behind the collection is loss, and reporting it as an
oracle disagreement put it behind the same exemption, so a connector losing merged-path documents
passed a scenario named "never loses" as long as no account vanished outright.
Exemptions with no ceiling. An exemption states a cause, and a cause implies a volume: "one
replayed transaction" is tens of documents, measured at 40 to 100. Uncapped, the same
justification also absorbs a connector that re-delivered the whole workload.
at-least-once-never-losesnow caps its three at 500 — an order of magnitude above measurement,a tenth of the workload — and an exemption over its ceiling stops absorbing anything, so the run
fails with the violations themselves rather than a count. Ceilings are per invariant, not per
exemption: a real subject also carries the blanket monotonicity exemption, and the broadest claim
governs or a narrower ceiling would fail a subject nobody made that claim about.
An expectation read from a collection still growing. Disabling the captures is published, and a
publication returns once the spec is stored; activation carries it to the data plane afterwards.
The only guard was a plateau — two equal collection reads three seconds apart — and a capture
pausing across that window reads as a finished one.
join-after-splitfailed with 131 violationsagainst a materialization that was right: the destination held 1310 log rows where the expectation
held 1236 documents, and nothing had been duplicated. The run now waits for every capture shard's
spec to carry
disableand for no shard to report primary — the first says the runtime has beentold, the second says the transaction it was in is behind us. Waiting for the shards to disappear
does not work and was tried: a disabled shard is not deleted, its spec stays listed.
A drain that looked at the first moment it could pass. The completion gate is "at least as many
rows as the collection holds", so it was met the instant the last expected document landed —
handing the checkers a destination that a duplicate still in flight had yet to reach. One quiet
poll now confirms it, and a complete destination that never settles is handed over at the deadline
rather than erroring, since "stopped short" is the wrong thing to say about it.
A precondition nothing acted on. A repeated
(id, seq)in the collection makes the comparisonunsound — the expectation folds it to one, a reducing binding counts it twice — and was merely
recorded in a field. It now fails the run, as an unsound workload rather than a violation, because
the fault is in what the harness was given.
A gap declaration that could not be wrong. The
RuntimeGapassertion was unconditional, so theday the runtime closed a gap the suite would keep printing the old diagnosis of a run that no
longer matched it. A pass now fails too, with the opposite message — and it earned its keep on its
first outing, catching the stall change described above.
And three found earlier in the work.
An exemption nobody could audit. An exemption that suppresses nothing is paperwork rather than a
weakened guarantee, and until
each one reported its count there was no way to tell them apart. Measuring deleted three:
destination-ahead-of-checkpointandrecovery-reconciles-with-destination's monotonicityexemptions, where nothing reorders delivery, and
at-least-once-never-loses'sStandardDeltaAgreement, whose premise a symmetric replay cannot reach. Measuring also keptfive that a reading of the code said were unreachable: they suppress 21-126 violations apiece,
every run. A zero count is corroboration for deleting an exemption, never the argument — the
same scenario's conservation exemption measures zero on most runs and is load-bearing when a
transfer's two legs straddle the replayed transaction's boundary.
A connector driven into a mode it does not support and reported as defective for it — see "What
it found".
A reference connector more forgiving than the connector it models. It kept a ledger
absorbing re-delivery across staged batches, claiming to model "a real destination recognising a
staged file it has already loaded";
materialize-databricksdedupes at file granularity withfresh-UUID names, so the same row in a new file loads again. It absorbed nothing in any recorded
run, and removing it caught
NonIdempotentAcknowledgewith 754 violations where the ledger hadbeen suppressing all but 110 of them. And any harness error
in a scenario's defective half counted as the defect being caught — including a publish that
failed from stack contention — so a flaky control plane could silently vacate a pairing.
stack::PublishFailedis a typed marker in the error chain, and the defective half now failsloudly on it instead.
Reading a failure. A failing run writes
evidence.jsonbeside its trace: what each bindingdelivered, and the expectation each was compared against, per account. Diagnosing from the
violation list alone means guessing which side is wrong, and both sides are impossible to
reconstruct afterwards because the run's tasks are deleted on the way out. Three signatures are
worth recognising. Duplicates with no losses in a counted-channel scenario is the runtime gap
above — 2072 rows against 2070, the extra rows being
(id, seq)pairs the expectation holds. Adestination holding rows the expectation does not contain at all is the opposite: an expectation
read too early, which is the harness's fault and now gated. And a merged binding disagreeing with
its own delivered rows in both directions, total not conserved, while append-only bindings are
exact, is a connector reading its destination before
Flush.One check is exempt for a real subject, and the distinction matters. A table read returns
rows in no guaranteed order, which threatened every check whose meaning depends on order. Only
Monotonicityis exempted, because it is a claim about delivery order and a table scan cannotwitness it. The arithmetic checks are not exempted:
check_merged_deltaestablishes the orderit needs itself, sorting on
(id, seq), so they still hold a real subject to conservation andoracle agreement. Exempting them too would have been the easy move and a considerably weaker
suite. That exemption is deliberately uncapped, unlike the scenario-level ones: order is not
recoverable through such a read at all, so no volume of disorder would mean anything, and an
unbounded exemption is what lifts a narrower scenario ceiling for the same invariant.
What it found, including a false positive worth reading. Run against
materialize-databricks, the suite verifies every applicable scenario clean — including all sixmembership-change scenarios that existed when it was measured (
split-after-commit-before-applywas added afterwards and is so far verified on the reference connector only):
Getting there took retracting two filed issues (estuary/connectors#4986 and #4987), and the
reason is the most useful thing this exercise produced.
materialize-databricksgatesmulti-shard operation behind its
scale_outfeature flag, off by default, and the configused did not set it. Without the flag its state is not scoped by key range and no shard defers
to a primary, so two shards issue their own
COPY INTOagainst one table. The suite reporteda crash-loop and 93 silently lost documents; both were the configuration, and with the flag on
the same scenarios pass over 5,338 and 9,108 documents respectively.
So the harness could drive a connector into a mode it does not support and report the result
as the connector's fault. It cannot know a given connector's flag names, so the requirement is
documented where someone configuring a subject will read it, and this PR asks reviewers to
treat "a scenario failed against my connector" as a question about configuration first.
split-lands-on-prepared-transactionpasses against databricks, and the strength of that resultshould be stated carefully: databricks stages during
Store, so even a split landing squarely ina prepared transaction leaves nothing in its destination for the children to apply twice. The pass
is therefore consistent with the gap reaching the counted channel alone, but it is not on its own
proof that the window was reached on that run. The reference connector is what establishes the
contrast, being the only subject that can be switched between classes over the same perturbation.
Known residuals, recorded in the design doc's Deferred section rather than left to be
rediscovered.
A retried
Acknowledgearriving after the nextStartCommitwould promote a transaction the loghas not confirmed. The connector cannot detect it —
Acknowledgecarries no transaction identity —and closing it needs the protocol to say which transaction is being confirmed. Commented at the
site.
Four protocol surfaces no scenario perturbs: a crash during
Apply, which is where a connector isleast likely to be idempotent since it is written as though it runs once; a backfill counter bump;
a binding disabled and re-enabled; and a second crash during a replay, which the fired-marker
that makes a fault one-shot currently stands in the way of.
Loss that cancels itself in the reduced views. The merged bindings detect loss arithmetically, so
losing two documents of one account whose deltas cancel is invisible there; the log binding holds a
row per document and settles it exactly, which is why every run has one and why a subject without
delta-updates support is refused. What remains uncovered is a connector losing on the merged path
alone with a cancelling coincidence. Two fixes were weighed and both cost more than the hole: a
summed per-document counter would change the soak fixture this suite deliberately reuses
unmodified, and checking the oracle's set membership compares an order-dependent value that around
a membership change would need a reordering exemption as broad as monotonicity's — trading an exact
check for a suppressed one.
A note on the fidelity boundary. The shim observes and perturbs; it never synthesizes a
message the runtime did not send. The zombie is the single exception — the messages it
replays are real runtime messages, only their scheduling is the shim's — and that is the one
place where the suite's claim is weaker than "a real runtime drove this".
Suggested reading order:
scenarios.rsfor what is verified and why, thendocs/materialize/consistency-testing.mdfor the reasoning behind the design, thenreference/mod.rsfor the class state machines.