dynamo-ci: CausationTrackingOutboxTests reads an eventually-consistent GSI and asserts it is up to date

[iancooper]

dynamo-ci is the least reliable job on master, and every failure in the recent window is the same test class. This is a test defect rather than a product one: the assertions require an immediacy that a DynamoDB Global Secondary Index cannot promise.

The census — 5 failures in 13 master runs, all the same class

2026-09-12T14:14:54Z  run 34698723388  failure   CausationTrackingOutboxTests
2026-09-12T09:14:33Z  run 34685324878  success
2026-09-11T15:52:07Z  run 34618666177  success
2026-09-11T14:14:18Z  run 34608934318  failure   CausationTrackingOutboxTests
2026-09-11T09:13:05Z  run 34583067399  success
2026-09-10T14:27:57Z  run 34489230421  success
2026-09-10T08:00:04Z  run 34452809953  success
2026-09-10T07:37:47Z  run 34450911909  failure   CausationTrackingOutboxTests
2026-09-09T20:12:04Z  run 34399594080  success
2026-09-08T10:12:10Z  run 34214176968  success
2026-09-07T20:32:17Z  run 34159782273  success
2026-09-07T20:30:03Z  run 34159638104  failure   CausationTrackingOutboxTests
2026-09-07T16:50:22Z  run 34145065842  failure   CausationTrackingOutboxTests

5 of 5 dynamo-ci failures are Outbox.Causation.CausationTrackingOutboxTests, across five days, in both the Paramore.Brighter.DynamoDB.Tests and Paramore.Brighter.DynamoDB.V4.Tests assemblies. The named tests are When_replaying_causation_on_outbox_should_clear_dispatch_state, its _async twin, and When_replaying_causation_for_messages_deposited_in_bulk_should_clear_dispatch_state.

The failure is always the same shape:

Assert.DoesNotContain() Failure: Item found in collection

Why it cannot be made reliable as written

tests/Paramore.Brighter.Base.Test/Outbox/CausationTrackingOutboxBaseTests.cs:85-93 writes and then immediately reads back, asserting the write is already visible:

Outbox.MarkDispatched(firstWithA.Id, contextA, dispatchedAt);
Outbox.MarkDispatched(secondWithA.Id, contextA, dispatchedAt);
Outbox.MarkDispatched(messageWithB.Id, contextB, dispatchedAt);

// all three start dispatched, so none are outstanding
var outstandingBefore = Outbox.OutstandingMessages(TimeSpan.Zero, contextA).Select(m => m.Id).ToArray();
Assert.DoesNotContain(firstWithA.Id, outstandingBefore);
Assert.DoesNotContain(secondWithA.Id, outstandingBefore);

On DynamoDB, OutstandingMessages is a GSI query — src/Paramore.Brighter.Outbox.DynamoDB.V4/DynamoDbOutbox.cs:835-836 and :937-938:

IndexName = _configuration.OutstandingAllTopicsIndexName,
ConsistentRead = false,

ConsistentRead = false is not a choice that can be reversed here. DynamoDB does not support strongly consistent reads on a Global Secondary Index at all — requesting one is an error. GSI propagation after a write is asynchronous and, while usually within milliseconds, is not bounded. So a message marked dispatched can still be returned by the outstanding index for a short window, and Assert.DoesNotContain finds it.

The same class reads consistently where it can — :99 uses ConsistentRead = true for a primary-key get — which is the contrast that makes the diagnosis concrete rather than speculative.

This is not the product misbehaving

Eventual consistency on the outstanding index is inherent to the store, and the Outbox Sweeper tolerates it by design: a message that appears outstanding when it is not is re-swept, and the Inbox de-duplicates. The defect is that the test asserts an immediacy the store never offered.

Suggested fix

Something that expresses "eventually", rather than a sleep:

• Poll OutstandingMessages until the expected state is reached or a generous timeout expires, and assert on the settled result. This keeps the test meaningful — it still fails if the state never becomes correct — without asserting a propagation deadline.
• Or override the pre-Act assertion for the DynamoDB implementations, since outstandingBefore at :91-93 is arranging rather than asserting the behaviour under test. The assertions that matter are after ReplayCausation.

The bulk variant fails the same way and for the same reason.

Scope

Found while working on #4343, which is unrelated — the failures are on master without it, which is how it was attributed rather than guessed at. Not filed as a flake to be re-run: the cause is identified and it will keep recurring at roughly the observed rate until the assertions change.

[iancooper]

#4319 reported this same failure five days earlier without the diagnosis, and has been closed as a duplicate of this issue. This is now the single place for it.

Carrying over the evidence from there, since it is a different kind from this issue's master census and shouldn't be lost behind a closed issue.

#4319's contribution: it fails on branches that share nothing

Three simultaneous branches, 2026-09-07:

branch	what that branch changes	variant that failed
#4297	generated transport conformance tests	.V4
#4311	an ASB URI-parsing fix, off master	non-V4
#4317	an ASB test harness provisioning probe	.V4

None of them touches DynamoDB. A fourth sighting on #4300 was noted at the time as never diagnosed and is probably the same test. Where this issue's census shows "5 of 5 dynamo-ci failures on master are this class", #4319 shows "and it is indifferent to what the branch changes" — the same conclusion from the other direction.

A controlled pair from #4297 today

The same test failed in Paramore.Brighter.DynamoDB.V4.Tests on d8bb21176 and in Paramore.Brighter.DynamoDB.Tests on 93b27ae15 — consecutive CI runs about an hour apart, on trees identical except for two files in the Kafka test project. Same input, different variant, one run apart. That is what an unbounded GSI propagation window looks like, and it is hard to explain any other way.

Why it alternates, and why only DynamoDB fails

#4319 asked whether the two variants share a template, so for the record: they share it, and so do 17 other store test projects. Both derive from CausationTrackingOutboxBaseTests in tests/Paramore.Brighter.Base.Test/Outbox/ — where this issue's cited assertions live. grep -rln CausationTrackingOutboxBaseTests tests/ returns 19 projects.

• Alternating V3/V4: the same base test runs in both, so either can lose the race on a given run.
• Only DynamoDB failing: of those 19, only the DynamoDB outbox answers OutstandingMessages from an eventually-consistent secondary index. MySQL, MSSQL, Postgres, Sqlite and InMemory read their own writes, so the immediacy the test assumes happens to hold there — which is why this reads as a DynamoDB problem when it is really an assertion that only one store can refuse.

A single fix in the base covers both variants, and would need to keep the other 17 passing — the assertion is legitimate for a read-your-writes store, so weakening it for everyone is a cost worth stating deliberately rather than absorbing.

Housekeeping note

#4319 carried the Bug label and an assignee; this issue has neither. Worth fixing so consolidating the two doesn't lose it from a filter.