What happened
One run of make evidence at 98a6d24 (tag v0.3.0) failed on the mixed-fleet test; the immediate re-run passed. Roughly a coin flip over the two runs observed.
=== RUN TestFleet_HeterogeneousShardsAreWherePoolingShouldHurt
fleet_test.go:276: mixed fleet (3 loop replicas, 3 zipf replicas):
fleet_test.go:277: local 40.79% across 3 policies
fleet_test.go:278: pooled/leader 32.69% across 2 policies (-8.10 vs local)
fleet_test.go:280: pooled/shared 35.67% across 1 policies (-5.12 vs local)
fleet_test.go:283:
Error: Not equal:
expected: 1
actual : 2
Messages: leader election commits the whole fleet to one policy
--- FAIL: TestFleet_HeterogeneousShardsAreWherePoolingShouldHurt (0.62s)
Environment: darwin/arm64 (Apple M1 Max), go1.25.5, clean tree.
Why it flakes
The assert at bench/fleet_test.go:283 requires the end-of-replay snapshot of pooled.Policies to be exactly 1. But the end state races with coordination: cache epochs are wall-clock (EpochDuration: 20ms in fleetSettings()), the coordination epoch is 50ms, and a replica that has not yet adopted the leader's latest decision when the replay stops reports a second policy in use.
Ending on more than one policy is a normal transient, not a pooling defect — the coordination-ladder test in the same suite shows it in a fully passing run:
fleet_test.go:322: coordination epoch 25ms : 69.82% (-0.83 vs local), 2 policies at the end
Why it matters
make evidence is advertised as the one-command reproduction of every published number. A timing-dependent assert means the command intermittently fails on a healthy tree, which undercuts exactly the claim it exists to back.
Options
- Snapshot after a short drain: keep replaying (or just keep the clock running) for one or two coordination epochs after the workload ends, then read
Policies — asserts the steady state instead of a mid-migration frame.
- Assert convergence, not instant unanimity: e.g.
assert.LessOrEqual(t, pooled.Policies, 2) plus a strict check that the leader's published decision is single, which is what leader election actually guarantees.
- Count policies over the tail of the run (last K epochs) rather than the final instant.
The first option seems truest to the test's intent: leader election does commit the fleet to one policy, it just needs a moment to finish saying so.
What happened
One run of
make evidenceat98a6d24(tagv0.3.0) failed on the mixed-fleet test; the immediate re-run passed. Roughly a coin flip over the two runs observed.Environment: darwin/arm64 (Apple M1 Max), go1.25.5, clean tree.
Why it flakes
The assert at
bench/fleet_test.go:283requires the end-of-replay snapshot ofpooled.Policiesto be exactly 1. But the end state races with coordination: cache epochs are wall-clock (EpochDuration: 20msinfleetSettings()), the coordination epoch is 50ms, and a replica that has not yet adopted the leader's latest decision when the replay stops reports a second policy in use.Ending on more than one policy is a normal transient, not a pooling defect — the coordination-ladder test in the same suite shows it in a fully passing run:
Why it matters
make evidenceis advertised as the one-command reproduction of every published number. A timing-dependent assert means the command intermittently fails on a healthy tree, which undercuts exactly the claim it exists to back.Options
Policies— asserts the steady state instead of a mid-migration frame.assert.LessOrEqual(t, pooled.Policies, 2)plus a strict check that the leader's published decision is single, which is what leader election actually guarantees.The first option seems truest to the test's intent: leader election does commit the fleet to one policy, it just needs a moment to finish saying so.