Acknowledge a result the router has already finished with

[partouf]

Why

When a result arrives, handleMessage acks it, unsubscribes, and drops the subscription. If that ack never reaches the worker — lost, or the sender's connection already gone (Failed to send ack … disconnected) — the worker resends 3s later. By then there is no subscription to match, so the message falls off the end of the if (subscription) and is dropped without an ack. The worker cannot distinguish that from "nobody was listening", so it spends its whole retry budget, about 9s, on a result the user already has.

That is not free. isReadyForNewMessages() requires pendingAcks to be empty, and both worker threads share one PersistentEventsSender, so those 9s stop the entire instance pulling from SQS. The queue deepens, more results pass their deadline, and more of them end up back on this same path — the amplifier described in infra/docs/ce-router-failure-atlas.md §5.2, which also recommends exactly this fix.

Measured on prod, in an 85-minute window, orphaned results split as:

pattern	count	meaning
D F F F	7	delivered, then three pointless resends — this PR
never delivered	9	result never reached the router; unaffected

What this does

Guids the router has served, or given up waiting for, are remembered for two minutes. A retransmission of one is acked and discarded. The worker is then released on its first retry instead of never, cutting each occurrence from ~12s of ack-gate stall to ~3s.

Two minutes clears the longest window a stale retransmission can appear in — the 60s requester deadline plus the worker's ~9s of retries — with room to spare. At prod's 2-3 compiles/s per router that is a few hundred entries, each a guid and a timestamp, pruned from the front on insert.

The waitForResult timeout records the guid too, so a compile that finishes after the router gave up is acked rather than retried into a void. compiler-explorer#9160 already makes the worker give up quickly there, so that half is belt-and-braces.

What this does not fix

Anything where the result never reached the router: the never-delivered cases above, and a separate guid seen resending 104 times over 161 minutes against a maxRetries = 3 budget, which remains unexplained.

Testing

typecheck and lint clean, 7/7 tests, three new: re-ack after serving, no ack for a guid never served, and no ack once the TTL has lapsed.

🤖 Generated with Claude Code