Skip to content

Retry the final key TTL after a failover - #2691

Merged
amankrx merged 1 commit into
TraceMachina:mainfrom
amankrx:fix/redis-ttl-expire-retry
Aug 14, 2026
Merged

Retry the final key TTL after a failover#2691
amankrx merged 1 commit into
TraceMachina:mainfrom
amankrx:fix/redis-ttl-expire-retry

Conversation

@amankrx

@amankrx amankrx commented Aug 14, 2026

Copy link
Copy Markdown
Member

What and why

Follow-up to #2688. @corcillo asked for the reconnect-and-retry on both
EXPIRE calls; only the temp-key one got it and I mistakenly reported both as
done. The final-key EXPIRE still failed outright on a transient failover,
while every other step in update reconnects and retries once.

Same change routes it through ttl_seconds(), so the clamp to at least one
second now covers both sites rather than just the temp key. Not reachable
today since config rejects zero, but it was meant to be defence in depth at
both.

Also corrected the comment above it, which still described the pre-review
design where only the final key carried a TTL.

How this was verified

Added update_retries_the_ttl_after_a_failover: the mock returns a connection
reset on the first EXPIRE of the real key and succeeds on the retry. It
fails with the retry arm removed, with the error the reviewer predicted:

Error { code: Unavailable, messages: ["Io: connection reset by peer",
  "While setting TTL on 3031...30-2 in RedisStore::update()"] }

Full store suite (30 binaries) and clippy are clean.

Risk

Low. One more round trip only on a failover that would previously have failed
the write.

The retry is bounded to a single attempt, matching the rename directly above
it, so a persistently unreachable master still surfaces an error rather than
looping.


This change is Reviewable

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nativelink Ready Ready Preview Aug 14, 2026 5:20am
nativelink-aidm Ready Ready Preview Aug 14, 2026 5:20am

Request Review

@amankrx
amankrx force-pushed the fix/redis-ttl-expire-retry branch from 87c89ca to ce9aada Compare August 14, 2026 05:19
@amankrx
amankrx merged commit 3f1fb5d into TraceMachina:main Aug 14, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants