Skip to content

fix(cluster): refuse an unroutable command before recording anything - #133

Merged
mcatanzariti merged 2 commits into
mainfrom
claude/feed-error-pubsub
Nov 14, 2026
Merged

mcatanzariti merged 2 commits into
mainfrom
claude/feed-error-pubsub

Conversation

@mcatanzariti

Copy link
Copy Markdown
Member

Problem

A command the cluster connection cannot route (keys spanning several slots where one is required, or a command with the special request policy) was refused only when the network task fed it, after the handler had already recorded its bookkeeping and fed the commands ahead of it. Three consequences, each reproduced by a new test that fails on main:

  1. Poisoned subscription queue. A refused cross-slot SSUBSCRIBE left its pending batch at the head of the router's pending_subscriptions. Only the head is ever matched, so the confirmation of the next ssubscribe was consumed by the ghost batch (still waiting for its other channel) and that caller was never answered. This is very likely the "never gets a message" symptom of the original report.
  2. Reply desync in a pipeline. The commands fed ahead of the refused one were already written; their replies came back with no caller waiting and were handed to the next caller (a GET received the OK of an orphaned SET).
  3. Stranded batch and leaked budget. send_messages returned on the first feed error: every message behind it in the same batch stayed in the send queue until another message woke the send path (a caller on an idle client waited forever), and the failed message's byte charge was never released from the send-queue budget.

Fix

  • The cluster routing decision now lives in one classifier, route(command) -> Result<Route>, read both by internal_feed and by ClusterConnection::check_routable. What the handler accepts up front is therefore exactly what the feed serves; nothing is duplicated.
  • NetworkHandler::handle_message calls Connection::check_routable on every command of a message first, before any pub/sub bookkeeping, shedding decision or queuing. A refused message leaves no trace and writes nothing. Standalone and sentinel connections accept everything, as before.
  • A feed that still fails (broken node connection, slot no node covers) fails its own message, releases its byte charge, and lets the rest of the batch be fed and flushed.

Side effect: the CommandNotSupportedInCluster error now carries the command name in its context, like MismatchedKeySlots already did.

Known limitation

A residual feed error in the middle of a multi-command message (I/O or topology, no longer a routing refusal) can still leave earlier commands of that message written. Those errors come with a broken or reconfigured connection, which the reconnection path handles; making that case exact would need per-command rollback in the cluster request queue, which is out of scope here.

Tests

  • a_refused_cross_slot_ssubscribe_leaves_the_client_usable (pub/sub): times out on main, passes here.
  • a_pipeline_with_a_cross_slot_command_runs_nothing (cluster): desyncs on main, passes here.
  • a_feed_error_does_not_strand_the_rest_of_the_batch (cluster): uses a new one-shot ClusterTestHook::arm_feed_error_for to fail a feed past the routing check; times out on main, passes here.
  • cargo clippy --all-targets -- -D warnings (both CI feature sets), ./run_tests.sh --hermetic, and the pub/sub and cluster suites on a local 3-master cluster: no new failure.

Generated by Claude Code

mcatanzariti and others added 2 commits November 14, 2026 04:31
A command the cluster connection cannot route (keys spanning several slots
where one is required, or a command with the `special` request policy) was
refused only when the network task fed it, after the handler had already
recorded its bookkeeping and fed the commands ahead of it:

- a refused SSUBSCRIBE left its pending batch at the head of the router's
  subscription queue; the next subscription's confirmation was matched
  against it and its caller was never answered;
- in a pipeline, the commands fed ahead of the refused one were written and
  their replies were handed to the next callers;
- `send_messages` returned on the error, stranding every message behind it
  in the send queue until something else woke the send path, and leaking the
  refused message's byte charge from the send-queue budget.

The routing decision now lives in one classifier, `route`, read both by the
feed path and by `check_routable`, which the handler calls before anything
is recorded or queued. A feed that still fails (a broken node connection, a
slot no node covers) fails its own message, releases its charge, and lets
the rest of the batch be fed and flushed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo8jBqM8KTsnpJz9oRkK5G
`a_read_goes_to_a_replica_when_the_read_preference_asks_for_it` polls the
replica until the value written to the master has reached it, but read it
as a `String`. Until replication lands, the replica answers nil, which a
`String` refuses with `UnexpectedNil`, and the `?` ended the test on the
first attempt: the retry loop never ran. Seen on CI as `Received response
Null` followed by the failure. The value is now read as an `Option<String>`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo8jBqM8KTsnpJz9oRkK5G

Copy link
Copy Markdown
Member Author

CI on 0b3af53 failed one test out of 1216: cluster_server::a_read_goes_to_a_replica_when_the_read_preference_asks_for_it. It is not caused by this diff (which does not touch replica reads or single-shard routing), but it is not a flake either: the test's replication-wait loop read the value as a String, so the first Null from a not-yet-replicated replica raised UnexpectedNil and the ? ended the test before any retry. The log shows exactly Received response Null followed by that error.

Fixed in b2d29ec by reading an Option<String>, so the loop now actually waits for replication.


Generated by Claude Code

@mcatanzariti
mcatanzariti merged commit 1f14ea7 into main Nov 14, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant