feat(storage): shard every table by queue - #543
Merged
Merged
Conversation
behinddwalls
marked this pull request as ready for review
August 7, 2026 18:10
mnoah1
approved these changes
Aug 7, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Aug 7, 2026
## Summary
### Why?
An audit of all 22 `schema/*.sql` files found five tables that are not shardable by queue: their primary key does not lead with the queue, so one queue's rows stay reachable through another queue's binding and the tables cannot be split across shards. The five are `speculation_path_set`, `counter`, `request_summary`, `request_log`, and `change_uri_request_mapping`.
Three of those — the gateway read-model tables — were declared permanently unshardable by `submitqueue/extension/storage/storage.go` and the schema README, on the grounds that their lookups "start from identifiers that arrive without queue context". This change removes that exception at its source rather than working around it: the gateway read APIs now require the queue alongside the sqid or change URI, so the queue is always available at the call site and no identifier ever has to be parsed to recover one.
The five platform `messagequeue` tables are deliberately out of scope. They are a message-queue backend keyed by `(consumer_group, topic, partition_key)`, not a domain table set, and sharding them is a separate problem (a global `AUTO_INCREMENT` offset, and a subscriber-heartbeat table whose fair-leasing logic genuinely needs the whole live subscriber set).
### What?
All 17 in-scope tables now lead their primary key with the queue.
**Independent re-keys.** `speculation_path_set` moves to `(queue, head)` — a batch ID is unique only within its queue, so the head alone was never a safe key. `counter` moves to `(queue, domain)` and gains the standard extension `Config`/`Factory` shape; per the extension contract the factory *implementations* live in the three service `main.go` files, not under `platform/extension/`.
**Minted identifiers are unchanged.** Counter domains simplify to bare `"request"` / `"batch"`, but every emitted ID keeps its exact current format: `{queue}/{seq}`, `{queue}/batch/{n}`, and stovepipe's `request/{queue}/{seq}`. Stovepipe previously reused its counter domain *as* the ID prefix, which would have silently turned `request/queueA/7` into `request/7`; the two are now written independently so they cannot drift into each other. Note that re-keying `counter` restarts every sequence at 1, so its table must be recreated in the same cutover as `request`, `batch`, `request_summary`, and `request_log` — never on its own, or freshly minted sqids will collide with surviving rows.
**Queue required in the gateway contract.** `CancelRequest`, `GetRequestSummaryByIDRequest`, `GetRequestSummaryByChangeURIRequest`, `GetRequestHistoryByIDRequest`, and `GetRequestHistoryByChangeURIRequest` each gain a `queue` field, validated on entry through the existing `validateQueueIdentifier`. Two behaviour changes follow. The change-URI lookups are now scoped to one queue, so the same URI landed into several queues needs one call per queue. And `Cancel` no longer overrides the caller's queue with the stored one — a mismatched queue yields `NotFound`, since a sqid is simply not resolvable outside its own queue.
**Read-model tables re-keyed and folded in.** `request_summary`, `request_log`, and `change_uri_request_mapping` move to queue-leading keys and join the `Storage` aggregate; the `SetGlobalStores` seam and the "deliberately not part of this aggregate" carve-out are deleted. The queue travels as an explicit field on `entity.RequestLog` and `entity.RequestURI`, stamped by producers that already know it — no ID parsing is introduced anywhere.
**Enforcement.** A new `//tool/linter/queueshard` walks the schema directories and fails if any primary key does not lead with the queue, or if any secondary index does not — a non-queue-leading index would reintroduce exactly the cross-queue access path the primary key just closed. It is wired into `make lint`.
Schema changes are clean recreates, not online migrations, on the schema README's statement that these tables are created empty at rollout and never backfilled.
## Test Plan
- ✅ `make test` — 93/93 pass, including the new linter's own tests
- ✅ `bazel test //test/integration/...` — 8/8 suites pass against real MySQL, including new cross-queue isolation coverage in the counter, storage, request-URI and request-log contract suites
- ✅ `bazel test //test/e2e/...` — both suites pass against the full Docker Compose stack (land → landed, plus status/history/cancel/list through the queue-scoped APIs)
- ✅ `make lint` — fmt, license headers, and `queueshard` ("All 17 tables are shardable by queue")
- ✅ `make fmt` / `make gazelle` / `make mocks` / `make tidy` — no drift
The e2e and image-building integration targets need `--sandbox_writable_path=$HOME/.docker` when run locally; `make e2e-test` does not pass it, which is a pre-existing local-only issue unrelated to this change.
behinddwalls
force-pushed
the
queue-shard
branch
from
August 7, 2026 18:39
bdae7c3 to
48be602
Compare
behinddwalls
temporarily deployed
to
stack-rebase
August 7, 2026 19:00 — with
GitHub Actions
Inactive
This was referenced Aug 7, 2026
behinddwalls
added a commit
that referenced
this pull request
Aug 7, 2026
…551) ## Summary ### Why? Speculation never funded a path, so no build was ever dispatched and the submitqueue e2e hung until Bazel killed it at 300s, having spilled ~11MB of retry logs. Two producers build an entity without naming its queue, and both of their stores are bound to one queue and reject a write that disagrees: - `speculate/dispatch.go` created a head's first `SpeculationPathSet` as `{Head: batch.ID}`, so `Create` failed with `queue "" does not match the store's bound queue`. That is the first write of every newly funded head, so no head was ever funded. - `build/build.go` created the `PathBuild` link as `{PathID, Attempt, BuildID}`, so the link write failed the same way. Without the link a path is never observed moving to `building`, so it stays `pending` and every later run re-dispatches it. Both stores gained their queue-bound guard when their tables were re-keyed to lead with the queue (#543 for `speculation_path_set`, #502 for `path_build`); the producers were not updated to match. Unit tests missed it because they use the gomock stores, which do not enforce the binding, and the storage contract suite missed it because it builds its own fixtures with the queue set. Only the e2e exercises a real producer against a real queue-bound store. ### What? Stamps the queue at both construction sites: the path set takes the run's queue, the link takes its batch's. Turns both into regression tests rather than leaving them to the e2e: the build controller's link expectations are exact struct matches that now require the queue, and the speculate run test asserts the created set names it. Each fails against the unfixed code. ## Test Plan ✅ `go test ./submitqueue/... ./service/...` ✅ Verified each new assertion fails when its fix is reverted ✅ `make lint` / `make check-gazelle`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
An audit of all 22
schema/*.sqlfiles found five tables that are not shardable by queue: their primary key does not lead with the queue, so one queue's rows stay reachable through another queue's binding and the tables cannot be split across shards. The five arespeculation_path_set,counter,request_summary,request_log, andchange_uri_request_mapping.Three of those — the gateway read-model tables — were declared permanently unshardable by
submitqueue/extension/storage/storage.goand the schema README, on the grounds that their lookups "start from identifiers that arrive without queue context". This change removes that exception at its source rather than working around it: the gateway read APIs now require the queue alongside the sqid or change URI, so the queue is always available at the call site and no identifier ever has to be parsed to recover one.The five platform
messagequeuetables are deliberately out of scope. They are a message-queue backend keyed by(consumer_group, topic, partition_key), not a domain table set, and sharding them is a separate problem (a globalAUTO_INCREMENToffset, and a subscriber-heartbeat table whose fair-leasing logic genuinely needs the whole live subscriber set).What?
All 17 in-scope tables now lead their primary key with the queue.
Independent re-keys.
speculation_path_setmoves to(queue, head)— a batch ID is unique only within its queue, so the head alone was never a safe key.countermoves to(queue, domain)and gains the standard extensionConfig/Factoryshape; per the extension contract the factory implementations live in the three servicemain.gofiles, not underplatform/extension/.Minted identifiers are unchanged. Counter domains simplify to bare
"request"/"batch", but every emitted ID keeps its exact current format:{queue}/{seq},{queue}/batch/{n}, and stovepipe'srequest/{queue}/{seq}. Stovepipe previously reused its counter domain as the ID prefix, which would have silently turnedrequest/queueA/7intorequest/7; the two are now written independently so they cannot drift into each other. Note that re-keyingcounterrestarts every sequence at 1, so its table must be recreated in the same cutover asrequest,batch,request_summary, andrequest_log— never on its own, or freshly minted sqids will collide with surviving rows.Queue required in the gateway contract.
CancelRequest,GetRequestSummaryByIDRequest,GetRequestSummaryByChangeURIRequest,GetRequestHistoryByIDRequest, andGetRequestHistoryByChangeURIRequesteach gain aqueuefield, validated on entry through the existingvalidateQueueIdentifier. Two behaviour changes follow. The change-URI lookups are now scoped to one queue, so the same URI landed into several queues needs one call per queue. AndCancelno longer overrides the caller's queue with the stored one — a mismatched queue yieldsNotFound, since a sqid is simply not resolvable outside its own queue.Read-model tables re-keyed and folded in.
request_summary,request_log, andchange_uri_request_mappingmove to queue-leading keys and join theStorageaggregate; theSetGlobalStoresseam and the "deliberately not part of this aggregate" carve-out are deleted. The queue travels as an explicit field onentity.RequestLogandentity.RequestURI, stamped by producers that already know it — no ID parsing is introduced anywhere.Enforcement. A new
//tool/linter/queueshardwalks the schema directories and fails if any primary key does not lead with the queue, or if any secondary index does not — a non-queue-leading index would reintroduce exactly the cross-queue access path the primary key just closed. It is wired intomake lint.Schema changes are clean recreates, not online migrations, on the schema README's statement that these tables are created empty at rollout and never backfilled.
Test Plan
make test— 93/93 pass, including the new linter's own testsbazel test //test/integration/...— 8/8 suites pass against real MySQL, including new cross-queue isolation coverage in the counter, storage, request-URI and request-log contract suitesbazel test //test/e2e/...— both suites pass against the full Docker Compose stack (land → landed, plus status/history/cancel/list through the queue-scoped APIs)make lint— fmt, license headers, andqueueshard("All 17 tables are shardable by queue")make fmt/make gazelle/make mocks/make tidy— no driftThe e2e and image-building integration targets need
--sandbox_writable_path=$HOME/.dockerwhen run locally;make e2e-testdoes not pass it, which is a pre-existing local-only issue unrelated to this change.