Skip to content

feat(storage): shard every table by queue - #543

Merged
behinddwalls merged 1 commit into
mainfrom
queue-shard
Aug 7, 2026
Merged

feat(storage): shard every table by queue#543
behinddwalls merged 1 commit into
mainfrom
queue-shard

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

An audit of all 22 schema/*.sql files found five tables that are not shardable by queue: their primary key does not lead with the queue, so one queue's rows stay reachable through another queue's binding and the tables cannot be split across shards. The five are speculation_path_set, counter, request_summary, request_log, and change_uri_request_mapping.

Three of those — the gateway read-model tables — were declared permanently unshardable by submitqueue/extension/storage/storage.go and the schema README, on the grounds that their lookups "start from identifiers that arrive without queue context". This change removes that exception at its source rather than working around it: the gateway read APIs now require the queue alongside the sqid or change URI, so the queue is always available at the call site and no identifier ever has to be parsed to recover one.

The five platform messagequeue tables are deliberately out of scope. They are a message-queue backend keyed by (consumer_group, topic, partition_key), not a domain table set, and sharding them is a separate problem (a global AUTO_INCREMENT offset, and a subscriber-heartbeat table whose fair-leasing logic genuinely needs the whole live subscriber set).

What?

All 17 in-scope tables now lead their primary key with the queue.

Independent re-keys. speculation_path_set moves to (queue, head) — a batch ID is unique only within its queue, so the head alone was never a safe key. counter moves to (queue, domain) and gains the standard extension Config/Factory shape; per the extension contract the factory implementations live in the three service main.go files, not under platform/extension/.

Minted identifiers are unchanged. Counter domains simplify to bare "request" / "batch", but every emitted ID keeps its exact current format: {queue}/{seq}, {queue}/batch/{n}, and stovepipe's request/{queue}/{seq}. Stovepipe previously reused its counter domain as the ID prefix, which would have silently turned request/queueA/7 into request/7; the two are now written independently so they cannot drift into each other. Note that re-keying counter restarts every sequence at 1, so its table must be recreated in the same cutover as request, batch, request_summary, and request_log — never on its own, or freshly minted sqids will collide with surviving rows.

Queue required in the gateway contract. CancelRequest, GetRequestSummaryByIDRequest, GetRequestSummaryByChangeURIRequest, GetRequestHistoryByIDRequest, and GetRequestHistoryByChangeURIRequest each gain a queue field, validated on entry through the existing validateQueueIdentifier. Two behaviour changes follow. The change-URI lookups are now scoped to one queue, so the same URI landed into several queues needs one call per queue. And Cancel no longer overrides the caller's queue with the stored one — a mismatched queue yields NotFound, since a sqid is simply not resolvable outside its own queue.

Read-model tables re-keyed and folded in. request_summary, request_log, and change_uri_request_mapping move to queue-leading keys and join the Storage aggregate; the SetGlobalStores seam and the "deliberately not part of this aggregate" carve-out are deleted. The queue travels as an explicit field on entity.RequestLog and entity.RequestURI, stamped by producers that already know it — no ID parsing is introduced anywhere.

Enforcement. A new //tool/linter/queueshard walks the schema directories and fails if any primary key does not lead with the queue, or if any secondary index does not — a non-queue-leading index would reintroduce exactly the cross-queue access path the primary key just closed. It is wired into make lint.

Schema changes are clean recreates, not online migrations, on the schema README's statement that these tables are created empty at rollout and never backfilled.

Test Plan

  • make test — 93/93 pass, including the new linter's own tests
  • bazel test //test/integration/... — 8/8 suites pass against real MySQL, including new cross-queue isolation coverage in the counter, storage, request-URI and request-log contract suites
  • bazel test //test/e2e/... — both suites pass against the full Docker Compose stack (land → landed, plus status/history/cancel/list through the queue-scoped APIs)
  • make lint — fmt, license headers, and queueshard ("All 17 tables are shardable by queue")
  • make fmt / make gazelle / make mocks / make tidy — no drift

The e2e and image-building integration targets need --sandbox_writable_path=$HOME/.docker when run locally; make e2e-test does not pass it, which is a pre-existing local-only issue unrelated to this change.

@behinddwalls
behinddwalls marked this pull request as ready for review August 7, 2026 18:10
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 7, 2026 18:10
@behinddwalls
behinddwalls added this pull request to the merge queue Aug 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Aug 7, 2026
## Summary

### Why?

An audit of all 22 `schema/*.sql` files found five tables that are not shardable by queue: their primary key does not lead with the queue, so one queue's rows stay reachable through another queue's binding and the tables cannot be split across shards. The five are `speculation_path_set`, `counter`, `request_summary`, `request_log`, and `change_uri_request_mapping`.

Three of those — the gateway read-model tables — were declared permanently unshardable by `submitqueue/extension/storage/storage.go` and the schema README, on the grounds that their lookups "start from identifiers that arrive without queue context". This change removes that exception at its source rather than working around it: the gateway read APIs now require the queue alongside the sqid or change URI, so the queue is always available at the call site and no identifier ever has to be parsed to recover one.

The five platform `messagequeue` tables are deliberately out of scope. They are a message-queue backend keyed by `(consumer_group, topic, partition_key)`, not a domain table set, and sharding them is a separate problem (a global `AUTO_INCREMENT` offset, and a subscriber-heartbeat table whose fair-leasing logic genuinely needs the whole live subscriber set).

### What?

All 17 in-scope tables now lead their primary key with the queue.

**Independent re-keys.** `speculation_path_set` moves to `(queue, head)` — a batch ID is unique only within its queue, so the head alone was never a safe key. `counter` moves to `(queue, domain)` and gains the standard extension `Config`/`Factory` shape; per the extension contract the factory *implementations* live in the three service `main.go` files, not under `platform/extension/`.

**Minted identifiers are unchanged.** Counter domains simplify to bare `"request"` / `"batch"`, but every emitted ID keeps its exact current format: `{queue}/{seq}`, `{queue}/batch/{n}`, and stovepipe's `request/{queue}/{seq}`. Stovepipe previously reused its counter domain *as* the ID prefix, which would have silently turned `request/queueA/7` into `request/7`; the two are now written independently so they cannot drift into each other. Note that re-keying `counter` restarts every sequence at 1, so its table must be recreated in the same cutover as `request`, `batch`, `request_summary`, and `request_log` — never on its own, or freshly minted sqids will collide with surviving rows.

**Queue required in the gateway contract.** `CancelRequest`, `GetRequestSummaryByIDRequest`, `GetRequestSummaryByChangeURIRequest`, `GetRequestHistoryByIDRequest`, and `GetRequestHistoryByChangeURIRequest` each gain a `queue` field, validated on entry through the existing `validateQueueIdentifier`. Two behaviour changes follow. The change-URI lookups are now scoped to one queue, so the same URI landed into several queues needs one call per queue. And `Cancel` no longer overrides the caller's queue with the stored one — a mismatched queue yields `NotFound`, since a sqid is simply not resolvable outside its own queue.

**Read-model tables re-keyed and folded in.** `request_summary`, `request_log`, and `change_uri_request_mapping` move to queue-leading keys and join the `Storage` aggregate; the `SetGlobalStores` seam and the "deliberately not part of this aggregate" carve-out are deleted. The queue travels as an explicit field on `entity.RequestLog` and `entity.RequestURI`, stamped by producers that already know it — no ID parsing is introduced anywhere.

**Enforcement.** A new `//tool/linter/queueshard` walks the schema directories and fails if any primary key does not lead with the queue, or if any secondary index does not — a non-queue-leading index would reintroduce exactly the cross-queue access path the primary key just closed. It is wired into `make lint`.

Schema changes are clean recreates, not online migrations, on the schema README's statement that these tables are created empty at rollout and never backfilled.

## Test Plan

- ✅ `make test` — 93/93 pass, including the new linter's own tests
- ✅ `bazel test //test/integration/...` — 8/8 suites pass against real MySQL, including new cross-queue isolation coverage in the counter, storage, request-URI and request-log contract suites
- ✅ `bazel test //test/e2e/...` — both suites pass against the full Docker Compose stack (land → landed, plus status/history/cancel/list through the queue-scoped APIs)
- ✅ `make lint` — fmt, license headers, and `queueshard` ("All 17 tables are shardable by queue")
- ✅ `make fmt` / `make gazelle` / `make mocks` / `make tidy` — no drift

The e2e and image-building integration targets need `--sandbox_writable_path=$HOME/.docker` when run locally; `make e2e-test` does not pass it, which is a pre-existing local-only issue unrelated to this change.
@behinddwalls
behinddwalls merged commit 2544f02 into main Aug 7, 2026
14 checks passed
@behinddwalls
behinddwalls deleted the queue-shard branch August 7, 2026 19:00
behinddwalls added a commit that referenced this pull request Aug 7, 2026
…551)

## Summary

### Why?

Speculation never funded a path, so no build was ever dispatched and the
submitqueue e2e hung until Bazel killed it at 300s, having spilled ~11MB
of retry logs.

Two producers build an entity without naming its queue, and both of
their stores are bound to one queue and reject a write that disagrees:

- `speculate/dispatch.go` created a head's first `SpeculationPathSet` as
`{Head: batch.ID}`, so `Create` failed with `queue "" does not match the
store's bound queue`. That is the first write of every newly funded
head, so no head was ever funded.
- `build/build.go` created the `PathBuild` link as `{PathID, Attempt,
BuildID}`, so the link write failed the same way. Without the link a
path is never observed moving to `building`, so it stays `pending` and
every later run re-dispatches it.

Both stores gained their queue-bound guard when their tables were
re-keyed to lead with the queue (#543 for `speculation_path_set`, #502
for `path_build`); the producers were not updated to match.

Unit tests missed it because they use the gomock stores, which do not
enforce the binding, and the storage contract suite missed it because it
builds its own fixtures with the queue set. Only the e2e exercises a
real producer against a real queue-bound store.

### What?

Stamps the queue at both construction sites: the path set takes the
run's queue, the link takes its batch's.

Turns both into regression tests rather than leaving them to the e2e:
the build controller's link expectations are exact struct matches that
now require the queue, and the speculate run test asserts the created
set names it. Each fails against the unfixed code.

## Test Plan

✅ `go test ./submitqueue/... ./service/...`
✅ Verified each new assertion fails when its fix is reverted
✅ `make lint` / `make check-gazelle`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants