Skip to content

[BUG](sysdb): Exclude deleted collections from DLQ - #7878

Open
rescrv wants to merge 1 commit into
mainfrom
rescrv/fix-deleted-collections-dlq
Open

rescrv wants to merge 1 commit into
mainfrom
rescrv/fix-deleted-collections-dlq

Conversation

@rescrv

@rescrv rescrv commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Description of changes

A compaction that finishes after its collection is soft-deleted can increment the collection's failure count and trigger the compaction DLQ growth alert. GetDLQFailureCounts included these deleted collections indefinitely until physical cleanup.

Exclude is_deleted = true collections from the DLQ metric query. This corrects existing inflated counts as well as failures recorded after deletion. Compactor retry behavior is unchanged.

The production count rose from 23 to 24 immediately after a collection soft deleted compaction failure on October 9. Honeycomb failure history.

Test plan

  • Regression coverage for deletion from shared and unique failure buckets, a failure increment after deletion, and deletion of the last live failed collection.
  • Targeted TestCollectionDb_GetDLQFailureCounts passes against PostgreSQL via OrbStack.
  • Full collection DAO suite passes with the race detector: go test -race ./pkg/sysdb/metastore/db/dao -run '^TestCollectionDbTestSuiteSuite$' -count=1 (with DOCKER_HOST set to the OrbStack socket).

Migration plan

No schema migration or data cleanup required. Deploy the Go sysdb service; the next metric observation excludes existing soft-deleted collections.

Observability plan

Monitor compaction_dlq_size and the "Compaction failures spreading to more collections" alert after deployment. Counts may decrease as deleted collections are excluded.

Documentation Changes

Clarify that GetDLQFailureCounts counts live collections. No public API changes.

Soft-deleted collections can retain compaction failures, including
failures recorded by compactions that finish after deletion. Exclude
them from the DLQ metric so they cannot trigger false growth alerts.

Cover deletion from shared and unique buckets, failures after deletion,
and an empty live DLQ in the collection DAO regression test.

Co-authored-by: AI

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants