Repository navigation
[v0.21.x-branch] Backport #11413: htlcswitch: avoid per-row queries when pruning network result store - #11418
Merged
Conversation
The network result store is pruned at startup, keeping only results for payments that are still in flight. Previously, cleanStore iterated over the bucket and deleted each stale key individually. On SQL backends, each iteration step and deletion requires separate database queries. A store with N entries therefore incurred O(N) sequential client/server round trips and could make startup take hours when using remote PostgreSQL. Read retained results with one streaming query, drop and recreate the bucket, then restore the retained entries. PostgreSQL still processes O(N) rows internally, but the operation no longer performs a round trip for every stored result. Exercise cleanup across the supported kvdb backends, including partial, empty, and unchanged result sets. Add PostgreSQL benchmarks covering 1,000 and 10,000 entries with fully stale and partially retained stores. Fixes #11353. Co-authored-by: ziggie <ziggie1984@protonmail.com> (cherry picked from commit e092b72)
Contributor
|
Did a test of this backport, it seems to be working: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #11413
Supersedes #11408.
The original implementation is by @ZZiigguurraatt. Its commit is preserved as the first commit in this PR with the original authorship. The second commit fixes the PostgreSQL test setup and incorporates review feedback.
Change Description
Pruning the network result store previously performed one database operation per stored result. On SQL-backed nodes, especially those using a remote PostgreSQL database, the accumulated round trips could make startup cleanup take hours.
This change:
kvdb_postgresbuild tag; andFixes #11353.
Testing
go test ./htlcswitch -run '^TestNetworkResultStoreCleanStore -count=1go test -tags=kvdb_sqlite ./htlcswitch -run '^TestNetworkResultStoreCleanStore -count=1go test -tags=kvdb_postgres ./htlcswitch -run '^TestNetworkResultStoreCleanStore -count=1make lint pkg=htlcswitch