Skip to content

Add reservation benchmark - #151

Open
jerbaroo wants to merge 7 commits into
masterfrom
jerbaroo-benchmark
Open

Add reservation benchmark#151
jerbaroo wants to merge 7 commits into
masterfrom
jerbaroo-benchmark

Conversation

@jerbaroo

@jerbaroo jerbaroo commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

This PR adds a benchmark which can be run with just bench-chunks-select. It produces the SVG contained in this PR. Output from the Rust bench is written to a CSV (not committed) which is then read by Python to produce a graph. The Rust bench also prints the information to stdout, which looks like:

SHAPE                          STRATEGY                            SIZE         MEDIAN     P10 / P90
---------------------------------------------------------------------------------------------------------
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=100:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 10
Beginning benchmark of strategy Random
FewSubmissionsManyChunks       Random                                 100          56.5       51.4 / 65.9
Beginning benchmark of strategy PreferDistinct { meta_key: "metadata_value", underlying: Oldest }
FewSubmissionsManyChunks       PreferDistinct(metadata_value, Oldest) 100          111.1      103.1 / 130.8
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=500:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 50
Beginning benchmark of strategy Random
FewSubmissionsManyChunks       Random                                 500          55.8       53.1 / 66.6
Beginning benchmark of strategy PreferDistinct { meta_key: "metadata_value", underlying: Oldest }
FewSubmissionsManyChunks       PreferDistinct(metadata_value, Oldest) 500          182.6      174.3 / 256.1
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=1000:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 100
Beginning benchmark of strategy Random
FewSubmissionsManyChunks       Random                                 1000         55.9       52.1 / 63.2
Beginning benchmark of strategy PreferDistinct { meta_key: "metadata_value", underlying: Oldest }
FewSubmissionsManyChunks       PreferDistinct(metadata_value, Oldest) 1000         344.3      260.8 / 384.3
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=2000:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 200
Beginning benchmark of strategy Random
FewSubmissionsManyChunks       Random                                 2000         55.3       50.9 / 62.7
Beginning benchmark of strategy PreferDistinct { meta_key: "metadata_value", underlying: Oldest }
FewSubmissionsManyChunks       PreferDistinct(metadata_value, Oldest) 2000         499.6      438.2 / 681.2
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=4000:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 400
Beginning benchmark of strategy Random

@ReinierMaas ReinierMaas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't review opsqueue/benches/chunks_select.rs yet, only the top block comment. But I think I already have some good advice that can be helpful.

Comment thread justfile Outdated
Comment thread default.nix
Comment thread opsqueue/benches/plot_chunks_select.py
Comment thread opsqueue/benches/chunks_select.rs Outdated
@jerbaroo
jerbaroo force-pushed the jerbaroo-benchmark branch from 46043aa to bde9a83 Compare July 28, 2026 16:28
@jerbaroo
jerbaroo requested a review from ReinierMaas July 30, 2026 08:46
@jerbaroo
jerbaroo force-pushed the jerbaroo-benchmark branch from 2dfa1ef to dfed300 Compare July 30, 2026 09:50

@ReinierMaas ReinierMaas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I left a few smaller changes you could still make.

Comment thread justfile Outdated
Comment thread opsqueue/benches/chunks_select.rs Outdated
dispatcher: &Dispatcher,
) -> ChunkId {
use tokio::sync::mpsc::unbounded_channel;
let (notifier, _) = unbounded_channel();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using an underscore here _ drop the receiving side instantly. The sending side will receive errors when it tries to send anything over this channel. I haven't looked at the underlying code but does the reservation hold if we drop this receiving side of the channel? We could opt to return the receiving side together with the ChunkId and retain them until we finish our reservation to ensure that we can receive messages from the view of the reserver.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reservation holds if we drop the receiving side of the channel.

The sending side of the channel is not used during the reservation except for being stored in the Reserver so that it can be used when the reservation expires.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On reservation timeout if a sending error occurs it is just dropped:

                    // Only error case is if receiver is no longer listening
                    // In that case, nobody cares about the value being evicted anymore.
                    // So `let _ =` is correct here.
                    let _ = val.1.send(val.0);

Comment on lines +217 to +218
if let [(chunk, _)] = reserved.as_slice() {
ChunkId::from((chunk.submission_id, chunk.chunk_index))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are we dropping here? Should we retain that until we finish our chunk reservation?

@jerbaroo jerbaroo Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fetch_and_reserve_chunks the stream of chunk that are successfully reserved are joined with submission info. reserved is a Vec<(Chunk, Submission)>, but here with as_slice the second value of the tuple is only a reference to the Submission, so nothing is dropped.

Comment thread opsqueue/benches/chunks_select.rs Outdated
Comment thread opsqueue/benches/chunks_select.rs Outdated
@jerbaroo
jerbaroo force-pushed the jerbaroo-benchmark branch from 05e3cbc to 080f986 Compare August 6, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants