Add hot storage layer and storage benchmarks by mo4islona · Pull Request #2 · subsquid/sqd-query-engine

mo4islona · 2026-03-21T11:16:35Z

Summary

Hot storage architecture: memory buffer + IPC crash log + sorted parquet flush + dataset store
Legacy sqd_storage backend optimized with two-phase reads, stats pruning, fused masks
Comprehensive benchmarks across all storage tiers (GCP 30-vCPU, CPU=1-30)
Hot storage plan with rationale, memory estimation, and implementation phases

Storage Tiers

Tier	Format	Use Case
Memory	Arrow RecordBatch	Chain tip, immediate queries
Spillover	Sorted parquet (8K RGs)	Unfinalized overflow
Cold	Sorted parquet (mmap)	Historical data

Benchmark Results (CPU=30, vs Legacy+RocksDB)

Query	Legacy	Best New	Speedup
evm/usdc_transfers	196 rps	768 rps	3.9x
evm/traces+diffs	55 rps	118 rps	2.1x
sol/whirlpool	206 rps	716 rps	3.5x
sol/all_blocks	17K rps	200K rps	11.5x

New engine wins all 9/9 queries (1.7x-11.5x faster).

New Files

src/scan/memory_backend.rs — MemoryChunkReader
src/scan/composite_reader.rs — CompositeChunkReader with block range routing
src/scan/crash_log.rs — IPC crash log writer + recovery
src/scan/parquet_writer.rs — Sorted parquet flush
src/scan/dataset_store.rs — Orchestrates all tiers
benches/hot_bench/ — Hot storage benchmark
docs/hot-storage-plan.md — Architecture plan with rationale

Test plan

112 tests passing (82 existing + 30 new)
Benchmarks on 12-core x86_64 and 30-vCPU GCP
Crash recovery, reorg, spillover, multi-chunk query tests

🤖 Generated with Claude Code

Copilot

Pull request overview

This PR introduces a new “hot storage” stack for the query engine (memory buffer + crash log durability + parquet spill/compact) and adds benchmarking harnesses to compare storage tiers/backends against legacy baselines.

Changes:

Added hot storage components: in-memory ChunkReader, IPC crash log, parquet flush writer, composite tier reader, and DatasetStore orchestrator.
Added alternative storage backends (LMDB, RocksDB) plus an optimized legacy sqd_storage adapter for the new engine.
Added/updated benchmark suites and documentation; removed some legacy fixture/docs artifacts and adjusted metadata weights.

Reviewed changes

Copilot reviewed 31 out of 33 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
tests/e2e_fixtures.rs	Removes production-pattern fixture registrations from e2e fixtures list.
TASK.md	Updates project task description wording/formatting.
src/scan/scanner.rs	Exposes internal filter fields/functions to `pub(crate)` for reuse by new backends.
src/scan/rocks_backend.rs	Adds RocksDB-backed `ChunkReader` storing Arrow IPC per block.
src/scan/predicate.rs	Changes boolean OR kernel usage and removes NULL-propagation regression tests.
src/scan/parquet_writer.rs	Adds sorted parquet flush implementation for memory → disk spill/compact.
src/scan/mod.rs	Registers new scan modules and feature-gated backends.
src/scan/memory_backend.rs	Adds in-memory `ChunkReader` hot buffer with reorg/drain utilities.
src/scan/lmdb_backend.rs	Adds LMDB-backed `ChunkReader` storing Arrow IPC per block (optionally zstd).
src/scan/legacy_backend.rs	Adds legacy `sqd_storage` adapter with two-phase reads and stats pruning.
src/scan/kv_scan.rs	Adds shared scan/filter pipeline for KV backends (LMDB/RocksDB).
src/scan/dataset_store.rs	Adds DatasetStore orchestrating memory + crash log + parquet chunks and routing.
src/scan/crash_log.rs	Adds IPC crash log writer + recovery logic for durability across restarts/reorgs.
src/scan/composite_reader.rs	Adds tier-merging `ChunkReader` with block-range routing optimization.
src/scan/chunk.rs	Adds ParquetChunkReader `table_names()` and `read_all()` helpers for loaders/benches.
src/output/weight.rs	Changes weight limiting column resolution to use `resolve_output_columns` helpers.
src/bin/generate_fixtures.rs	Removes legacy fixture generation binary.
README.md	Updates README usage snippet and notes about cached ParquetTable reuse.
QUERY_PATTERNS.md	Removes query-pattern analysis document.
metadata/solana.yaml	Adjusts Solana metadata (notably removes `weight: 0` for `a7`).
metadata/evm.yaml	Adjusts EVM metadata (removes weights for `logs_bloom` and `extra_data`).
LEGACY_BENCH.md	Adds legacy engine benchmark write-up (parquet vs `sqd_storage`).
docs/weight-calculation.md	Removes weight calculation documentation.
docs/hot-storage-plan.md	Adds detailed hot storage architecture/plan and benchmark rationale.
Cargo.toml	Adds new feature flags and benches for storage/hot/legacy benchmarking.
BENCHMARKS.md	Expands benchmark results and adds hot storage benchmark run instructions.
benches/storage_bench/main.rs	Adds multi-backend benchmark runner (legacy + new engine variants).
benches/storage_bench/loader.rs	Adds loaders to populate LMDB/RocksDB/legacy storage from parquet fixtures.
benches/legacy_bench/main.rs	Adds legacy-only benchmark comparing legacy parquet vs legacy `sqd_storage`.
benches/legacy_bench/loader.rs	Adds loader for legacy `sqd_storage` benchmark datasets.
benches/hot_bench/main.rs	Adds hot-tier benchmark comparing memory/spillover/parquet vs legacy backends.
benches/hot_bench/loader.rs	Adds loaders for memory + spillover parquet tiers and legacy storage for hot bench.

Comments suppressed due to low confidence (2)

src/scan/predicate.rs:555

Using Arrow's boolean or() here changes NULL semantics (e.g., true OR null becomes null), and Arrow filtering treats nulls as false. This can incorrectly drop rows when OR-ing predicates across different nullable columns. Use or_kleene() (or otherwise ensure true OR null = true) and add a regression test covering mixed-column nulls.

impl ArrayPredicate for OrPredicate {
    fn evaluate(&self, array: &dyn Array) -> BooleanArray {
        let mut result: Option<BooleanArray> = None;
        for pred in &self.predicates {
            let mask = pred.evaluate(array);
            result = Some(match result {
                None => mask,
                Some(prev) => or(&prev, &mask).unwrap(),
            });
        }
        result.unwrap_or_else(|| BooleanArray::from(vec![false; array.len()]))
    }

metadata/solana.yaml:232

instructions.a7 no longer has weight: 0 while other account columns a1..a6/a8..a15 and rest_accounts are explicitly weight=0. This inconsistency will cause a7 to default to 32 bytes/row and can change weight limiting behavior unexpectedly. If a7 is intended to be "free" like the other account columns, restore weight: 0; otherwise consider documenting why it differs.

        weight: 0
      a6:
        type: string
        json_encoding: base58
        dictionary: true
        weight: 0
      a7:
        type: string
        json_encoding: base58
      a8:
        type: string
        json_encoding: base58
        dictionary: true
        weight: 0

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-21T11:22:18Z

 /// Combine multiple row predicates with OR (multiple request items).
 pub fn or_row_predicates(predicates: &[&RowPredicate], batch: &RecordBatch) -> BooleanArray {
    let mut result: Option<BooleanArray> = None;
    for pred in predicates {
        let mask = pred.evaluate(batch);
        result = Some(match result {
            None => mask,
-            Some(prev) => or_kleene(&prev, &mask).unwrap(),
+            Some(prev) => or(&prev, &mask).unwrap(),
        });
    }
    result.unwrap_or_else(|| BooleanArray::from(vec![false; batch.num_rows()]))


Same NULL-propagation issue as above: or() will yield null for true OR null, and filter_record_batch will exclude those rows. This breaks OR-combining RowPredicates when different predicates reference different nullable columns. Switch back to Kleene OR semantics and restore a test that asserts the expected mask behavior with NULLs.

Copilot · 2026-03-21T11:22:18Z

+        let log = if let Some(log) = self.tables.get_mut(table) {
+            log
+        } else {
+            let path = self.dir.join(format!("{}.ipc", table));
+            let file = OpenOptions::new()
+                .create(true)
+                .write(true)
+                .truncate(true)
+                .open(&path)
+                .with_context(|| format!("creating crash log file {}", path.display()))?;
+            let buf_writer = BufWriter::new(file);
+            let writer = StreamWriter::try_new(buf_writer, &batch.schema())
+                .context("creating IPC stream writer")?;
+
+            self.tables.entry(table.to_string()).or_insert(TableLog {
+                path,
+                writer,
+                block_offsets: Vec::new(),
+            })
+        };


OpenOptions::truncate(true) will wipe any existing crash log file the first time a table is appended after a restart (since CrashLogWriter::open() doesn't preload existing logs into self.tables). That can silently drop recovered-but-uncompacted blocks from durability, so a subsequent crash would lose data. Consider either (a) clearing + rewriting the crash log from the recovered in-memory buffer on startup, or (b) opening existing files without truncation and rotating to a new file format that supports append safely.

Copilot · 2026-03-21T11:22:18Z

+    // Group by block_number across all tables.
+    // Each IPC file has one RecordBatch per block, in order.
+    // We need to find the block_number column to correlate.
+    // Since blocks arrive in order and each batch = one block,
+    // we zip by index: batch[0] from all tables = block 0, etc.
+
+    // Find max batch count across tables
+    let max_batches = table_batches.values().map(|v| v.len()).max().unwrap_or(0);
+
+    let mut result: Vec<(u64, HashMap<String, RecordBatch>)> = Vec::new();
+
+    for i in 0..max_batches {
+        let mut block_tables: HashMap<String, RecordBatch> = HashMap::new();
+        let mut block_number: Option<u64> = None;
+
+        for (table_name, batches) in &table_batches {
+            if i < batches.len() {
+                let batch = &batches[i];
+
+                // Try to extract block_number from the batch
+                if block_number.is_none() {
+                    block_number = extract_block_number(batch, table_name);
+                }
+
+                block_tables.insert(table_name.clone(), batch.clone());
+            }
+        }
+
+        if !block_tables.is_empty() {
+            let bn = block_number.unwrap_or(i as u64);
+            result.push((bn, block_tables));
+        }
+    }


Crash recovery currently "zips" batches across tables by index. This is incorrect whenever a table has zero rows for a block (append skips empty batches), or when tables naturally have sparse presence per block—batch indices will drift and you'll attach a table's batch to the wrong block_number. Recovery should instead group per table by the actual block_number extracted from each batch (e.g., build a map block_number -> {table -> batch}) and merge by key.

Copilot · 2026-03-21T11:22:19Z

+                tables,
+            });
+        }
+


After recover_crash_log() rebuilds the in-memory buffer, the crash log on disk is left as-is. Given the current CrashLogWriter behavior (table logs opened lazily), subsequent appends can truncate/overwrite existing .ipc files, and recovery itself may need to normalize sparse tables. Consider clearing and rewriting the crash log from self.buffer at the end of recover() so the on-disk log is guaranteed to represent the current buffer state for the next restart.

Suggested change

// After rebuilding the in-memory buffer, normalize the on-disk crash log

// so that subsequent appends start from a clean state and cannot

// truncate or overwrite stale `.ipc` files from previous runs.

if crash_log_dir.exists() {

std::fs::remove_dir_all(&crash_log_dir)

.with_context(|| format!("clearing crash log dir {}", crash_log_dir.display()))?;

}

std::fs::create_dir_all(&crash_log_dir)

.with_context(|| format!("recreating crash log dir {}", crash_log_dir.display()))?;

Copilot · 2026-03-21T11:22:19Z

+        let mut all_batches = Vec::new();
+
+        let iter = self.db.iterator_cf(&cf, rocksdb::IteratorMode::Start);
+        for result in iter {
+            let (_key, value) = result?;
+            let batches = decode_batches(&value)?;
+            all_batches.extend(batches);
+        }
+
+        apply_scan_filters(all_batches, request, schema)


scan() currently iterates the entire column family from IteratorMode::Start, decodes every IPC value, and only then applies from_block/to_block filtering. For large datasets this negates the benefit of a block-keyed KV layout and makes "latest" range queries O(total_blocks). Use an iterator starting at block_key(from_block) (when provided) and stop once the key exceeds to_block, so only the relevant blocks are decoded.

Copilot · 2026-03-21T11:22:19Z

+    // Build sort columns
+    let sort_cols: Vec<SortColumn> = sort_columns
+        .iter()
+        .filter_map(|name| {
+            batch.column_by_name(name).map(|col| SortColumn {
+                values: col.clone(),
+                options: None, // ascending, nulls last (default)
+            })
+        })
+        .collect();
+
+    if sort_cols.is_empty() {
+        return Ok(batch.clone());
+    }


sort_batch() silently drops sort key columns that aren't found in the batch via filter_map. If metadata is wrong or a table schema changes, this will still write a parquet file but with an incomplete/incorrect sort order, which can hurt row-group pruning and correctness assumptions downstream. Consider returning an error when any requested sort key column is missing, or at least logging/metrics so it can't fail silently.

Copilot · 2026-03-21T11:22:20Z

        type: string
        json_encoding: hex
      logs_bloom:
        type: string
        json_encoding: hex
-        weight: 512
      sha3_uncles:
        type: string
        json_encoding: hex
      extra_data:
        type: string
        json_encoding: hex


The weight annotations for blocks.logs_bloom and blocks.extra_data were removed. This will make compute_weight_params() fall back to the default fixed weight (32) for these columns, which can severely under-estimate response size (logs_bloom should be 512 bytes/row; extra_data should use extra_data_size). That can cause responses to exceed the intended MAX_RESPONSE_BYTES limit. Please restore the correct weights or adjust weight limiting logic accordingly.

Copilot · 2026-03-21T11:22:20Z

 ```rust
 use sqd_query_engine::metadata::loader::load_dataset_description;
 use sqd_query_engine::query::{parse::parse_query, plan::compile};
 use sqd_query_engine::output::assembly::execute_plan;

-let meta = load_dataset_description(Path::new("metadata/evm.yaml"))?;
-let parsed = parse_query(query_json, &meta)?;
-let plan = compile(&parsed, &meta)?;
-let result = execute_plan(&plan, &meta, chunk_dir, Vec::new())?;
+let meta = load_dataset_description(Path::new("metadata/evm.yaml")) ?;
+let parsed = parse_query(query_json, & meta) ?;
+let plan = compile( & parsed, & meta) ?;
+let result = execute_plan( & plan, & meta, chunk_dir, Vec::new()) ?;
+```
+
+Use `execute_plan_cached()` to reuse `ParquetTable` instances across calls:
+
+```rust
+use sqd_query_engine::output::assembly::execute_plan_cached;
+
+let mut cache: HashMap<String, ParquetTable> = HashMap::new();
+let result = execute_plan_cached( & plan, & meta, chunk_dir, Vec::new(), & mut cache) ?;
 ```


The Rust examples in the README have extra spaces around ?, &, and parentheses (e.g., load_dataset_description(...) ?;, & meta), which makes the snippet not compile when copied. Please format these examples as valid Rust (and include missing imports like std::path::Path, std::collections::HashMap, and the ParquetTable type if referenced).

Copilot · 2026-03-21T11:22:20Z

+    eprintln!("\n=== Verifying correctness ===");
+    for (case, is_evm) in &all_cases {
+        // Use parquet as reference
+        let ref_backend = if *is_evm { &backends[1].evm } else { &backends[1].sol };
+        let ref_out = ref_backend.as_ref().map(|b| b.run_query(case)).unwrap_or_default();
+
+        let mut ok = true;
+        for backend in &backends {
+            if backend.name == "New+Parquet" { continue; }
+            let b = if *is_evm { &backend.evm } else { &backend.sol };
+            let out = b.as_ref().map(|b| b.run_query(case)).unwrap_or_default();
+            if out != ref_out {
+                eprintln!("  MISMATCH {}: {} ({} bytes) vs Parquet ({} bytes)", case.name, backend.name, out.len(), ref_out.len());
+                ok = false;
+            }


The correctness verification says "Use parquet as reference" but ref_backend is taken from backends[1] (Leg+RocksDB), and the loop skips New+Parquet rather than comparing everything against the actual parquet reference. This can hide discrepancies and the mismatch message also labels it as "vs Parquet" incorrectly. Use backends[0] (Leg+Parquet) or backends[2] (New+Parquet) consistently as the reference, and compare all other backends against it.

Copilot · 2026-03-21T11:22:20Z

        let table_desc = metadata.table(&table_plan.table);

-        let weight_cols = weight_projection(&table_plan.output_columns, table_desc);
-        let (fixed_weight, weight_col_names) = compute_weight_params(&weight_cols, table_desc);
+        let resolved_cols = resolve_output_columns(table_plan, table_desc.unwrap());
+        let (fixed_weight, weight_cols) = compute_weight_params(&resolved_cols, table_desc);



Weight calculation now uses resolve_output_columns(...) to decide which columns contribute to weight. This includes non-user columns (join keys, tag column, and relation source-predicate columns) that legacy weight limiting typically excluded, so it can change which blocks fit under MAX_RESPONSE_BYTES. If legacy-compatible sizing is still required, consider computing weight from a dedicated weight-projection (primary key + user-requested outputs) and keep resolve_output_columns only for scan/projection needs; otherwise update docs/tests to reflect the new semantics.

…nics Addresses ISSUES.md #2, #3, #13, #19: - Fix Int32→u64 sign-extension in block number reads across all files (scanner stats, weight, block_index). Parquet stores UInt32 as Int32 physical type; reinterpret via u32 to avoid sign-extension (#19) - Remove silent Fallback variant from semi_join TypedExtractor; error on unsupported join key types instead of producing wrong keys (#2) - Add else-error branch to hierarchical make_group_key to prevent unsupported column types from silently collapsing groups (#3) - Cache true_count() result instead of calling it twice per batch in semi_join and hierarchical join loops (#13) - Replace .unwrap() with graceful let-else in predicate evaluate() to return all-false instead of panicking on type mismatch - Pre-build HashSet<Vec<u8>> for FixedSizeBinary InList predicates to avoid per-row-group set reconstruction - Extract read_block_number() helper in block_index.rs to deduplicate the Int32/UInt32/Int64/UInt64 downcast chains Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

New storage architecture replacing sqd_storage (RocksDB): - MemoryChunkReader: in-memory buffer, blocks queryable immediately - CompositeChunkReader: merges tiers with block range routing - CrashLogWriter: IPC append-only durability (never queried) - ParquetWriter: flush to sorted compressed parquet (8K row groups) - DatasetStore: orchestrates all tiers with reorg, compaction, spillover, eviction Legacy backend optimizations: - Two-phase read (filter columns first, data columns for matches only) - Stats-based row filtering (predicates + block range + key filter blocks) - Fused arrow masks, direct RangeList construction Benchmarks (CPU=12, vs Legacy+RocksDB): - New+Memory: 8/9 wins (up to 2.1x faster) - New+Spillover: 9/9 wins (up to 6.5x faster) - New+Parquet cold: 8/9 wins (up to 8.6x faster) Storage backends: LMDB, RocksDB (Arrow IPC), legacy sqd_storage adapters. 112 tests passing (82 existing + 30 new). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Replace per-table IPC crash log with single-file WAL (append-only, faster) - Atomic parquet writes: write to .tmp- dir, rename on success (prevents corrupt chunks on crash) - Clean up .tmp- dirs on startup - Per-dataset compact_threshold config (human-readable: "800MB", "1GB") - Remove maybe_spillover (was unsafe: could write unfinalized blocks to parquet, causing silent data corruption on reorg) - Rename memory_cap → memory_warning (monitoring-only, no longer controls spillover) - Add first_block() and memory_block_count() accessors to DatasetStore - Fix integration_store tests to use numeric accessors - Add deep fork test (Test 11) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…racking and stat cast - Remove panicking unwrap() on array downcasts in predicates, return all-false on type mismatch - Pre-build HashSet<Vec<u8>> for FixedSizeBinary InList predicates (avoids per-row-group rebuild) - Remove Fallback variant from TypedExtractor in semi_join, error on unsupported types - Fix Int32→u64 stat cast to reinterpret via u32 (correct UInt32 row group pruning) - Fix WAL: track logical byte offsets instead of fstat() on flush, fix table count header bug - Replace /dev/null hack with Option<BufWriter> for WAL truncation/clear - Update OVH benchmark tables with latest results (+18% sol/instr+bal throughput) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…t overflow - #1/#4: Skip null source addresses in hierarchical join build side (null was indexed as [] which prefix-matches everything) - #6: ParquetTable::read now errors on missing columns instead of silently dropping them - #8: Use u32::try_from instead of truncating cast in ListUInt32 filters - #14: O(n²) → O(n) column dedup in order_columns_by_metadata via HashSet - #18: Use saturating_add for weight accumulation to prevent u64 overflow - #20: Convert rg_indices to HashSet for O(1) row group selection - #22: Add join key arity check at the top of semi_join Regression tests for null source address, key arity mismatch, and missing column errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…edup Semi-join hot path (#9, #10): - TypedExtractor now stores downcast array references (lifetime-tied to batch) instead of column indices — eliminates per-row column() + downcast_ref() overhead - Probe side reuses a scratch buffer and looks up via Borrow<[u8]> on CompositeKey — zero heap allocation per probe row Hierarchical joins (#12): - Source addresses stored in HashSet<Vec<u32>> instead of Vec, eliminating duplicate entries that multiplied prefix scan cost Encoder (#15, #16): - encode_list resolves element encoder once before the loop instead of per-element DataType match - encode_struct pre-resolves field encoders and camelCase key prefixes per call instead of per-row per-field Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Use u32::try_from instead of wrapping cast — negative i32 values are dropped instead of wrapping to huge u32 (e.g. -1 → 4294967295). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…columns - GroupKeyCol enum downcasts once per batch (same pattern as TypedExtractor) - Probe side reuses scratch buffer, lookups via Borrow<[u8]> on GroupKey - Fix #5: negative Int32 address elements dropped via u32::try_from Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Query Engine x86_64 throughput improved across all benchmarks: - General queries median: 44% → 79% faster at CPU=12 - Full block scans: 440% → 505% faster at CPU=12 - sol/instr+balances: 118% → 130% faster - sol/hard: 86% → 89% faster Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Full hot_bench re-run on OVH Xeon E-2136 after perf optimizations. Latency uses divan numbers (more stable than hot_bench latency). Throughput at CPU=1,4,8,12 from hot_bench (all 5 backends same run). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

All numbers from single hot_bench run on OVH Xeon E-2136 after drop_caches. Consistent with previous sessions at all CPU levels. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…nics Addresses ISSUES.md #2, #3, #13, #19: - Fix Int32→u64 sign-extension in block number reads across all files (scanner stats, weight, block_index). Parquet stores UInt32 as Int32 physical type; reinterpret via u32 to avoid sign-extension (#19) - Remove silent Fallback variant from semi_join TypedExtractor; error on unsupported join key types instead of producing wrong keys (#2) - Add else-error branch to hierarchical make_group_key to prevent unsupported column types from silently collapsing groups (#3) - Cache true_count() result instead of calling it twice per batch in semi_join and hierarchical join loops (#13) - Replace .unwrap() with graceful let-else in predicate evaluate() to return all-false instead of panicking on type mismatch - Pre-build HashSet<Vec<u8>> for FixedSizeBinary InList predicates to avoid per-row-group set reconstruction - Extract read_block_number() helper in block_index.rs to deduplicate the Int32/UInt32/Int64/UInt64 downcast chains Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings March 21, 2026 11:16

Copilot started reviewing on behalf of mo4islona March 21, 2026 11:17 View session

mo4islona force-pushed the feat/storage-benchmark-impl branch 2 times, most recently from 8fe9111 to 4647d69 Compare March 21, 2026 11:21

Copilot AI reviewed Mar 21, 2026

View reviewed changes

mo4islona force-pushed the feat/storage-benchmark-impl branch 25 times, most recently from 2f146cf to 8d1bd30 Compare March 21, 2026 21:48

mo4islona force-pushed the feat/storage-benchmark-impl branch 10 times, most recently from c88375b to 60c9cdb Compare March 24, 2026 09:52

mo4islona mentioned this pull request Mar 27, 2026

Fix correctness bugs: Int32 cast, silent type fallbacks, predicate panics #3

Merged

2 tasks

mo4islona force-pushed the feat/storage-benchmark-impl branch from 37a0e06 to b6c201b Compare March 27, 2026 21:01

mo4islona and others added 10 commits March 29, 2026 03:00

Fix negative Int32 address elements in extract_address

4af2210

Use u32::try_from instead of wrapping cast — negative i32 values are dropped instead of wrapping to huge u32 (e.g. -1 → 4294967295). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Update Storage Tiers benchmarks from clean hot_bench run

be6de2f

All numbers from single hot_bench run on OVH Xeon E-2136 after drop_caches. Consistent with previous sessions at all CPU levels. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

mo4islona force-pushed the feat/storage-benchmark-impl branch from 21755b0 to b700e5f Compare March 28, 2026 23:02

mo4islona force-pushed the feat/storage-benchmark-impl branch from b700e5f to be6de2f Compare July 9, 2026 14:47

mo4islona force-pushed the master branch from 4a40f7e to feb625b Compare July 9, 2026 14:47

mo4islona marked this pull request as draft July 9, 2026 15:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add hot storage layer and storage benchmarks#2

Add hot storage layer and storage benchmarks#2
mo4islona wants to merge 10 commits into
masterfrom
feat/storage-benchmark-impl

mo4islona commented Mar 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Copilot AI Mar 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

+        // After rebuilding the in-memory buffer, normalize the on-disk crash log
+        // so that subsequent appends start from a clean state and cannot
+        // truncate or overwrite stale `.ipc` files from previous runs.
+        if crash_log_dir.exists() {
+            std::fs::remove_dir_all(&crash_log_dir)
+                .with_context(|| format!("clearing crash log dir {}", crash_log_dir.display()))?;
+        }
+        std::fs::create_dir_all(&crash_log_dir)
+            .with_context(|| format!("recreating crash log dir {}", crash_log_dir.display()))?;

Uh oh!

Conversation

mo4islona commented Mar 21, 2026

Summary

Storage Tiers

Benchmark Results (CPU=30, vs Legacy+RocksDB)

New Files

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants