Repository navigation
Faster slice reads/writes, serialization and NFS opens; fixes for memory-safety bugs - #833
Closed
FabianIsensee wants to merge 14 commits into
Closed
FabianIsensee wants to merge 14 commits into
FabianIsensee wants to merge 14 commits into
Conversation
A frame without chunks has no offsets chunk: off_chunk stays NULL and off_cbytes 0. frame_from_schunk() still passed it to memcpy() (in-memory frames) or to the I/O write (on-disk ones, where blosc2_stdio_write() rejects the NULL pointer and traces an error). memcpy() from NULL is undefined even for zero bytes, and UndefinedBehaviorSanitizer (nonnull-attribute) reports it in every test that creates an empty super-chunk: with -fno-sanitize-recover, 68 of the 2083 tests and fuzz targets fail. Skip the copy when there is nothing to copy; with that, the whole suite passes under ASan, UBSan and LSan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
For typesize 1 a byte shuffle is the identity, but the SIMD dispatchers have no 1-byte case and fall back to the generic routine, which copies one byte at a time. With SHUFFLE in the pipeline (python-blosc2's default for every dtype) uint8, int8 and bool data paid that loop on every block, compressing and decompressing. The filter pipeline now treats a SHUFFLE of 1-byte elements (the typesize, or the filter meta when set) as a no-op like NOFILTER, so with SHUFFLE as the only filter the codec reads and writes the caller's buffers directly; blosc2_shuffle()/blosc2_unshuffle() copy the block. Chunk headers still record SHUFFLE, so the format is unchanged, and the chunks are byte-identical (checked with 1 thread over uint8/int8/bool/S1 and other typesizes, DELTA+SHUFFLE, SHUFFLE+BITSHUFFLE and three codecs). As both pipeline loops now skip the no-op filters up front, the NOFILTER and TRUNC_PREC guards further down could not fail any more and are removed; the bytes a SHUFFLE groups are computed in one helper. Measured on a Ryzen 9 7950X with 300x512x512 uint8/bool arrays, 1 thread: full decompression 2.5-3.2x (ZSTD 5) and 3.5-4.2x (LZ4HC 5) faster, box reads from a mapped file 2.8-6.8x, compression 1.01-1.13x; with all threads, decompression and box reads 1.08-1.49x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
Every append, insert and update of a chunk in a frame checked that regular and VL-block chunks are not mixed by reading a reference chunk (chunk 0, or 1 when updating chunk 0) in full, just for its flags2 byte. Updates and deletes read the replaced chunk in full for its nbytes and cbytes (updates twice: in schunk_update_chunk_unlocked() and again in frame_update_chunk()), and appends read the last chunk as a lazy chunk for its nbytes. With the pread backend each of these is a malloc plus a pread of the whole compressed chunk (or of its header and block starts), e.g. 3 x 5 MB for every chunk that nnU-Net's preprocessing writes. frame_get_chunk_header() copies just the 32-byte chunk header into the caller's buffer, synthesizing it for chunks coded in their offset as frame_get_lazychunk() does. The schunk layer uses it for flags2, nbytes and the sizes of a replaced chunk, and passes the old cbytes on to frame_update_chunk(). Plain super-chunks still look at the chunk in place. A delete of a chunk with a malformed header no longer leaks the chunk it read. Measured on a cluster node (Xeon Gold 6330), 10x10x10 setitem into a (50, 512, 512) float32 array on disk: 16.1 -> 11.3 ms on local XFS and 19.6 -> 12.8 ms on NFS; overwriting a whole 315 MB array on disk takes 31% (XFS) and 38% (NFS) less time, 20% less on a workstation. New test_frame_chunk_header compares the header with that of the lazy chunk for in-memory, file and directory frames, chunks coded in their offset and a short last chunk; test_no_mix_regular_and_vlblocks now also covers inserts and updates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
On NFS every open and close of a file is a synchronous round trip to the server (OPEN and CLOSE RPCs, and a close after writes flushes them), and frames opened and created their files more often than needed: - frame_from_file_offset() opened the file to read the header and the trailer and closed it again; the first read through the frame, still inside blosc2_schunk_open(), then opened it once more for the cached reader. That first handle now becomes the cached reader. - frame_from_schunk() created the file "wb", wrote it and closed it, and the trailer update right after reopened it "rb+" for the cached writer. On the filesystem backend the file is now created "wb+" and that handle becomes the cached writer. Since every update reads the frame header first, a read-only stdio handle on a dup() of its descriptor becomes the cached reader as well: it shares the open file description, so it costs nothing on the server. A handle is adopted only where one would be cached anyway (filesystem backend, contiguous frame, POSIX, no locking outside a lock, handle budget not exhausted); its identity is recorded as for any cached handle, so a file replaced underneath is still noticed. The rules for caching a handle are shared by the acquire and adopt paths through reader_cacheable(), writer_cacheable(), cache_slot_claim() and cache_install(). The error paths of frame_from_file_offset() close the handle in one place. Measured through python-blosc2 on a cluster node, NFSv4.1, pread backend (medians; the server latency varied 2x): opening and closing an array takes 1 instead of 2 OPEN and CLOSE RPCs (1.21 -> 1.04 ms), creating one with blosc2.zeros(..., urlpath=...) 4 instead of 9 RPCs (REMOVE, OPEN, WRITE, CLOSE; 7.7 -> 5.7 ms for 300x512x512 float32, 9.8 -> 6.7 ms for 100x100). Unchanged on local XFS. New tests in test_frame_writer_handle: an opened frame holds exactly one descriptor before and after its first read, still notices a replaced file, and a created frame holds at most two and none after the free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
blosc2_schunk_copy() appended the chunks one by one. For a contiguous frame destination every append decompresses and recompresses all the offsets, rewrites the header and the trailer and grows the frame, after the caller has made a private copy of the chunk. This is the path behind blosc2_schunk_to_buffer() for anything but an in-memory frame, blosc2_schunk_to_file() and b2nd_copy() to a file. frame_fill_from_schunk() now writes every chunk once, straight from the source, into a frame sized up front (or sequentially to the file through the cached writer), and the offsets, header and trailer once at the end. An on-disk source is read under its shared lock, with its frame header read once and 32 bytes of chunk header per chunk, each chunk going straight to its place. The checks and counters of the append path (VL-block mixing, chunksize detection, no two consecutive short chunks, special chunks coded in offsets) are kept in a tally that shares its rules with schunk_append_chunk_unlocked() and frame_append_chunk() through small helpers (chunksize_after_adding(), chunk_special_in_offset(), frame_takes_chunk(), compress_offsets()), so the frame is byte-identical to the one built by appending. get_chunk_flags2() moves to blosc-private.h for that. blosc2_schunk_to_buffer() hands out a frame that the library allocated, so a caller that needs it in memory of its own (python-blosc2 returns a bytes object) has to copy it once more. The new blosc2_schunk_to_buffer_alloc() asks the caller for a buffer of the exact frame length through a callback and writes the frame there. Unless the super-chunk already is an in-memory frame (copied as it is), a pass over the chunk headers gives the offsets and the length and a second pass copies the chunks, without building the frame first. The bytes are the same as from blosc2_schunk_to_buffer(); a source that changes between the passes makes the call fail. blosc2_schunk_copy() is split into a skeleton part and the vlmetalayer copy, so that both paths build the same header and trailer. Measured on a Ryzen 9 7950X (28 threads) with a 300x512x512 float32 array (241 MB compressed), through python-blosc2 calling the new function: to_cframe() of the default in-memory array 103 -> 15 ms (54 ms with the one-pass fill alone) and half the peak memory (+467 -> +230 MB), to_cframe() of an on-disk array 103 -> 18 ms, save() and copy(urlpath=...) 57 -> 45 ms, which is the raw write speed of the disk (on a cluster node with local XFS 118 -> 67 ms). New test_schunk_to_buffer_alloc checks to_buffer, to_buffer_alloc and to_file against each other for in-memory, file and directory sources with special chunks, a short last chunk, metalayers and vlmetalayers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
get_set_slice() (de)compressed straight from/into the caller's buffer only when the slice was exactly one chunk. Every other slice went through a chunk-sized scratch: decompress there, then copy block by block into the buffer on the calling thread (and the reverse for writes, after zeroing the scratch or decompressing the old chunk into it). On top of that, every read first zeroed the whole destination. - Direct path: any chunk that lies entirely inside the slice and maps to one contiguous run of the buffer in its decompressed layout (blocks in C order, chunk spanning the trailing buffer dims) is decompressed into, or compressed from, the buffer with the context's threads. This covers a[:], np.asarray(a), a[:] = x and row ranges of arrays chunked along the first axis, and subsumes the single-chunk fast path. Reads zero only the buffer bytes outside the slice. - Scatter path for reads: for chunks that are partly read or not contiguous in the destination (32^3 blocks of a 128^3 patch, unaligned boxes, edge chunks), a b2nd-internal postfilter on the super-chunk's decompression context copies each block from the worker thread that decoded it (still in cache) straight to its place in the buffer. - Gather path for writes: a b2nd-internal prefilter on the compression context assembles each block from the caller's buffer (and, for partly overwritten chunks, the old contents) in the compression thread. Blocks that the slice overwrites entirely are masked out of the decompression of the old chunk; a partly overwritten chunk takes this path only when the slice covers at least half of it. - Handles with a postfilter or prefilter of their own (python-blosc2's lazy UDFs and expressions) and DELTA keep the scratch path, as do chunks that turn out to use DELTA or an instrumented codec. The compact path for reads of few blocks is unchanged. Uninitialized and missing chunks read as zeros on the new paths. - With transparent huge pages, threads writing neighbouring blocks of a fresh buffer fault on the same huge page at once, and each clears a whole huge page before all but one throw theirs away (a 300 MB read with 28 threads spent several times its wall time in the kernel). When a read fills a fresh buffer of 4 MB or more that gets huge pages (probed with mincore() and one MADV_POPULATE_WRITE) and the context has several threads, each chunk's region is prefaulted from the calling thread right before it is decompressed. Linux only, best effort. The masked decompressions of slice reads go through one helper, schunk_decompress_masked(), which also clears the block mask when the read fails before decompressing (the mask stayed on the context, and the next decompression with it skipped blocks). filters_contain() replaces the loops that looked for DELTA in a pipeline. Measured on a Ryzen 9 7950X (28 threads) through python-blosc2 with a 300x512x512 float32 array (315 MB): a[:] in memory 40.8 -> 24.3 ms, from a file 59.1 -> 39.7 ms (mapped 41.4 -> 25.8 ms), unaligned 200x400x400 boxes 1.35-1.66x, random 128^3 patches of nnU-Net-like chunks 1.25-1.88x, a[:] = x in memory 93.9 -> 75.4 ms and on disk 148.9 -> 114.7 ms; with 1 thread, a[:] 1.27x. New test_b2nd_slice_paths runs slice reads and writes through each path (direct, scatter, gather, scratch) with padded edge chunks and blocks, buffers larger than the slice, a user postfilter, special chunks, 1 and 4 threads and all storage kinds; test_failed_read_mask checks that a read that fails on a truncated file leaves no block mask behind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
Slice reads and writes on frames allocated their chunk-sized buffers per call or per chunk and freed them again: b2nd's chunk and block scratch, the compression destination of every written chunk and, for frames read through the pread backend, the compressed chunk. glibc returns memory freed at the top of the heap to the kernel, so in a long-running process (Python) these buffers were page-faulted in again on every call. Add blosc2_tmp_malloc()/blosc2_tmp_free() (internal): a per-thread cache (POSIX thread key, freed at thread exit and by blosc2_destroy() for the calling thread) of up to four buffers between 64 KB and 32 MB, at most 64 MB per thread, each handed out only to requests of at least half its size. The buffers stay plain malloc memory; Windows keeps malloc/free. b2nd slice reads and writes on frames, frame_decompress_chunk() and the frame fill take their temporaries from it. Plain in-memory super-chunks keep plain malloc: they keep each chunk written to them, and with their scratch cached glibc's dynamic mmap threshold stayed low, so every new chunk became a fresh mmap (an in-memory a[3, 5, 7] = 1.0 took 1.96-2.17 instead of 1.70 ms). A frame stores its own copy of every chunk written to it, so b2nd lends it the cached compression buffer through the internal schunk_update_chunk_lent(), which writes it out without the private copy that copy=true makes (the library cannot tell where a caller's chunk points, e.g. into another handle's mapping of the same file, so copy=true keeps copying). For that, frame_append/insert/update_chunk() no longer free the chunk on success; the schunk layer frees what is left to it in one place, frame_write_done(): a private copy always, a chunk given with copy=false once it is written. Ownership at the API is unchanged. Measured on a Ryzen 9 7950X through python-blosc2: writing [1:95, 1:511, 1:511] of an on-disk array with nnU-Net-like chunks in 16 calls 70.8 -> 38.4 ms (18.5k -> 4 page faults per call; now as fast as one call), reading a 128^3 patch from nnU-Net data and segmentation files reopened per sample (mmap, 1 thread) 3.48 -> 2.38 ms, a 10x10x10 setitem on disk 5.63 -> 4.22 ms. New test_tmp_cache checks reuse, the limits, the release at thread exit, and that slice reads and writes on in-memory and on-disk frames leave their chunk-sized buffers in the cache. test_frame_self_chunk writes chunks that point into the frame's own storage (in memory, mapped) or into another handle's mapping of the file back to the frame, which must keep working with copy=true. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
An uninitialized chunk decodes to nothing, so its destination keeps what it had. Slice reads zeroed that only where the destination is the slice buffer (whole chunks) or where the scatter postfilter writes it; the compact path (one block at a time) and the chunk-sized scratch path (DELTA, postfilters, instrumented codecs) copied the stale scratch into the slice, and a partial write into such a chunk stored the stale items around the slice. With the per-thread temporary cache that scratch holds items of the last array sliced on the thread, not just recycled heap. The scratch is now zeroed whenever a decompression into it decoded nothing, unless a postfilter computes the items. New slice_uninit test in test_b2nd_slice_paths: each operation runs on an initialized array first, so the scratch is dirty; reads of an uninitialized array must return zeros, and partial writes must leave zeros around the slice. Without this change it fails on the compact and scratch reads and on both kinds of partial writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
blosc2_schunk_get_slice_buffer() read every chunk that a slice covers only in part with getitem, which decodes the needed blocks one after the other in the calling thread. For large chunks this made an unaligned slice far slower than an aligned one: half of an 8 MB chunk (16 of 32 blocks, LZ4) took twice as long as decompressing the whole chunk with the threads. When the part spans at least 4 blocks and 1 MB and the decompression context has more than one thread, mask out the other blocks and decompress the chunk with the context threads. A postfilter copies each decoded block's share straight to the caller's buffer from the thread that decoded it, so the chunk-sized destination that the decompressor asks for is never written. Chunks that getitem reads for free (special values, memcpyed), VL-block chunks, DELTA (decoded against the first destination block), instrumented codecs and contexts with their own postfilter keep getitem. This replaces the masked read that was disabled with `if (false)`. Measured through python-blosc2 SChunk[a:b] on a Ryzen 9 7950X, LZ4 clevel 5, 8 MB chunks of 256 KB blocks, 28 threads: half a chunk 0.53 -> 0.16 ms, one chunk's worth across two chunks 1.08 -> 0.33 ms, 8 blocks 0.27 -> 0.11 ms, in memory and on disk alike. Reads of 2-3 blocks stay on getitem, where the threaded path was 1.5x slower. test_get_slice_buffer gets slices of 1 MB chunks that take the threaded path, also across a zero chunk, with DELTA and a shorter last chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
With locking on, b2nd_get_slice_cbuffer() refreshed the shape (taking and releasing the shared frame lock) and then read the chunks of that shape, each access taking the lock again on its own. A writer could shrink the array in between, although it held the exclusive lock across its whole resize and fill, and the read failed with "nchunk exceeds the number of chunks" (python-blosc2 readers saw this on 7-27% of reads under a resize-heavy locked writer). blosc2_schunk_get_slice_buffer() likewise took the lock once per chunk to fetch the (lazy) chunk, and decompressed it, reading its blocks from the file, after letting go, so a slice could mix chunks of several states of the frame. Both now hold the shared lock from the refresh to the last chunk read; the chunk accesses nest on it. b2nd_set_slice_cbuffer(), which took the exclusive lock only after refreshing, now refreshes under it; both b2nd calls go through one wrapper, locked_get_set_slice(). Unlocked handles neither lock nor (for SChunk slices) refresh, and keep SWMR's documented transient errors. Writers on other handles now wait for a whole multi-chunk read. New tests: in test_b2nd_get_slice_lock a forked writer resizes up and down under one exclusive bracket per resize and fill while the parent reads the whole array, and in test_get_slice_buffer_lock a forked writer rewrites all chunks with a new version per bracket while the parent reads a slice across all of them; every read must succeed and see one state. Without this change the SChunk test fails on its first reads, the b2nd one (a race) in most runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
With use_dict on, data that did not compress fell back to being stored
as is (memcpyed), but the chunk kept the BLOSC2_USEDICT flag that the
header got for the pass with a dictionary, although the copy took that
dictionary's place. Reading it then took the first data bytes for the
dictionary size: decompression and getitem failed with
BLOSC2_ERROR_CODEC_DICT or BLOSC2_ERROR_READ_BUFFER, so such chunks
could not be read back (LZ4, LZ4HC and ZSTD, any typesize and thread
count; in python-blosc2 e.g. blosc2.asarray(random_uint8,
cparams={"use_dict": True})[:]).
The writer now clears the flag when the pass with a dictionary ends in a
copy, and the reader ignores the flag on memcpyed chunks, which never
hold a dictionary, so chunks written by earlier versions are readable
too. Chunks that do compress with a dictionary are unchanged byte for
byte.
New test_dict_memcpyed compresses incompressible data with a dictionary
for each of these codecs (typesizes 1 and 4, 1 and 4 threads) and reads
it back, also from a chunk that still carries the flag (as older
versions wrote it); it fails without this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
frame_insert_chunk(), frame_update_chunk(), frame_delete_chunk(), frame_reorder_offsets() and frame_from_schunk() each had their own copy of the offsets compression that append and the frame fills do through compress_offsets(). The copies of insert, update, delete and reorder did not pin the context typesize to that of the offsets, so with the BLOSC_TYPESIZE environment variable set they compressed the offsets with the user's typesize (still readable) and gave another frame than appending the same chunks. frame_from_schunk() used the default cparams (its only caller, blosc2_schunk_new(), hands it a super-chunk without chunks, which has no offsets chunk, so no frame changes there). All of them now go through compress_offsets(), which pins typesize 8; without the variable the frames are the same as before. The helper takes the size of the destination, since that can change the result, and delete keeps compressing one offset less into the room of all of them. The allocations of the offsets chunk are now checked everywhere, and a reorder that cannot create the context no longer leaks the offsets. New test_offsets_typesize sets BLOSC_TYPESIZE and checks that inserting the last chunk, or updating a chunk in place, gives the frame that appending the same chunks gives; it fails without this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
Error paths that main does not cover yet (most found with ASan and LSan): - The library's own callers of chunk writes with copy=false now follow the ownership rule (a failed call leaves the chunk to the caller): blosc2_schunk_set_slice_buffer() leaked its chunk on a failed compression or update, and its scratch on every error; b2nd_full() and b2nd_concatenate() leaked their chunk when an update failed, and growing a b2nd array leaked the chunk of a failed insert. The docs of blosc2_schunk_update_chunk() now state the rule, as those of append and insert do. - blosc2_schunk_new() returned NULL without freeing the super-chunk (and frame) it had started, e.g. when the destination file or directory existed, and frame_from_schunk() leaked its header, offsets and offsets chunk on its error paths. array_new() (b2nd_empty(), b2nd_zeros(), ...) and b2nd_copy() leaked the half-built array and left it in *array when creating or copying the super-chunk failed, b2nd_copy() also when copying a vlmetalayer failed. array_new() now fails through one label that frees the array with b2nd_free(), and b2nd_copy() copies the vlmetalayers with the schunk helper, which takes the cparams for that. - blosc2_schunk_copy() stored a pointer to its local cparams in the caller's storage when that had none, which dangled after the return (a second copy with the same storage read a dead stack frame); it now works on a copy of the storage. A copy with other cparams appended a full chunksize for a shorter last chunk, padding it with the tail of the previous chunk. - frame_delete_chunk() leaked the offsets or the offsets chunk on its later error paths, and a failed realloc of an in-memory frame lost the frame there and in frame_update_chunk(); frame_append_chunk() and frame_insert_chunk() did not check the offsets allocation. These frame writers and frame_from_schunk() now fail through one cleanup label each. - A failed orthogonal selection leaked its selection arrays, block mask and chunk buffer, and a read that failed before decompressing left its block mask on the context, so the next decompression skipped blocks. blosc2_set_maskout() left a freed mask on the context when its allocation failed. New tests: test_chunk_ownership (failed updates and slice writes into a frame whose file was removed, failed blosc2_schunk_new() calls; leaks and double frees under ASan/LSan), test_copy_errors (copies and b2nd creations failing at each stage, for every kind of destination; its storage and b2nd checks fail in any build without this change) and the orthogonal-selection case of test_failed_read_mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
Describe the performance changes (1-byte shuffle, b2nd and SChunk slices, frame copies and blosc2_schunk_to_buffer_alloc(), header-only reads, per-thread temporaries, NFS opens) with their measured effect, the behaviour changes (locked slice reads, offsets typesize, uninitialized chunks read as zeros, the update ownership docs) and the fixes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
This was referenced Oct 30, 2026
Closed
Contributor
Author
|
Closing in favour of the split that Francesc proposed in Blosc/python-blosc2#744 (comment):
All six together give exactly the tree of this PR. The python-blosc2 side is split the same way, in Blosc/python-blosc2#746 to Blosc/python-blosc2#749. (The split and this comment were made with Claude Code on my behalf.) |
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Francesc,
after #821 we looked at the rest of the read/write paths (b2nd slices, frame copies, opening files, SChunk slices) with nnU-Net-style and general workloads, locally and on NFS. This PR collects what came out of it: performance improvements with byte-identical results, plus a set of memory-safety fixes we found on the way (mostly with ASan/UBSan/LSan, which now pass the whole test suite). It is big; every commit is self-contained and has a descriptive message, so it can be reviewed (or split) commit by commit.
Performance
Measured on a Ryzen 9 7950X (28 threads, local ext4) and a 2x Xeon 6330 node (16 threads, local XFS and NFSv4.1), interleaved against main, medians (python-blosc2 numbers, since that is how we drive it; the companion python PR contributes the Python-side part):
a[:]of a 315 MB float32 array: 41 -> 23 ms (node 143 -> 80), unaligned boxes 1.3-1.7x, full writes 1.3-1.65x.blosc2_schunk_copy()filled a new frame by appending chunk by chunk (two copies per chunk, offsets recompressed every time); it now fills it in one pass. Newblosc2_schunk_to_buffer_alloc()writes the frame straight into caller memory.to_cframe()97 -> 15 ms with half the peak memory;save/copyto a file now run at write bandwidth.Bug fixes (each with a test that fails without it)
Not included: the fixes you already made today for the same areas (LZ4 dictionary memmove, failed frame append/insert leaks and state,
blosc2_schunk_copyerror-path leaks, b2nd slice/meta/open leaks); this branch is merged with them and uses your versions. On top of those:use_dict(LZ4/LZ4HC/ZSTD) and data that does not compress, the chunk is stored as-is but keepsBLOSC2_USEDICT, so the reader takes the first data bytes as a dictionary size and fails withBLOSC2_ERROR_CODEC_DICT(e.g.blosc2.asarray(random_uint8, cparams={"use_dict": True})[:]). The writer now clears the flag in that case and the reader ignores it for memcpyed chunks, so files already written this way are readable again.blosc2_schunk_copy()left a pointer to a local struct in the caller's storage; it padded a shorter last chunk when recompressingblosc2_schunk_new()/frame_from_schunk(),blosc2_schunk_set_slice_buffer(),b2nd_full(),b2nd_concatenate(),b2nd_resize()(failed insert),b2nd_copy();b2nd_empty()& co. handed back a freed array when creating onto an existing fileframe_delete_chunk(): the offsets chunk leaked on the later error paths and a failed realloc lost the in-memory frameblosc2_set_maskout()left a dangling mask on malloc failure; a failed orthogonal selection leaked its selection arraysmemcpyof the offsets of a frame without chunks (UBSan)BLOSC_TYPESIZEenv value when that was set (append pinned typesize 8); now all pin 8blosc2_schunk_update_chunk()withcopy=falsenow states the same contract as append/insert (caller keeps the chunk on failure), which main already implementsBehaviour changes to be aware of
copy=falseleaves the chunk with the caller; every library caller of these functions (b2nd writes, copy, set_slice_buffer) now frees its chunk exactly once on failure. python-blosc2's matching change ships in the companion PR.BLOSC_TYPESIZEis set).blosc2_destroy(); no cache on Windows.blosc2_schunk_to_buffer_alloc().Not in this PR / questions for you
a[:] = yon an existing 315 MB file moves 4-6 GB (NFS 4.3-7.3 s vs 0.37 s for a new file). We have a prototype that defers compaction to the end of eachb2nd_set_slice_cbuffercall (6-11x); we would like to discuss the design (holes during a call, SWMR readers, a begin/end-updates API) in a separate PR.nthreads=1every context keeps its own serial scratch (~512 KB for 128 KB blocks) for its lifetime, so many open single-threaded handles get expensive (~1.3 MB per nnU-Net case); one per OS thread would do. Separate PR as well.update_buffercounterpart ofblosc2_schunk_append_buffer()).locking=Truea shrinking writer can still make its reads fail; holding one shared lock across refresh, slice computation and read would need a public shared variant ofblosc2_schunk_lock()(only the exclusive one is exported).zstd_wrap_compresstests the already-mapped level and never fires (clevel 8 maps to ZSTD 15). Changing it would slow every clevel-8 user, so we left it.Tests
every commit builds and passes the full ctest (final: 2088/2088) in Release and under ASan+UBSan+LSan incl. the fuzz targets (
-fno-sanitize=alignment,shift-base, final 2095/2095); file-based C tests with the working directory on NFS; python-blosc2's suite with the companion branch; randomized differential tests against main (slices, special chunks, storage kinds, cframes: identical data; with nthreads=1 identical chunk bytes).AI disclosure: I used Claude Code (Anthropic) for the investigation, the code, the tests and a first draft of this description. I reviewed and tested everything.
Best,
Fabian