Repository navigation
Write cframes straight into the returned bytes (part 4/4 of #744) - #749
Open
FabianIsensee wants to merge 13 commits into
Open
FabianIsensee wants to merge 13 commits into
FabianIsensee wants to merge 13 commits into
Conversation
This was referenced Nov 3, 2026
FabianIsensee
force-pushed
the
split-cframe
branch
from
November 3, 2026 11:32
207cb0e to
ea2afe0
Compare
This was referenced Nov 3, 2026
NDArray.get_slice_numpy() decompressed with the GIL held, so threads reading separate arrays, or separate handles of one file, ran one at a time; on NFS this included the wait for the server. Release the GIL around b2nd_get_slice_cbuffer(). The GIL was also what kept two threads from using one super-chunk at the same time. Replace the per-NDArray read_lock with one reentrant lock per super-chunk, shared by the SChunk wrapper, the NDArray, its views and vlmeta, and hold it in every entry point into C. Keep the GIL when a Python postfilter, codec or filter could run during the read, and on wasm32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj7Wi7Sc8HtQpELyNXB
Hold the operands' handle locks while the miniexpr prefilter of a lazy expression reads their super-chunks from C. Count the reads that run without the GIL, so that exit and the registration of Python codecs and filters wait for them, and free handle locks that other threads held at a fork in the child. Lock the metalayer counters too. Skip the new tests on wasm32, shorten them, retry the GIL check on busy machines, and add tests for lazy expressions and fork. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj7Wi7Sc8HtQpELyNXB
The miniexpr prefilter read the chunks of every in-memory operand from sc->data, and the fast matmul prefilter did so for every operand. Frames have no chunk table there: in-memory arrays created with contiguous=True, arrays from ndarray_from_cframe() (copy=False, the default) and, for matmul, any on-disk array. Any lazy expression or matmul with such an operand dereferenced a NULL table and crashed the interpreter. Read frame-backed in-memory operands through blosc2_schunk_get_chunk(), as on-disk ones already are, and keep frame-backed operands off the fast matmul path: in its Python checks, and in _set_pref_matmul() itself, which now raises ValueError for them before changing any state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
The general indexing path (ndindex) raised, or returned a wrong shape, for keys that NumPy indexes fine: 554 of 18000 random keys in a comparison with NumPy. Fix the classes found: - slice bounds beyond the index range, and bool bounds or steps, next to ..., None or NumPy integers raised IndexError or TypeError; normalize them as NumPy reads them (clamped, bools as ints) before ndindex sees them; - 0-d integer arrays index as integers (AttributeError before); - scalar bools are 0-d boolean indices, as in NumPy: their dimension goes among adjacent advanced indices (integers included), else in front, and next to array indices they broadcast (True: no effect, False: an empty result or IndexError). The result shape comes from NumPy on a stride-0 stand-in for the array, so it costs no memory; - None next to (adjacent) array indices is supported; - 0-d arrays take NumPy scalar bools (AttributeError before) and raise IndexError for integer lists (ValueError before). Advanced indices separated by slices, ... or None still raise NotImplementedError (NumPy moves their dimensions to the front there). After this, none of the 18000 keys gives another result than NumPy; the remaining differences are that limitation and which error a key that is invalid for several reasons reports first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
- update_special() built its special chunk in a temporary SChunk with default cparams, so on an NDArray's super-chunk the chunk header had another block size than the array (e.g. 20000 instead of 800 bytes) and every later read of the array failed: b2nd reads blocks selectively and checks their number per chunk. The new SChunk._special_chunk() makes zero, NaN, uninitialized and repeated-value chunks with the super-chunk's own cparams. - update_data() and insert_data() with copy=False did not free their compressed chunk when the C call failed (nor on a compression error), so it leaked wherever c-blosc2 did not free it itself. They now free it whenever the call fails. This needs the next c-blosc2 release (or the companion c-blosc2 PR) for its caller-keeps-the-chunk-on-failure rule; with an older c-blosc2, a failed frame update can free the chunk twice. - decompress_chunk() without dst fetched the whole compressed chunk only to read its size from the header, then read it again to decompress it; it now gets the lazy chunk (header and block offsets) for the size. astype(float64) of a 300x512x512 float32 array on disk: 385 -> 366 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
LazyExpr._collect_flat_indices_from_bool_ndarray(), used for 1-D masks, took every chunk as chunk_len items, so the padding of the last chunk could yield positions past the end of the array: blosc2.full(1000, True, chunks=(300,)) gave 1200 positions. Clip each chunk to the items inside the array. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
C-Blosc2 uses BLOSC_NTHREADS, when set to a number it accepts, for every compression and decompression context, whatever nthreads the parameters ask for. blosc2.nthreads, and so the CParams/DParams defaults and numexpr's thread count, ignored it: with BLOSC_NTHREADS=4, Python reported 28 threads and gave numexpr 28 while C used 4. Start blosc2.nthreads from what C-Blosc2 gives a compression context (the new blosc2_ext.context_nthreads(), via blosc2_ctx_get_cparams()), and say in set_nthreads()'s docs that the variable wins at the C level. Importing blosc2 also overrode NUMEXPR_NUM_THREADS, which set_nthreads() recommends setting; it now leaves numexpr alone then, and otherwise gives it at most numexpr.MAX_THREADS (BLOSC_NTHREADS can go beyond that, which numexpr refuses with an error). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
Keys made of integers (Python or NumPy, not bools), unit-step slices, one Ellipsis and, for reads, None went through the general path: 8-30 us of Python per call (ndindex as soon as a NumPy integer, ... or None appears), more than decompressing a small selection. NDArray.get_basic_numpy() classifies such keys in Cython, resolves them against the shape and reads the result under one acquisition of the handle lock, through _get_slice_cbuffer(), the GIL-releasing read that get_slice_numpy() now shares with it. get_basic_start_stop() resolves the same keys (without None) for __setitem__. Any other key, or one that does not fit the shape, gives None and takes the unchanged general path, which raises the same exceptions as before. NumPy integers without __index__ (timedelta64) are left to it too, where they raise IndexError as in NumPy. SWMR: a read resolves the key against the cached shape and relies on the refresh inside the C read; if b2nd_array_t.last_tick shows that this refresh changed the shape (or the key did not fit, or the read failed), it refreshes, resolves and reads again, so a key is never resolved against a stale shape, at one file check per on-disk read instead of two. A write refreshes under the handle lock before resolving. Keys with None now return arrays that own their data (they were views of a temporary). Measured (Ryzen 9 7950X): a[3, 5, 7] of a 300x512x512 float32 array 34 -> 25 us in memory and 48 -> 35 us on disk, a[500] of a 1000x1000 array 21.5 -> 13.9 us, a[np.int64(7), np.int32(9)] 37.5 -> 7.3 us, iterating over 2000 rows 64 -> 44 ms; a[np.int64(i), j] = v 22 us less. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
Creating a small array cost ~30 us, mostly Python bookkeeping: - compute_chunks_blocks() and _check_ndarray_kwargs() converted CParams and Storage with dataclasses.asdict(), which deep-copies; a shallow dict (vars() for Storage, whose fields are plain values) is enough; - every SChunk wrapper, so every NDArray, built its cparams and dparams eagerly under the handle lock; they are now built on first use; - create_cparams_from_kwargs() looked up Filter.BYTEDELTA.value (a slow enum descriptor) twice per filter slot; now once per call; - is_inside_new_expr() and is_inside_ne_evaluate(), which arange, linspace, eye, reshape, fromiter and lazy functions such as clip run on every call, used inspect.stack(), which reads the source of every frame; walking the frames gives the same answer in under 1 us. __setitem__ also wrapped plain NumPy values in a SimpleProxy, whose constructor runs compute_chunks_blocks(); other array types keep the wrapping, NumPy arrays give the same outcome without it (checked for contiguous, strided, Fortran-order, 0-d, broadcast and converted values). Measured (Ryzen 9 7950X): zeros((100, 100)) 31 -> 20 us, eye() 311 -> 179 us, reshape() 437 -> 276 us, clip(a, 0, 1) 84 -> 12 us; a 3x3 write into a 100x100 array ~12 us less (asdict() ~5 us, SimpleProxy ~7 us). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
a[region] = scalar built np.full() of the whole region, and a[region] = ndarray decompressed all of the source first: an uncompressed copy of the region in memory (a fill of a large on-disk array could exceed RAM), then one set_slice. Scalars: every chunk the region fully covers gets one shared special (repeated-value) chunk from SChunk._special_chunk(), and the rest is written in boxes of bounded size (32 MB, or one chunk) from one buffer sized to the largest box. Edge chunks get a special chunk only for zero, as writing them leaves zeros in their padding (which a resize brings to light). Chunks go last to first: a frame compacts its payload behind an updated chunk, and that tail then holds the tiny special chunks already written. Lossy codecs and filters, and dictionaries, keep compressing the values. NDArray values: read box by box along the destination's chunk boundaries into reused buffers and written with one set_slice per box; when both arrays encode chunks the same way (dtype, chunks, blocks, lossless cparams) and the region starts at a chunk boundary, covered chunks without padding are copied compressed. A value that may share storage with the destination (same super-chunk or same file) is still read in full first. Both hold the handle locks (HandleLocks with the source) for the whole write, so that other threads see it as atomic as one set_slice. Regions smaller than a chunk, and arrays whose frame takes the cross-process lock (locking=True or BLOSC_LOCKING, destination or source), keep the single set_slice, which holds that lock from the shape refresh to the last chunk. A shared flat chunk index helper now also serves detect_aligned_chunks(), and __setitem__ marks indexes stale through one helper. Measured (300x512x512 float32, default chunks, Ryzen 9 7950X): a[:] = 0.0 41 -> 0.2 ms with no extra memory (was 303 MB); the first a[:] = 1 into an existing on-disk array 588 -> 29 ms; a[:] = b with the same encoding 142 -> 18 ms and 306 -> 28 MB peak. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
asarray() of an NDArray, and so astype(), sliced every chunk with __getitem__ and cast it into fresh arrays: chunk-sized allocations per chunk, which glibc trims and page-faults back in each time. When source and destination have the same shape, chunks and blocks (the default for astype), chunks without padding are now decompressed into one reused buffer, cast item by item into another and compressed from there; edge chunks keep the per-chunk path (their padding may hold anything, and casting it could warn). Chunks encoded the same way in both arrays are copied compressed (e.g. asarray(a, copy=True, urlpath=...)). Measured (300x512x512 float32, Ryzen 9 7950X, this change toggled): astype(np.uint8) 110 -> 56 ms, with nnU-Net-like chunks 286 -> 267 ms; astype(np.float64) unchanged (compression-bound). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
Co-Authored-By: Codex <noreply@openai.com>
NDArray.to_cframe() and SChunk.to_cframe() got the frame from b2nd_to_cframe() / blosc2_schunk_to_buffer() in a C buffer and copied it into a new bytes object. They now call blosc2_schunk_to_buffer_alloc() with a callback that creates the bytes object of the final size, and the frame is written into it directly: no second copy of the whole frame. SChunk.to_cframe() still releases the GIL only under set_releasegil(True); the callback takes it back to create the bytes. Needs blosc2_schunk_to_buffer_alloc() from the next c-blosc2 release (or the companion c-blosc2 PR). Measured with it (300x512x512 float32 in memory, default chunks): to_cframe() 54 -> 15 ms and peak memory 456 -> 230 MB. Adds round-trip tests of to_cframe() / from_cframe() for every storage kind and for special, updated and resized chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
FabianIsensee
force-pushed
the
split-cframe
branch
from
November 4, 2026 13:30
a1ba70e to
eb21db1
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Francesc,
this is part 4 of 4 of the split of #744:
to_cframe()writes the frame straight into the returnedbytes.Dependencies:
blosc2_schunk_to_buffer_alloc(). It is in c-blosc2 3.5.0, which main now bundles, so this branch no longer needs a TEMP commit.blosc2_ext.pyx. Until those are merged, this PR also shows their commits. The new ones:eb21db1Write cframes straight into the returned bytesWhat changed
NDArray.to_cframe()andSChunk.to_cframe()got the frame in a C buffer and copied it into a newbytesobject. They now callblosc2_schunk_to_buffer_alloc()with a callback that creates thebytesobject of the final size, and the frame is written into it directly. Together with the one-pass frame fill of c-blosc2 part 4,to_cframe()of a 300x512x512 float32 array in memory went from 97 to 15 ms (54 -> 15 ms on top of the one-pass fill alone), and its peak memory from 456 to 230 MB.Tests
pytest tests: 10477 passed, 283 skipped, 2 failed (the same 2b2viewhttp failures as on main). New: round trips ofto_cframe()/from_cframe()for every storage kind, and for special, updated and resized chunks.AI disclosure: I used Claude Code (Anthropic) for the investigation, the code and the tests of #744, which I reviewed and tested. Claude Code also did this split and opened this PR on my behalf, as agreed in #744 (comment). The description is assembled from the one of #744.
Best,
Fabian
🤖 Generated with Claude Code