Skip to content

Write scalars and NDArray values chunk by chunk; cast whole chunks in asarray/astype (part 3/4 of #744) - #748

Open
FabianIsensee wants to merge 15 commits into
Blosc:mainfrom
FabianIsensee:split-writes
Open

Write scalars and NDArray values chunk by chunk; cast whole chunks in asarray/astype (part 3/4 of #744)#748
FabianIsensee wants to merge 15 commits into
Blosc:mainfrom
FabianIsensee:split-writes

Conversation

@FabianIsensee

@FabianIsensee FabianIsensee commented Nov 3, 2026

Copy link
Copy Markdown

Hi Francesc,

this is part 3 of 4 of the split of #744: chunk-wise writes in __setitem__, and chunk-wise casts in asarray/astype.

Depends on #747 (part 2, stacked on part 1, #746, and on #743). The scalar writes use part 1's SChunk._special_chunk(), and both writes and casts change the __setitem__ code that part 2 changes. Until those are merged, this PR also shows their commits. The new ones:

  • 88d3ffc Write scalars and NDArray values chunk by chunk in NDArray.setitem
  • 35078a6 Cast whole chunks with reused buffers in asarray/astype
  • their release notes

The last commit (TEMP) builds against Blosc/c-blosc2#835, like parts 1 and 2.

What changed (300x512x512 float32, default chunks, Ryzen 9 7950X; numbers from the commit messages)

  • __setitem__ with a scalar fills fully covered chunks with special chunks instead of building np.full(region). a[:] = 0.0 went from 41 ms (and a 303 MB uncompressed copy of the region in RAM) to 0.2 ms, and the first a[:] = 1 into an existing on-disk array from 588 to 29 ms.
  • With an NDArray value it goes chunk by chunk, and copies compressed chunks when both arrays encode them the same way. a[:] = b with the same encoding went from 142 ms and a 306 MB peak to 18 ms and 28 MB.
  • Arrays opened with locking=True keep the single atomic set_slice path.
  • asarray/astype of NDArrays cast whole chunks in reused buffers. astype(np.uint8) went from 110 to 56 ms.

Memory bugs fixed

  • a[region] = scalar built an uncompressed np.full of the whole region, so a fill of a large on-disk array could exceed RAM.
  • a[region] = other_ndarray decompressed the whole source first.

Both now work chunk by chunk.

Found, not fixed

  • Compressing in memory with many threads right after reads can be slower than on main (float32, 28 threads: 1.4-2.4x; equal in a fresh process). update_data & co. allocate a fresh chunk-sized output per chunk, and main's read path happened to warm glibc's heap for that size, which the faster read path no longer does. Planned follow-up: reuse the output buffer, on the python side or with an update_buffer call in c-blosc2.

Tests

pytest tests: 10388 passed, 280 skipped, 2 failed (the same 2 b2view http failures as on main). New: test_setitem_chunkwise (scalar fills incl. special and edge chunks, lossy/dict cparams, NDArray values on disk and in memory, compressed copies, shared storage, buffer sizes, astype), and cases in test_locking for handle locks, threads, other processes and resized arrays.

AI disclosure: I used Claude Code (Anthropic) for the investigation, the code and the tests of #744, which I reviewed and tested. Claude Code also did this split and opened this PR on my behalf, as agreed in #744 (comment). The description is assembled from the one of #744.

Best,
Fabian

🤖 Generated with Claude Code

FabianIsensee and others added 12 commits November 3, 2026 10:07
@FabianIsensee @claude
Release the GIL in NDArray slice reads
77d409c 
NDArray.get_slice_numpy() decompressed with the GIL held, so threads
reading separate arrays, or separate handles of one file, ran one at a
time; on NFS this included the wait for the server.  Release the GIL
around b2nd_get_slice_cbuffer().

The GIL was also what kept two threads from using one super-chunk at
the same time.  Replace the per-NDArray read_lock with one reentrant
lock per super-chunk, shared by the SChunk wrapper, the NDArray, its
views and vlmeta, and hold it in every entry point into C.  Keep the
GIL when a Python postfilter, codec or filter could run during the
read, and on wasm32.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj7Wi7Sc8HtQpELyNXB
@FabianIsensee @claude
Make GIL-free slice reads safe around fork, exit and lazy expressions
58328e7 
Hold the operands' handle locks while the miniexpr prefilter of a lazy
expression reads their super-chunks from C.  Count the reads that run
without the GIL, so that exit and the registration of Python codecs and
filters wait for them, and free handle locks that other threads held at
a fork in the child.  Lock the metalayer counters too.

Skip the new tests on wasm32, shorten them, retry the GIL check on busy
machines, and add tests for lazy expressions and fork.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj7Wi7Sc8HtQpELyNXB
@FabianIsensee @claude
Fix crashes with frame-backed operands in miniexpr and matmul
7019168 
The miniexpr prefilter read the chunks of every in-memory operand from
sc->data, and the fast matmul prefilter did so for every operand.  Frames
have no chunk table there: in-memory arrays created with contiguous=True,
arrays from ndarray_from_cframe() (copy=False, the default) and, for
matmul, any on-disk array.  Any lazy expression or matmul with such an
operand dereferenced a NULL table and crashed the interpreter.

Read frame-backed in-memory operands through blosc2_schunk_get_chunk(),
as on-disk ones already are, and keep frame-backed operands off the fast
matmul path: in its Python checks, and in _set_pref_matmul() itself,
which now raises ValueError for them before changing any state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
@FabianIsensee @claude
Make the general NDArray.__getitem__ path accept the keys NumPy accepts
f4c5f9c 
The general indexing path (ndindex) raised, or returned a wrong shape,
for keys that NumPy indexes fine: 554 of 18000 random keys in a
comparison with NumPy.  Fix the classes found:

- slice bounds beyond the index range, and bool bounds or steps, next to
  ..., None or NumPy integers raised IndexError or TypeError; normalize
  them as NumPy reads them (clamped, bools as ints) before ndindex sees
  them;
- 0-d integer arrays index as integers (AttributeError before);
- scalar bools are 0-d boolean indices, as in NumPy: their dimension
  goes among adjacent advanced indices (integers included), else in
  front, and next to array indices they broadcast (True: no effect,
  False: an empty result or IndexError).  The result shape comes from
  NumPy on a stride-0 stand-in for the array, so it costs no memory;
- None next to (adjacent) array indices is supported;
- 0-d arrays take NumPy scalar bools (AttributeError before) and raise
  IndexError for integer lists (ValueError before).

Advanced indices separated by slices, ... or None still raise
NotImplementedError (NumPy moves their dimensions to the front there).
After this, none of the 18000 keys gives another result than NumPy; the
remaining differences are that limitation and which error a key that is
invalid for several reasons reports first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
@FabianIsensee @claude
Fix SChunk.update_special(), failed data writes and decompress_chunk()
2ef6a36 
- update_special() built its special chunk in a temporary SChunk with
  default cparams, so on an NDArray's super-chunk the chunk header had
  another block size than the array (e.g. 20000 instead of 800 bytes)
  and every later read of the array failed: b2nd reads blocks
  selectively and checks their number per chunk.  The new
  SChunk._special_chunk() makes zero, NaN, uninitialized and
  repeated-value chunks with the super-chunk's own cparams.
- update_data() and insert_data() with copy=False did not free their
  compressed chunk when the C call failed (nor on a compression error),
  so it leaked wherever c-blosc2 did not free it itself.  They now free
  it whenever the call fails.  This needs the next c-blosc2 release (or
  the companion c-blosc2 PR) for its caller-keeps-the-chunk-on-failure
  rule; with an older c-blosc2, a failed frame update can free the chunk
  twice.
- decompress_chunk() without dst fetched the whole compressed chunk only
  to read its size from the header, then read it again to decompress
  it; it now gets the lazy chunk (header and block offsets) for the
  size.  astype(float64) of a 300x512x512 float32 array on disk:
  385 -> 366 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
@FabianIsensee @claude
Clip the last chunk when collecting the True positions of a bool mask
bc48440 
LazyExpr._collect_flat_indices_from_bool_ndarray(), used for 1-D
masks, took every chunk as chunk_len items, so the padding of the last
chunk could yield positions past the end of the array:
blosc2.full(1000, True, chunks=(300,)) gave 1200 positions.  Clip each
chunk to the items inside the array.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
@FabianIsensee @claude
Start blosc2.nthreads from BLOSC_NTHREADS and keep NUMEXPR_NUM_THREADS
922573e 
C-Blosc2 uses BLOSC_NTHREADS, when set to a number it accepts, for every
compression and decompression context, whatever nthreads the parameters
ask for.  blosc2.nthreads, and so the CParams/DParams defaults and
numexpr's thread count, ignored it: with BLOSC_NTHREADS=4, Python
reported 28 threads and gave numexpr 28 while C used 4.  Start
blosc2.nthreads from what C-Blosc2 gives a compression context (the new
blosc2_ext.context_nthreads(), via blosc2_ctx_get_cparams()), and say in
set_nthreads()'s docs that the variable wins at the C level.

Importing blosc2 also overrode NUMEXPR_NUM_THREADS, which set_nthreads()
recommends setting; it now leaves numexpr alone then, and otherwise
gives it at most numexpr.MAX_THREADS (BLOSC_NTHREADS can go beyond that,
which numexpr refuses with an error).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
@FabianIsensee @claude
Add release notes for the NDArray fixes
2fe10ab 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FabianIsensee @claude
Resolve basic keys of NDArray.__getitem__/__setitem__ in Cython
37c0ecd 
Keys made of integers (Python or NumPy, not bools), unit-step slices,
one Ellipsis and, for reads, None went through the general path: 8-30 us
of Python per call (ndindex as soon as a NumPy integer, ... or None
appears), more than decompressing a small selection.

NDArray.get_basic_numpy() classifies such keys in Cython, resolves them
against the shape and reads the result under one acquisition of the
handle lock, through _get_slice_cbuffer(), the GIL-releasing read that
get_slice_numpy() now shares with it.  get_basic_start_stop() resolves
the same keys (without None) for __setitem__.  Any other key, or one that
does not fit the shape, gives None and takes the unchanged general path,
which raises the same exceptions as before.  NumPy integers without
__index__ (timedelta64) are left to it too, where they raise IndexError
as in NumPy.

SWMR: a read resolves the key against the cached shape and relies on the
refresh inside the C read; if b2nd_array_t.last_tick shows that this
refresh changed the shape (or the key did not fit, or the read failed),
it refreshes, resolves and reads again, so a key is never resolved
against a stale shape, at one file check per on-disk read instead of
two.  A write refreshes under the handle lock before resolving.

Keys with None now return arrays that own their data (they were views
of a temporary).

Measured (Ryzen 9 7950X): a[3, 5, 7] of a 300x512x512 float32 array
34 -> 25 us in memory and 48 -> 35 us on disk, a[500] of a 1000x1000
array 21.5 -> 13.9 us, a[np.int64(7), np.int32(9)] 37.5 -> 7.3 us,
iterating over 2000 rows 64 -> 44 ms; a[np.int64(i), j] = v 22 us less.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
@FabianIsensee @claude
Lower the per-call overhead of creating arrays and of small writes
37c469c 
Creating a small array cost ~30 us, mostly Python bookkeeping:

- compute_chunks_blocks() and _check_ndarray_kwargs() converted CParams
  and Storage with dataclasses.asdict(), which deep-copies; a shallow
  dict (vars() for Storage, whose fields are plain values) is enough;
- every SChunk wrapper, so every NDArray, built its cparams and dparams
  eagerly under the handle lock; they are now built on first use;
- create_cparams_from_kwargs() looked up Filter.BYTEDELTA.value (a slow
  enum descriptor) twice per filter slot; now once per call;
- is_inside_new_expr() and is_inside_ne_evaluate(), which arange,
  linspace, eye, reshape, fromiter and lazy functions such as clip run
  on every call, used inspect.stack(), which reads the source of every
  frame; walking the frames gives the same answer in under 1 us.

__setitem__ also wrapped plain NumPy values in a SimpleProxy, whose
constructor runs compute_chunks_blocks(); other array types keep the
wrapping, NumPy arrays give the same outcome without it (checked for
contiguous, strided, Fortran-order, 0-d, broadcast and converted values).

Measured (Ryzen 9 7950X): zeros((100, 100)) 31 -> 20 us, eye() 311 ->
179 us, reshape() 437 -> 276 us, clip(a, 0, 1) 84 -> 12 us; a 3x3 write
into a 100x100 array ~12 us less (asdict() ~5 us, SimpleProxy ~7 us).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
@FabianIsensee @claude
Add release notes for the faster basic indexing and array creation
4a3a5e9 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FabianIsensee @claude
Write scalars and NDArray values chunk by chunk in NDArray.__setitem__
88b9578 
a[region] = scalar built np.full() of the whole region, and
a[region] = ndarray decompressed all of the source first: an
uncompressed copy of the region in memory (a fill of a large on-disk
array could exceed RAM), then one set_slice.

Scalars: every chunk the region fully covers gets one shared special
(repeated-value) chunk from SChunk._special_chunk(), and the rest is
written in boxes of bounded size (32 MB, or one chunk) from one buffer
sized to the largest box.  Edge chunks get a special chunk only for
zero, as writing them leaves zeros in their padding (which a resize
brings to light).  Chunks go last to first: a frame compacts its payload
behind an updated chunk, and that tail then holds the tiny special
chunks already written.  Lossy codecs and filters, and dictionaries,
keep compressing the values.

NDArray values: read box by box along the destination's chunk
boundaries into reused buffers and written with one set_slice per box;
when both arrays encode chunks the same way (dtype, chunks, blocks,
lossless cparams) and the region starts at a chunk boundary, covered
chunks without padding are copied compressed.  A value that may share
storage with the destination (same super-chunk or same file) is still
read in full first.

Both hold the handle locks (HandleLocks with the source) for the whole
write, so that other threads see it as atomic as one set_slice.  Regions
smaller than a chunk, and arrays whose frame takes the cross-process
lock (locking=True or BLOSC_LOCKING, destination or source), keep the
single set_slice, which holds that lock from the shape refresh to the
last chunk.  A shared flat chunk index helper now also serves
detect_aligned_chunks(), and __setitem__ marks indexes stale through one
helper.

Measured (300x512x512 float32, default chunks, Ryzen 9 7950X):
a[:] = 0.0 41 -> 0.2 ms with no extra memory (was 303 MB); the first
a[:] = 1 into an existing on-disk array 588 -> 29 ms; a[:] = b with the
same encoding 142 -> 18 ms and 306 -> 28 MB peak.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
This was referenced Nov 3, 2026
Open
Closed
FabianIsensee and others added 3 commits November 3, 2026 12:32
@FabianIsensee @claude
Cast whole chunks with reused buffers in asarray/astype
d9668a2 
asarray() of an NDArray, and so astype(), sliced every chunk with
__getitem__ and cast it into fresh arrays: chunk-sized allocations per
chunk, which glibc trims and page-faults back in each time.  When source
and destination have the same shape, chunks and blocks (the default for
astype), chunks without padding are now decompressed into one reused
buffer, cast item by item into another and compressed from there; edge
chunks keep the per-chunk path (their padding may hold anything, and
casting it could warn).  Chunks encoded the same way in both arrays are
copied compressed (e.g. asarray(a, copy=True, urlpath=...)).

Measured (300x512x512 float32, Ryzen 9 7950X, this change toggled):
astype(np.uint8) 110 -> 56 ms, with nnU-Net-like chunks 286 -> 267 ms;
astype(np.float64) unchanged (compression-bound).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SwMZj6jNaqsesMKnBde8Lr
@FabianIsensee @claude
Add release notes for the chunk-wise writes and casts
5f8f70c 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FabianIsensee @claude
TEMP: build against the c-blosc2 fixes PR branch
a396d88 
Drop this commit before merging: once
Blosc/c-blosc2#835 is merged and released, point
BLOSC2_BUNDLED_VERSION back to Blosc/c-blosc2 at a release tag that
contains it and bump BLOSC2_MIN_VERSION, since update_data() and
insert_data() rely on its rule that a failed chunk write without a copy
leaves the chunk with the caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@FabianIsensee