Skip to content

Fix loading little-endian models on big-endian hosts #970

Open
satyamg1620 wants to merge 2 commits intoexplosion:v8.3.xfrom
satyamg1620:big-endian-support
Open

Fix loading little-endian models on big-endian hosts #970
satyamg1620 wants to merge 2 commits intoexplosion:v8.3.xfrom
satyamg1620:big-endian-support

Conversation

@satyamg1620
Copy link
Copy Markdown

@satyamg1620 satyamg1620 commented Apr 21, 2026

Description

Loading spaCy pipelines (e.g. en_core_web_sm) on big-endian hosts like s390x currently fails with ValueError: Little-endian buffer not supported on big-endian compiler, because thinc's Cython typed memoryviews reject the little-endian arrays that ship in the serialized model. This PR adds a thinc.util.ensure_native_byteorder helper and routes both Ops.asarray and NumpyOps.asarray through it, so deserialized arrays are byteswapped to native order before reaching any Cython path. On little-endian platforms native arrays are returned by identity, so there is no behavioural or performance change.

Types of change

Bug fix — enables thinc (and downstream spaCy) to load little-endian model artifacts on big-endian CPUs (like s390x ).

Checklist

  • I confirm that I have the right to submit this contribution under the project's MIT license.
  • I ran the tests, and all new and existing tests passed.
  • My changes don't require a change to the documentation, or if they do, I've added all required information.

…lising byte order in Ops.asarray

Signed-off-by: satyamg1620 <Satyam.Gupta.3@ibm.com>
…25+ ndim>0 scalar-conversion deprecation

Signed-off-by: satyamg1620 <Satyam.Gupta.3@ibm.com>
@satyamg1620 satyamg1620 marked this pull request as ready for review April 21, 2026 19:32
@satyamg1620 satyamg1620 changed the title [WIP] Fix loading little-endian models on big-endian hosts Fix loading little-endian models on big-endian hosts Apr 21, 2026
@satyamg1620
Copy link
Copy Markdown
Author

@honnibal @ines Can you please review this PR

Copy link
Copy Markdown

@modassarrana89-new modassarrana89-new left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tehbone
Copy link
Copy Markdown

tehbone commented Apr 23, 2026

Shouldn't this also drop the lazy load of BigEndianOps in compat.py?

@satyamg1620
Copy link
Copy Markdown
Author

Shouldn't this also drop the lazy load of BigEndianOps in compat.py?

I think the lazy load isn't in compat.py, it's in thinc/backends/init.py inside _import_extra_cpu_backends(), mirroring the AppleOps pattern.

@tehbone
Copy link
Copy Markdown

tehbone commented Apr 24, 2026

Shouldn't this also drop the lazy load of BigEndianOps in compat.py?

I think the lazy load isn't in compat.py, it's in thinc/backends/init.py inside _import_extra_cpu_backends(), mirroring the AppleOps pattern.

Yes, my mistake. I was thinking of the wrong file. But it would be good to get rid of that if it is now just supported within the base numpy ops

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants