sentencepiece: build wheels for riscv64

[gounthar]

Adds build-sentencepiece.yml so sentencepiece riscv64 wheels build on the native ubuntu-24.04-riscv runner via cibuildwheel, and publish to the GitLab index on main.

This is the build workflow that the auto-generated #157 asks for ("No build workflow found at .github/workflows/build-sentencepiece.yml"). sentencepiece is already listed in ci_scripts/packages.txt, so no manifest change is needed here.

Shape

Mirrors build-numpy.yml:

• checks out google/sentencepiece at the requested tag (submodules: true, the C++ core is vendored)
• points cibuildwheel at the python/ subdirectory via package-dir, since that is where setup.py/pyproject.toml live
• matrix: cp312, cp313, cp314, cp314t
• publish job is gated on main and reuses the shared publish-to-gitlab action

Wheels are also tested inline: cibuildwheel installs each freshly built wheel into an isolated venv and runs the package's own pytest suite against it (CIBW_TEST_SOURCES + CIBW_TEST_COMMAND). Two files are skipped: gen_stubs_test.py imports a repo-level helper that is not part of the copied test tree, and clean_sentencepiece_test_manual.py is a manual cleanup script rather than a pytest module.

Verification

Built and tested on a native riscv64 board (BananaPi BPI-F3), cp312, cibuildwheel 4.1.0, manylinux_2_39_riscv64:

60 passed, 1 warning in 281.29s

produced sentencepiece-0.2.2-cp312-cp312-manylinux_2_39_riscv64.whl.

Refs #164

Trigger: sentencepiece:0.2.2

[threexc]

@gounthar I've been updating the development guide and I've also pushed some other workflows for matplotlib and cffi since your PRs were opened. Can you refactor and update them so we can review again? Keep in mind, we want the workflows to closely mirror the upstream workflows as much as possible, so prefer that over similarity with our existing workflows when possible.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://riseproject-dev.github.io/python-wheels/pr-preview/pr-165/
Built to branch gh-pages at 2026-08-04 20:09 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[gounthar]

Thanks, that makes sense. I had built this one by copying build-numpy.yml, which is exactly the habit you're pointing at. Pushed a rebase plus a rewrite that starts from google/sentencepiece's own wheel.yml instead.

The part that changed the most: I pinned cibuildwheel to 3.4.0, the version upstream pins in requirements/cibuildwheel.txt. With that version the project's own [tool.cibuildwheel] table works as-is, so the CIBW_TEST_SOURCES and CIBW_TEST_COMMAND overrides I had are gone. I didn't reuse their requirements files wholesale though, since base.txt also pins twine's chain (cffi, cryptography) and those would get compiled from source on the runner. Installing cibuildwheel alone at their pinned version felt like the smaller deviation, but if you'd rather see the hash-pinned install kept I can try it.

I also picked up the data/*.bin copy into src/sentencepiece/package_data that upstream does before building, which I had simply missed the first time. And their free-threading job, running against 3.14t rather than 3.13t to match the default matrix. That last one is the piece I'm least sure about: it installs sentencepiece[test] and pytest-run-parallel for 3.14t, and I haven't checked whether protobuf resolves there on riscv64. If the trigger run goes red on it I'll just drop the job.

Two things I'd like your read on. I dropped upstream's sdist round-trip (build_sdist.sh, then building a wheel from the sdist) and the twine check that validates it, going by the guide's rule about sdist builds that nothing consumes. On native riscv64 that round-trip is a second full C++ build per Python version, which is why I leaned that way, but it does mean we lose the twine --strict check on the wheels. And I made publish need the free-threading job, which is stricter than what pillow, cffi and numpy do. Both are easy to put back if you'd rather I matched the existing workflows.

zstandard gets the same treatment next, along with the QEMU point justeph raised on the upstream PR. You've been shaping these conventions for a while, so if I've read the guide the wrong way somewhere, say so and I'll redo it.

[threexc]

@gounthar I don't think we need the free-threading job in the workflow - AFAICT, it's only used for 3.13t upstream, and 3.14t is already covered by the default cibuildwheel step.

On a related note, the way this workflow sets up the matrix should be tweaked - I think it'd be better if you used CIBW_SKIP (with an extra comment) in the env to avoid building the older cp39, cp310, and cp311 versions, since this will make it a little easier to compare with upstream. It has the "downside" of serializing the build on one runner instead of parallelizing them across multiple, but I feel that it will be better overall for resource usage, unless the upstream developers would prefer to parallelize when they accept riscv64 support.

[gounthar]

Both make sense, though there's one thing I want to flag before I delete the free-threading job.

On the matrix: agreed. I'll move to CIBW_SKIP in env and let a single cibuildwheel invocation walk the versions. For the record on the resource trade, the four build jobs run 28 to 30 minutes each in parallel right now, so a run finishes in about 42 minutes wall. Serialized that's closer to two hours on one riscv64 runner. Still well inside the job limit, and if the goal is fewer runners tied up and an easier diff against upstream, that seems like the right call. Just wanted the number written down somewhere.

On free-threading: you're right that the wheel is covered. I checked cibuildwheel 3.4.0 and the gate in selector.py only matches cp313t-*, so cp314t gets built by default. That also means my CIBW_ENABLE: cpython-freethreading is doing nothing here, since there's no cp313t in the matrix. I'll drop that line either way.

The part that goes away with the job is pytest --parallel-threads 4, which isn't the same test cibuildwheel runs (that one is single-threaded via the pyproject test-command). And it has found something. On the 2026-07-28 re-run:

test/sentencepiece_test.py::TestSentencepieceProcessor::test_trainer_with_normalizer
Fatal Python error: Aborted
(core dumped) pytest -v --parallel-threads 4

The same job passed on the earlier run that morning, so it's intermittent, which is roughly what I'd expect from a real race. I haven't checked whether it reproduces on x86 3.13t, so I can't tell you yet if this is riscv64-specific or just upstream free-threading being upstream free-threading.

Happy to drop the job as you asked. Chasing upstream thread-safety probably isn't this repo's job. I'd just rather you made that call knowing it caught a crash. Want me to file an issue with the trace before removing it, or just take it out?

[threexc]

Both make sense, though there's one thing I want to flag before I delete the free-threading job.

On the matrix: agreed. I'll move to CIBW_SKIP in env and let a single cibuildwheel invocation walk the versions. For the record on the resource trade, the four build jobs run 28 to 30 minutes each in parallel right now, so a run finishes in about 42 minutes wall. Serialized that's closer to two hours on one riscv64 runner. Still well inside the job limit, and if the goal is fewer runners tied up and an easier diff against upstream, that seems like the right call. Just wanted the number written down somewhere.

On free-threading: you're right that the wheel is covered. I checked cibuildwheel 3.4.0 and the gate in selector.py only matches cp313t-*, so cp314t gets built by default. That also means my CIBW_ENABLE: cpython-freethreading is doing nothing here, since there's no cp313t in the matrix. I'll drop that line either way.

The part that goes away with the job is pytest --parallel-threads 4, which isn't the same test cibuildwheel runs (that one is single-threaded via the pyproject test-command). And it has found something. On the 2026-07-28 re-run:

test/sentencepiece_test.py::TestSentencepieceProcessor::test_trainer_with_normalizer
Fatal Python error: Aborted
(core dumped) pytest -v --parallel-threads 4

The same job passed on the earlier run that morning, so it's intermittent, which is roughly what I'd expect from a real race. I haven't checked whether it reproduces on x86 3.13t, so I can't tell you yet if this is riscv64-specific or just upstream free-threading being upstream free-threading.

Happy to drop the job as you asked. Chasing upstream thread-safety probably isn't this repo's job. I'd just rather you made that call knowing it caught a crash. Want me to file an issue with the trace before removing it, or just take it out?

That test difference is a good observation. I would file an issue and see if they get back to you quickly. Best-case, it gets triaged and there's a patch to backport. We want to build wheels in a way that avoids bogged down chasing edge case issues, but we also need to make sure that they're going to work for users.

Regarding the long build time, that's OK if we're confident it will succeed. I wonder if we could propose a similar parallelized wheel build upstream, bringing them in-line with what's done here?