Skip to content

snowflake-connector-python: add build-snowflake-connector-python.yml for riscv64 wheels - #343

Merged
luhenry merged 8 commits into
mainfrom
snowflake-connector-python
Aug 25, 2026
Merged

snowflake-connector-python: add build-snowflake-connector-python.yml for riscv64 wheels#343
luhenry merged 8 commits into
mainfrom
snowflake-connector-python

Conversation

@luhenry

@luhenry luhenry commented Aug 25, 2026

Copy link
Copy Markdown
Member

Adds a riscv64 wheel build for snowflake-connector-python 4.7.2.

What gets built

The wheel's value-add on riscv64 is the nanoarrow_arrow_iterator extension — a Cython module over ~25 vendored C++/C sources (nanoarrow, nanoarrow_ipc, flatcc, the Snowflake Arrow converters). PyPI ships per-interpreter cpXY-cpXY wheels for x86_64/aarch64/macOS/Windows but nothing for riscv64, so pip install there falls back to the sdist and needs a full Cython + C++17 toolchain.

Shape: build-from-checkout — upstream prunes test/ from the sdist, so the tests only exist in the git tree.

Deviations from upstream's [tool.cibuildwheel]

The project's table is written for its own release flow (ci/build_linux.sh drives the real release build, cibuildwheel only serves their PR CI), so three settings are overridden:

upstream override why
environment = {AUDITWHEEL_PLAT="manylinux2014_$(uname -m)"} CIBW_ENVIRONMENT: PIP_EXTRA_INDEX_URL=… there is no manylinux2014 policy for riscv64; keeping the image's manylinux_2_39_riscv64 is required for auditwheel repair to succeed
[linux] repair-wheel-command = "" the cibuildwheel default auditwheel repair upstream repairs out-of-band in ci/build_linux.sh; without it the wheel keeps a bare linux_riscv64 tag
test-skip = "*" CIBW_TEST_SKIP: '' upstream tests wheels in a separate job against a live Snowflake account

enable = ["cpython-freethreading"] is inherited, which is what makes the cp314t matrix entry buildable (the .pyx carries # cython: freethreading_compatible = True).

minicore

The repo carries prebuilt libsf_mini_core blobs for eight platforms; ci/build_linux.sh deletes all but the host's before building. There is no riscv64 blob, and setup.py's pruner no-ops on unrecognised architectures — which would leave foreign-arch .sos in the wheel for auditwheel to choke on. The workflow drops them all; _utils._CoreLoader._load swallows a missing-library error, so this only costs the (optional) minicore telemetry path.

Testing

Mirrors tox's py3XX-unit-parallel-ci environment: pytest -m "unit and not sso and not pandas and not lambda and not aio" test/unit, minus coverage and minus the aio suite (its aiohttp/aioboto extras have no riscv64 wheels). java-21-openjdk-headless is installed from Rocky 10's AppStream so the Wiremock server that the auth/oauth/proxy tests spin up can run; pyarrow- and pandas-gated tests self-skip. A post-build step asserts a .so actually landed in each wheel.

Test exclusions

The suite runs 1492 tests. Excluded:

  • test/unit/aio, test_ocsp, test_proxies, test_ssl_partial_chain_handshake — no riscv64 wheels for the aio extras; the rest reach out to live OCSP/TLS endpoints or start 2-3 Wiremock JVMs per test.
  • TestCoreLoader::test_e2e, test_get_present_binaries_contains_expected_paths, test_pruner_agrees_with_runtime_loader — need a native minicore blob, which upstream ships for eight platforms but not riscv64 (see above).
  • test_warn_config_file_owner, test_log_debug_config_file_parent_dir_permissions, TestSFDictFileCache::test_read_only, test_config_file_inaccessible_path, test_put_error — each chmods a path and asserts the resulting EACCES. cibuildwheel runs the test command as root, which bypasses the permission bits, so the error never arrives.

Patches

patches/snowflake-connector-python/4.7.2/0001-test-wiremock-raise-the-server-start-timeout-to-120s.patch raises WIREMOCK_START_MAX_RETRY_COUNT from 12 to 120. WiremockClient gives the standalone server that many seconds to answer /__admin/health; with four pytest-xdist workers each starting their own JVM, ~10 of the auth/oauth/redirect tests missed the 12s ceiling while the ~50 other wiremock-backed tests in the same run passed — a fixed-timeout artifact, not a defect. The wait loop returns as soon as the health check succeeds, so the larger ceiling costs nothing on faster hardware. Upstream-Status: Inappropriate [native runner specific].

patches/snowflake-connector-python/4.7.2/0002-test-detect_platforms-raise-the-generous-timeout-from.patch raises the deliberately-generous platform_detection_timeout_seconds in test/unit/test_detect_platforms.py from 1s to 30s. detect_platforms() runs six probes in a ThreadPoolExecutor and labels any that miss the budget <name>_timeout; the tests that want no such label pass 1s, with an inline comment saying the value exists only "to make sure no Thread-based timeout messes the results". On the riscv64 runner 1s is not generous enough — has_aws_identity builds a boto3 STS client, and botocore's first service-model load, with four pytest-xdist workers competing under the free-threaded interpreter, overshoots it: test_no_platforms_detected saw ['has_aws_identity_timeout'] instead of [] on cp314t while cp312/cp313/cp314 passed the same assertion. test/csp_helpers.py patches urllib3.connection.HTTPConnection.request to raise ConnectTimeout outright, so no probe ever waits on the network and the larger ceiling adds no wall-clock time; the sites that deliberately exercise the timeout path (None, 0, EXPECTED_MAX_TIMEOUT_FOR_PLATFORM_DETECTION) are left alone. Upstream-Status: Inappropriate [native runner specific].

patches/snowflake-connector-python/4.7.2/0003-test-auth-fix-the-mock_cnt-race-that-test_auth_mfa-s.patch closes the mock_cnt race in test_auth_mfa. Auth.authenticate() runs the MFA wait in a daemon Thread and abandons it with t.join(timeout=timeout); the test's third sub-case depends on that, passing timeout=1 against a mock whose mock_cnt == 1 branch sleeps 10 seconds. The abandoned thread then reaches the mock's trailing mock_cnt += 1 about nine seconds later — by which point the fourth sub-case has set mock_cnt = 2 to select the data: None response. The stray increment makes it 3, the mock returns its empty else dict, and authenticate() raises KeyError: 'data' instead of the expected snowflake.connector.errors.Error. The patch reads and advances the counter in one step at the top of the mock, before the sleep, so a call's effect on the counter is complete before the caller can walk away from it; branch selection is unchanged. This is the same race the test already carries an IS_WINDOWS skip for, and it is not architecture-specific — inserting time.sleep(11) after the fourth sub-case's mock_cnt = 2 reproduces KeyError: 'data' on macOS/arm64 and the patch makes it pass. Upstream-Status: To upstream.

cp314t runs the suite in-process

On cp314t the pytest controller took a SIGSEGV inside execnet's receiver thread (gateway_base._read_int4, under Fatal Python error: Segmentation fault / <Cannot show all threads while the GIL is disabled>) — pure-Python frames, no connector code on the stack, and intermittent (the same job completed the full suite on an earlier run). pytest-xdist's gateways are the only thing that spawns that thread, so cp314t passes -n 0, which makes xdist skip gateway creation entirely (plugin.py forces dist = "no", tx = []). The three GIL-ful interpreters keep upstream's parallel -n auto --dist loadfile invocation.

luhenry and others added 4 commits August 25, 2026 12:21
…for riscv64 wheels

Builds the nanoarrow C++/Cython extension (`nanoarrow_arrow_iterator`) from an
upstream checkout with cibuildwheel, per-interpreter for cp312/cp313/cp314/cp314t.

The project's `[tool.cibuildwheel]` table is written for its own release flow, so
three settings are overridden: the `AUDITWHEEL_PLAT=manylinux2014_$(uname -m)`
environment (no manylinux2014 policy exists for riscv64), the empty
`repair-wheel-command` (ci/build_linux.sh runs auditwheel separately), and
`test-skip = "*"` (upstream tests wheels in a downstream job).

Tests mirror tox's `py3XX-unit-parallel-ci` environment: the unit suite minus the
aio tests, with a JRE installed for the Wiremock server several auth/oauth/proxy
tests spin up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…le group

cibuildwheel 4 removed that group name (free-threaded builds are always
selectable) and hard-errors on it, so every matrix job failed before building.
`enable` merges rather than replaces across config sources, so it has to come
out of the checkout's pyproject.toml. Also install pytest-xdist, which registers
the xdist_group marker test_detect_platforms.py uses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… network-bound modules

Validated the test invocation against a local (x86-independent) run of the same
suite: with the four exclusions below it goes from "still running after 45
minutes" to 66s on 11 workers, with the only failures being the two arrow-stream
tests that need the compiled extension (absent in that pure-Python check) and one
needing azure-identity, now added to the test requirements.

Excluded: test_ocsp (live OCSP responders, one of them an unreachable preprod
host, so it stalls on TCP timeouts), test_proxies (2-3 Wiremock JVMs per test
streaming large result sets), and test_ssl_partial_chain_handshake (real TLS
handshake against an external host). The three deselected tests all assert a
loadable native minicore blob, which upstream ships for eight platforms but not
riscv64.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@luhenry
luhenry force-pushed the snowflake-connector-python branch from 09eaadd to e040786 Compare August 25, 2026 10:22
luhenry and others added 4 commits August 25, 2026 12:31
Installing the built wheel pulled cryptography>=46.0.5 and picked the newest
version across public PyPI and our registry - 50.0.0, which exists only as an
sdist for riscv64 - then failed in its build backend with "Target triple not
supported by rustup: riscv64-unknown-linux-gnu".

PIP_ONLY_BINARY=:all: makes pip backtrack to the newest riscv64 wheel we ship
(49.0.0). Scoped to CIBW_TEST_ENVIRONMENT per gotcha 12 so the build backend
can still compile Cython from sdist. Verified the whole closure - runtime deps
plus test requires - resolves to riscv64 wheels for cp312, cp314 and cp314t.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…y permission tests

The wheel now builds, installs and runs the unit suite on the riscv64 runners:
1492 passed, 5 failed, 10 errored.

The 10 errors are all WiremockClient startup timeouts. WIREMOCK_START_MAX_RETRY_COUNT
gives the standalone server 12 seconds to answer /__admin/health; with four
pytest-xdist workers each starting their own JVM that ceiling is regularly
missed on this hardware, while the ~50 other wiremock-backed tests in the same
run pass - i.e. it is a fixed-timeout artifact, not a defect. Patch it to 120s;
the wait loop returns as soon as the health check succeeds.

The 5 failures each chmod a path and assert the resulting EACCES. cibuildwheel
runs the test command as root, which bypasses file permission bits, so the
error never arrives (tmp paths in the log are under /tmp/pytest-of-root).
Deselect them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_detect_platforms passes a deliberately generous 1s budget to
detect_platforms() so that none of its six threaded probes gets labelled
'<name>_timeout'. On the riscv64 runner that is not generous enough: under
the free-threaded interpreter, with four pytest-xdist workers competing,
botocore's first STS service-model load overshoots it and
test_no_platforms_detected saw ['has_aws_identity_timeout'] instead of [].
It failed on cp314t only; cp312/cp313/cp314 passed the same assertion.

Patch the module's 1s sites up to 30s. All HTTP is mocked to raise
ConnectTimeout outright, so nothing waits and the ceiling costs no wall
time; the sites that deliberately exercise the timeout path are untouched.

Watch patches/ in the pull_request paths filter, as build-torch.yml does,
so patch-only changes still run CI.
…thout xdist

Two failures from the last run, unrelated to each other.

test_auth_mfa: Auth.authenticate() abandons the MFA wait thread on timeout,
and the request mock's trailing 'mock_cnt += 1' then fires ~9s later, inside
the next sub-case that has already reset the counter - so the mock returns
its empty else-branch and authenticate() raises KeyError: 'data' instead of
the expected Error. It is the race the test is already skipped for on
Windows, reached here because the fourth sub-case's setup is slow enough to
still be running. Patch the mock to advance the counter before it sleeps.
Reproduced and fixed on macOS/arm64 by inserting a sleep after the fourth
sub-case's 'mock_cnt = 2', so this is timing, not architecture.

cp314t: pytest's controller took a SIGSEGV inside execnet's receiver thread
(gateway_base._read_int4, 'Cannot show all threads while the GIL is
disabled') - pure-Python frames, no connector code involved. Drop to '-n 0'
there, which makes pytest-xdist skip gateway creation entirely; the other
three interpreters keep upstream's parallel invocation.
@luhenry
luhenry force-pushed the snowflake-connector-python branch from c295265 to 35c8f2a Compare August 25, 2026 14:53
@luhenry
luhenry merged commit cc85654 into main Aug 25, 2026
9 checks passed
@luhenry
luhenry deleted the snowflake-connector-python branch August 25, 2026 16:04
@luhenry luhenry linked an issue Aug 26, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

snowflake-connector-python riscv64 support

1 participant