snowflake-connector-python: add build-snowflake-connector-python.yml for riscv64 wheels - #343
Merged
Conversation
…for riscv64 wheels Builds the nanoarrow C++/Cython extension (`nanoarrow_arrow_iterator`) from an upstream checkout with cibuildwheel, per-interpreter for cp312/cp313/cp314/cp314t. The project's `[tool.cibuildwheel]` table is written for its own release flow, so three settings are overridden: the `AUDITWHEEL_PLAT=manylinux2014_$(uname -m)` environment (no manylinux2014 policy exists for riscv64), the empty `repair-wheel-command` (ci/build_linux.sh runs auditwheel separately), and `test-skip = "*"` (upstream tests wheels in a downstream job). Tests mirror tox's `py3XX-unit-parallel-ci` environment: the unit suite minus the aio tests, with a JRE installed for the Wiremock server several auth/oauth/proxy tests spin up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…le group cibuildwheel 4 removed that group name (free-threaded builds are always selectable) and hard-errors on it, so every matrix job failed before building. `enable` merges rather than replaces across config sources, so it has to come out of the checkout's pyproject.toml. Also install pytest-xdist, which registers the xdist_group marker test_detect_platforms.py uses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… network-bound modules Validated the test invocation against a local (x86-independent) run of the same suite: with the four exclusions below it goes from "still running after 45 minutes" to 66s on 11 workers, with the only failures being the two arrow-stream tests that need the compiled extension (absent in that pure-Python check) and one needing azure-identity, now added to the test requirements. Excluded: test_ocsp (live OCSP responders, one of them an unreachable preprod host, so it stalls on TCP timeouts), test_proxies (2-3 Wiremock JVMs per test streaming large result sets), and test_ssl_partial_chain_handshake (real TLS handshake against an external host). The three deselected tests all assert a loadable native minicore blob, which upstream ships for eight platforms but not riscv64. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
luhenry
force-pushed
the
snowflake-connector-python
branch
from
August 25, 2026 10:22
09eaadd to
e040786
Compare
Installing the built wheel pulled cryptography>=46.0.5 and picked the newest version across public PyPI and our registry - 50.0.0, which exists only as an sdist for riscv64 - then failed in its build backend with "Target triple not supported by rustup: riscv64-unknown-linux-gnu". PIP_ONLY_BINARY=:all: makes pip backtrack to the newest riscv64 wheel we ship (49.0.0). Scoped to CIBW_TEST_ENVIRONMENT per gotcha 12 so the build backend can still compile Cython from sdist. Verified the whole closure - runtime deps plus test requires - resolves to riscv64 wheels for cp312, cp314 and cp314t. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…y permission tests The wheel now builds, installs and runs the unit suite on the riscv64 runners: 1492 passed, 5 failed, 10 errored. The 10 errors are all WiremockClient startup timeouts. WIREMOCK_START_MAX_RETRY_COUNT gives the standalone server 12 seconds to answer /__admin/health; with four pytest-xdist workers each starting their own JVM that ceiling is regularly missed on this hardware, while the ~50 other wiremock-backed tests in the same run pass - i.e. it is a fixed-timeout artifact, not a defect. Patch it to 120s; the wait loop returns as soon as the health check succeeds. The 5 failures each chmod a path and assert the resulting EACCES. cibuildwheel runs the test command as root, which bypasses file permission bits, so the error never arrives (tmp paths in the log are under /tmp/pytest-of-root). Deselect them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_detect_platforms passes a deliberately generous 1s budget to detect_platforms() so that none of its six threaded probes gets labelled '<name>_timeout'. On the riscv64 runner that is not generous enough: under the free-threaded interpreter, with four pytest-xdist workers competing, botocore's first STS service-model load overshoots it and test_no_platforms_detected saw ['has_aws_identity_timeout'] instead of []. It failed on cp314t only; cp312/cp313/cp314 passed the same assertion. Patch the module's 1s sites up to 30s. All HTTP is mocked to raise ConnectTimeout outright, so nothing waits and the ceiling costs no wall time; the sites that deliberately exercise the timeout path are untouched. Watch patches/ in the pull_request paths filter, as build-torch.yml does, so patch-only changes still run CI.
…thout xdist Two failures from the last run, unrelated to each other. test_auth_mfa: Auth.authenticate() abandons the MFA wait thread on timeout, and the request mock's trailing 'mock_cnt += 1' then fires ~9s later, inside the next sub-case that has already reset the counter - so the mock returns its empty else-branch and authenticate() raises KeyError: 'data' instead of the expected Error. It is the race the test is already skipped for on Windows, reached here because the fourth sub-case's setup is slow enough to still be running. Patch the mock to advance the counter before it sleeps. Reproduced and fixed on macOS/arm64 by inserting a sleep after the fourth sub-case's 'mock_cnt = 2', so this is timing, not architecture. cp314t: pytest's controller took a SIGSEGV inside execnet's receiver thread (gateway_base._read_int4, 'Cannot show all threads while the GIL is disabled') - pure-Python frames, no connector code involved. Drop to '-n 0' there, which makes pytest-xdist skip gateway creation entirely; the other three interpreters keep upstream's parallel invocation.
luhenry
force-pushed
the
snowflake-connector-python
branch
from
August 25, 2026 14:53
c295265 to
35c8f2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a riscv64 wheel build for snowflake-connector-python 4.7.2.
What gets built
The wheel's value-add on riscv64 is the
nanoarrow_arrow_iteratorextension — a Cython module over ~25 vendored C++/C sources (nanoarrow, nanoarrow_ipc, flatcc, the Snowflake Arrow converters). PyPI ships per-interpretercpXY-cpXYwheels for x86_64/aarch64/macOS/Windows but nothing for riscv64, sopip installthere falls back to the sdist and needs a full Cython + C++17 toolchain.Shape: build-from-checkout — upstream prunes
test/from the sdist, so the tests only exist in the git tree.Deviations from upstream's
[tool.cibuildwheel]The project's table is written for its own release flow (
ci/build_linux.shdrives the real release build, cibuildwheel only serves their PR CI), so three settings are overridden:environment = {AUDITWHEEL_PLAT="manylinux2014_$(uname -m)"}CIBW_ENVIRONMENT: PIP_EXTRA_INDEX_URL=…manylinux2014policy for riscv64; keeping the image'smanylinux_2_39_riscv64is required forauditwheel repairto succeed[linux] repair-wheel-command = ""auditwheel repairci/build_linux.sh; without it the wheel keeps a barelinux_riscv64tagtest-skip = "*"CIBW_TEST_SKIP: ''enable = ["cpython-freethreading"]is inherited, which is what makes thecp314tmatrix entry buildable (the.pyxcarries# cython: freethreading_compatible = True).minicore
The repo carries prebuilt
libsf_mini_coreblobs for eight platforms;ci/build_linux.shdeletes all but the host's before building. There is no riscv64 blob, andsetup.py's pruner no-ops on unrecognised architectures — which would leave foreign-arch.sos in the wheel forauditwheelto choke on. The workflow drops them all;_utils._CoreLoader._loadswallows a missing-library error, so this only costs the (optional) minicore telemetry path.Testing
Mirrors tox's
py3XX-unit-parallel-cienvironment:pytest -m "unit and not sso and not pandas and not lambda and not aio" test/unit, minus coverage and minus the aio suite (itsaiohttp/aiobotoextras have no riscv64 wheels).java-21-openjdk-headlessis installed from Rocky 10's AppStream so the Wiremock server that the auth/oauth/proxy tests spin up can run;pyarrow- andpandas-gated tests self-skip. A post-build step asserts a.soactually landed in each wheel.Test exclusions
The suite runs 1492 tests. Excluded:
test/unit/aio,test_ocsp,test_proxies,test_ssl_partial_chain_handshake— no riscv64 wheels for the aio extras; the rest reach out to live OCSP/TLS endpoints or start 2-3 Wiremock JVMs per test.TestCoreLoader::test_e2e,test_get_present_binaries_contains_expected_paths,test_pruner_agrees_with_runtime_loader— need a native minicore blob, which upstream ships for eight platforms but not riscv64 (see above).test_warn_config_file_owner,test_log_debug_config_file_parent_dir_permissions,TestSFDictFileCache::test_read_only,test_config_file_inaccessible_path,test_put_error— eachchmods a path and asserts the resultingEACCES. cibuildwheel runs the test command as root, which bypasses the permission bits, so the error never arrives.Patches
patches/snowflake-connector-python/4.7.2/0001-test-wiremock-raise-the-server-start-timeout-to-120s.patchraisesWIREMOCK_START_MAX_RETRY_COUNTfrom 12 to 120.WiremockClientgives the standalone server that many seconds to answer/__admin/health; with four pytest-xdist workers each starting their own JVM, ~10 of the auth/oauth/redirect tests missed the 12s ceiling while the ~50 other wiremock-backed tests in the same run passed — a fixed-timeout artifact, not a defect. The wait loop returns as soon as the health check succeeds, so the larger ceiling costs nothing on faster hardware.Upstream-Status: Inappropriate [native runner specific].patches/snowflake-connector-python/4.7.2/0002-test-detect_platforms-raise-the-generous-timeout-from.patchraises the deliberately-generousplatform_detection_timeout_secondsintest/unit/test_detect_platforms.pyfrom 1s to 30s.detect_platforms()runs six probes in aThreadPoolExecutorand labels any that miss the budget<name>_timeout; the tests that want no such label pass 1s, with an inline comment saying the value exists only "to make sure no Thread-based timeout messes the results". On the riscv64 runner 1s is not generous enough —has_aws_identitybuilds a boto3 STS client, and botocore's first service-model load, with four pytest-xdist workers competing under the free-threaded interpreter, overshoots it:test_no_platforms_detectedsaw['has_aws_identity_timeout']instead of[]oncp314twhilecp312/cp313/cp314passed the same assertion.test/csp_helpers.pypatchesurllib3.connection.HTTPConnection.requestto raiseConnectTimeoutoutright, so no probe ever waits on the network and the larger ceiling adds no wall-clock time; the sites that deliberately exercise the timeout path (None,0,EXPECTED_MAX_TIMEOUT_FOR_PLATFORM_DETECTION) are left alone.Upstream-Status: Inappropriate [native runner specific].patches/snowflake-connector-python/4.7.2/0003-test-auth-fix-the-mock_cnt-race-that-test_auth_mfa-s.patchcloses themock_cntrace intest_auth_mfa.Auth.authenticate()runs the MFA wait in a daemonThreadand abandons it witht.join(timeout=timeout); the test's third sub-case depends on that, passingtimeout=1against a mock whosemock_cnt == 1branch sleeps 10 seconds. The abandoned thread then reaches the mock's trailingmock_cnt += 1about nine seconds later — by which point the fourth sub-case has setmock_cnt = 2to select thedata: Noneresponse. The stray increment makes it 3, the mock returns its emptyelsedict, andauthenticate()raisesKeyError: 'data'instead of the expectedsnowflake.connector.errors.Error. The patch reads and advances the counter in one step at the top of the mock, before the sleep, so a call's effect on the counter is complete before the caller can walk away from it; branch selection is unchanged. This is the same race the test already carries anIS_WINDOWSskip for, and it is not architecture-specific — insertingtime.sleep(11)after the fourth sub-case'smock_cnt = 2reproducesKeyError: 'data'on macOS/arm64 and the patch makes it pass.Upstream-Status: To upstream.cp314t runs the suite in-process
On
cp314tthe pytest controller took aSIGSEGVinside execnet's receiver thread (gateway_base._read_int4, underFatal Python error: Segmentation fault/<Cannot show all threads while the GIL is disabled>) — pure-Python frames, no connector code on the stack, and intermittent (the same job completed the full suite on an earlier run). pytest-xdist's gateways are the only thing that spawns that thread, socp314tpasses-n 0, which makes xdist skip gateway creation entirely (plugin.pyforcesdist = "no",tx = []). The three GIL-ful interpreters keep upstream's parallel-n auto --dist loadfileinvocation.