Skip to content

Run tests with ParallelTestRunner, require Julia 1.10 - #1364

Open
devmotion wants to merge 8 commits into
JuliaArrays:masterfrom
devmotion:dmw/parallel-test-runner
Open

devmotion wants to merge 8 commits into
JuliaArrays:masterfrom
devmotion:dmw/parallel-test-runner

Conversation

@devmotion

@devmotion devmotion commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1362.

The test suite takes about 30 min per CI job, and each push started 21 test jobs, which ties up the organisation's shared CI runners. This PR runs the test files in parallel with ParallelTestRunner.jl and removes the group-A/group-B split of the x86 jobs. Locally (M2), the full suite now takes about 3.5 min on Julia 1.10 and 1.13.

ParallelTestRunner requires Julia ≥ 1.10, so this PR also raises the minimum supported Julia version. The PR is split into six commits (plus a commit setting memory_per_worker and its revert):

  1. Require Julia 1.10: julia = "1.10" compat, version bumped to 1.10.0, CI and docs use lts instead of 1.6.
  2. Remove code for Julia < 1.10: removes the VERSION branches that are now dead in src/ and test/, and replaces @_inline_meta with @inline. Statistics is removed from [deps]: it was only needed to load StaticArraysStatisticsExt on Julia versions without package extensions.
  3. Run tests with ParallelTestRunner; drop test groups:
    • Test files are discovered automatically and each runs in its own module on a pool of worker processes. Only StaticArrays, Test and the helpers in test/testutil.jl are shared; the files load the other packages they use themselves.
    • check_bounds_no.jl runs on a dedicated worker started with --check-bounds=no (instead of a separate run(...)), so its results show up in the summary.
    • CI: 21 → 15 test jobs, and superseded CI runs on PRs are cancelled.
  4. Run CI with native Julia binaries: the macOS runners are Apple Silicon, but the jobs used x64 Julia under Rosetta. CI now uses setup-julia@v3, which installs Julia for the runner's own architecture by default.
  5. Cache the Julia depot in CI with julia-actions/cache. Among others, this keeps ParallelTestRunner's history of test durations (stored in a scratchspace), which it uses to start the slowest tests first.
  6. Use two test workers on the macOS runners: ParallelTestRunner's default number of workers is based on the available memory (docs). On the Linux and Windows runners it uses all 4 CPU threads, but on the macOS runners (3 CPU threads, 7 GB) the reported available memory only allows a single worker, which made the macOS jobs the bottleneck (e.g., ~59 min for Julia 1.10 vs. ~29 min with 3 workers). The ParallelTestRunner docs warn that too many workers can slow down tests on low-memory macOS machines, so this PR uses --jobs=2 on macOS as a compromise: in a CI run with 3 workers, each test file took ~1.3x longer than with a single worker, without a noticeable increase in GC time, and with 2 workers the memory usage should be lower.

Changes to be aware of

  • StaticArraysStatisticsExt is loaded only when Statistics is loaded.
  • Subsets of tests are selected via ParallelTestRunner's filters: Pkg.test("StaticArrays"; test_args=["SVector", "MVector"]) still works, but names are matched case-sensitively by prefix (previously: case-insensitive exact matches).
  • Every test file now starts from ParallelTestRunner's Random.seed!(1) instead of Random.seed!(42). No test was affected.
  • test/util.jl, which was never run before, is now part of the test suite. One outdated test in it (constructing a SizedMatrix{2,3} from a 3×2 matrix) was fixed.
  • Three tests compared error messages containing the names of types defined in the test files. These names now include the test file's module, so the expected messages interpolate the type instead of hardcoding its name.

🤖 Generated with Claude Code

devmotion and others added 8 commits October 15, 2026 10:12
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Statistics is no longer a hard dependency: it was only needed to load
StaticArraysStatisticsExt on Julia versions without package extensions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each test file now runs in its own module on a pool of worker processes,
with only StaticArrays, Test and the helpers in testutil.jl shared; files
load any other packages they use themselves. Tests are discovered
automatically, which also enables the previously unused test/util.jl.

Since a single CI job now runs the whole suite in parallel, the
group-A/group-B split of the x86 jobs is removed, and superseded CI runs
on PRs are cancelled.

Fixes JuliaArrays#1362

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macOS runners are Apple Silicon, so x64 Julia ran under Rosetta.
setup-julia v3 uses the runner's architecture by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Among others, this keeps ParallelTestRunner's history of test durations
(stored in a scratchspace), which it uses to start slow tests first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Test workers stay below 1 GiB of RSS, whereas ParallelTestRunner assumes
2 GiB per worker by default, which limited CI jobs (e.g. on the macOS
runners) to fewer workers than CPU threads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This reverts commit b1c6acb.

The premise was wrong: on CI (with coverage) test workers reach 1.5-2.7 GB
of RSS. Use ParallelTestRunner's default instead, which accounts for the
available memory (e.g., on the macOS runners).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the macOS runners, ParallelTestRunner's default number of workers,
which is based on the reported available memory, is one. Two workers
roughly halve the test time while leaving some memory headroom.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move tests to ParallelTestRunner

1 participant