Repository navigation
perf(witan): run the witan test suite in parallel - #517
Merged
Merged
Conversation
The `witan (memory + CLI)` job ran 1630 tests serially and took 5-8 minutes, almost all of it waiting on `omnigraph` subprocesses against a per-test store. The tests are already independent: each one inits its own store under tmp_path, and testsupport/hermetic.py gives every process its own fake HOME. So they spread across cores cleanly. Run them with `-n auto --dist worksteal` in witan-tests.yml, publish-witan.yml and `just test-witan-council`. No tests are removed or skipped. Locally, -n 4 (the ubuntu-latest core count) gives 1630 passed in ~67s, against ~7 minutes serially on the same machine, and it stayed stable across repeated and oversubscribed (-n 32) runs. Checked under xdist: - The hermetic leak guard still fails the run. The controller snapshots the real home before the workers start. A probe test that wrote to the real ~/.config/witan exited 1 with AGENT_KIT_STRICT_HERMETICITY=1. - Workers still find the omnigraph binary with ~/.local/bin stripped from PATH, with no requires_omnigraph skips. They inherit the PATH entry the controller re-added.
…mp dir It set TMPDIR without resetting tempfile's cached gettempdir(), so it wrote its stamp to the shared system temp dir and globbed it for *.tmp. Under pytest-xdist another worker's atomic write from witan/context.py can leave a *.tmp there for an instant and fail the assertion. Use the tmp_state_dir fixture, which resets the cache, so the test checks only its own tmp_path.
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The dependency, execution, and test-isolation changes are consistent, with no unresolved blocking issues.
No open issues 🎉
What changed in this PR
Speeds up Witan’s test suite by enabling parallel execution locally and in CI without removing coverage.
Changes:
- Adds pytest-xdist and enables automatic workers with work stealing.
- Documents
-n 0for serial local runs. - Isolates a maintenance test’s temporary files to avoid parallel-run interference.
| File | Description |
|---|---|
uv.lock |
Locks pytest-xdist and execnet dependencies. |
mcp/servers/witan/tests/test_maintenance.py |
Uses the isolated temporary-state fixture. |
mcp/servers/witan/pyproject.toml |
Adds pytest-xdist to test dependencies. |
justfile |
Enables parallel Witan tests locally. |
.github/workflows/witan-tests.yml |
Enables parallel tests in CI. |
.github/workflows/publish-witan.yml |
Enables parallel tests before publishing. |
Review effort: Balanced
blarghmatey
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
N/A
Description (What does it do?)
The witan tests take ~10 minutes per PR. This cuts that time without removing any test coverage, by running the suite in parallel with pytest-xdist.
witan-tests.yml,publish-witan.ymlandjust test-witan-councilrunpytest -n auto --dist worksteal. Pass-n 0to the recipe to run serially (needed for--pdb).pytest-xdistto witan'stestdependency group.Implementation details
The
Run testsstep took 5.2–7.9 minutes over the last 20 successful runs, nearly all of it spent waiting onomnigraphsubprocesses. Each test already builds its own store undertmp_path, andtestsupport/hermetic.pygives each process its own fake home directory, so the tests spread across workers as they are.workstealbecause single tests range from milliseconds to ~15s.Checked under xdist:
~/.config/witanexited 1 withAGENT_KIT_STRICT_HERMETICITY=1.~/.local/binremoved fromPATH, and norequires_omnigraphtest skips.test_mark_run_atomic_roundtripsetTMPDIRwithout resettingtempfile's cached temp dir, so it searched the shared/tmpfor*.tmpfiles. Other workers' atomic writes inwitan/context.pycreate those briefly. It now uses thetmp_state_dirfixture.uv.lockwas regenerated with uv 0.12, which keeps lockfile revision 3 and changes nothing beyond the two new packages.How can this be tested?
witan (memory + CLI)job: compare its time and its finalpassed/skippedline with a recentmainrun.just test-witan-council. With 4 workers I got 1630 passed, 1 skipped in ~67s, against ~7 minutes serially on the same machine, and the result held over repeated runs and with 32 workers.omnigraph-servertests fail on any machine with an outdatedomnigraph-serverbinary, serially as well. CI doesn't install that binary and skips them.