Skip to content

Speed up CI tests and use CPU-only PyTorch in pip CI - #580

Merged
RussTedrake merged 9 commits into
masterfrom
codex/ci-download-caches
Sep 8, 2026
Merged

Speed up CI tests and use CPU-only PyTorch in pip CI#580
RussTedrake merged 9 commits into
masterfrom
codex/ci-download-caches

Conversation

@RussTedrake

@RussTedrake RussTedrake commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Letter font and dimension tests repeatedly ran expensive convex decomposition, while model conversion tested every Menagerie scene on every CI run. Focus these tests on the behavior each needs to exercise and add timing reports to measure the result.

  • Font and dimension tests use bounding-box collision geometry. Dimension cases now assert actual mesh height and depth, and use separate output directories. Dedicated CoACD coverage remains for I, A, C, and D; the collision-mode test has an explicit depth and correct subtest labels.
  • Menagerie CI converts Panda and ANYmal B in temporary directories, checks exported mesh/material/texture references, then loads and finalizes the converted models with Drake. These cover STL meshes, includes/defaults, and file-backed textures/materials. The shared package cache is not modified.
  • TEST_ALL_MENAGERIE=1 retains the full conversion sweep for Menagerie revision updates. Its smoke-test contract remains conversion success; stronger asset and parser checks apply to the two representative scenes. Some other upstream OBJ files already contain dangling material references.
  • All three CI test jobs print the 30 slowest pytest phases and upload JUnit reports. Tests remain serial. Dependency installation and model prefetch are unchanged.

Drake's Menagerie test covers raw MJCF parsing (including expected STL failures), not this repository's conversion pipeline. Local synthetic converter tests remain in place.

Validation and measurements

  • Local default suite: 13 tests passed in 11.73s; representative Menagerie conversion/loading 5.19s, font test 0.01s, dimension test 0.03s.
  • The final strengthened asset checks also passed in the opt-in full sweep: 13 tests and 48 subtests passed in 64.24s.
  • Black, isort, and whitespace checks passed.
  • Final CI passed all jobs. Before/after timings compare against the preceding instrumentation-only revision:
Runner Letter tests before → after Menagerie before → after Total job before → after
macOS 55.7s → 5.8s 56.9s → 7.7s 9m42s → 7m48s
Ubuntu Poetry 41.9s → 4.2s 80.6s → 7.3s 11m23s → 9m16s

The targeted tests saved about 99s on macOS and 111s on Ubuntu. Other setup/notebook timings still vary across runners. The unchanged pip test job passed in 9m04s, and lint passed in 1m09s. JUnit reports from both Poetry runners were downloaded to verify these measurements.

Download caches were also evaluated during this PR and removed. Linux restore + installation was 13–25s slower; macOS showed no repeatable gain. The unused experimental caches were deleted. The final change adds no caching machinery.

CI-only CPU wheels

The Linux pip job now selects torch/torchvision from PyTorch's CPU wheel index and retains those exact builds using a temporary constraints file during the published manipulation[all] install. Import checks verify both CPU build tags and the absence of a CUDA build. This does not modify pyproject.toml, poetry.lock, or published dependency metadata; downstream GPU installations remain supported. Poetry CI is unchanged.

CPU-wheel CI passed all jobs. The pip job installed torch 2.14.0+cpu and torchvision 0.29.0+cpu, retained those builds through the manipulation install, and passed all 72 tests with no skips.

Pip CI Before CPU wheels
Dependency installation 3m46s 1m55s
Tests 3m47s 3m39s
Total job 9m04s 7m05s

The unchanged Ubuntu Poetry and macOS jobs passed in 8m42s and 8m13s respectively. Workflow YAML/shell syntax checks passed locally. No changes were made to the package manifest or lockfile.

Poetry experiment result

A temporary CPU override for Linux Poetry was also evaluated. It preserved locked versions and original metadata, but full setup took 2m08s versus 2m17s in the prior run: only a nine-second difference. Given runner variability, added re-locking machinery, and macOS remaining above eight minutes, the override was removed. That experiment was superseded after successful setup. The final tree is byte-for-byte identical to the previously passing CPU-pip-only revision 210d981.


This change is Reviewable

Ubuntu prerequisite cleanup

Ubuntu Poetry CI now skips system Jupyter packages (INSTALL_JUPYTER=0) because its Python environment supplies the notebook dependencies. Local prerequisite installation and pip CI keep their existing default; the pip notebook harness relies on system nbformat/nbconvert. Both Ubuntu jobs use the distribution Git package, retain apt lists until dependency setup completes, and avoid a redundant apt update in the prerequisite script. This preserves clean-container installation coverage while reducing duplicate setup work.

Validation: shell syntax and commit checks passed; final CI is green on all jobs. Ubuntu Poetry finished in 8m00s versus 9m22s on the preceding baseline. Dependency setup fell from 2m51s to 2m06s (the first attempt also measured 2m05s), prefetch from 51s to 29s, and tests from 5m04s to 4m59s. Faster downloads also contributed, so the full 82-second improvement is not attributed solely to the code change. Final pip CI: 6m55s; macOS: 7m40s.

@RussTedrake RussTedrake changed the title Cache CI dependency downloads and record test timings Record CI test timings and cache macOS dependency downloads Sep 7, 2026
@RussTedrake RussTedrake changed the title Record CI test timings and cache macOS dependency downloads Record CI test timings and JUnit reports Sep 7, 2026
@RussTedrake RussTedrake changed the title Record CI test timings and JUnit reports Speed up letter and Menagerie tests and report CI timings Sep 8, 2026
@RussTedrake RussTedrake changed the title Speed up letter and Menagerie tests and report CI timings Speed up CI tests and use CPU-only PyTorch in pip CI Sep 8, 2026

@RussTedrake RussTedrake left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

@RussTedrake reviewed 5 files and all commit messages, and made 1 comment.
Reviewable status: 0 of 5 files reviewed, all discussions resolved.

@RussTedrake
RussTedrake merged commit f46c1c1 into master Sep 8, 2026
5 of 6 checks passed
@RussTedrake
RussTedrake deleted the codex/ci-download-caches branch September 8, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant