Skip to content

Commit 974f033

Browse files
committed
Add the cppjit series and run both series in the browser.
The tutorials run only on a machine with a C++ toolchain and a kernel installed, and the C++-from-Python direction (a Python kernel with a C++ compiler inside it) has no series at all. Derivatives of the C++ in such notebooks have to be pre-generated, because no cppjit build carries Clad. Add a cppjit series of five notebooks (call a header from Python, use the derivative Clad writes for it, fit a C++ simulator on that exact gradient, fit the Higgs mass on CMS open data without ROOT, calibrate a tumour twin) and a JupyterLite site on which the C++ series and the cppjit series run with nothing installed: xeus-cpp + cppjit compiled to WebAssembly on the LLVM 23 line, with Clad built into CppInterOp, so clad::gradient works in a C++ cell and through cppjit. The native side runs on a cppjit wheel built with the same Clad plugin, and the notebooks generate their derivatives in the session on both. SciPy and matplotlib load on first import; the wire cost is 49.8 MB for the first notebook and 69.8 MB with SciPy loaded. The kernel and the wheel are release assets verified against digests committed in lite/kernel/SHA256SUMS; build-kernel.yml rebuilds both on a runner and proposes new digests as a pull request. The release kernel is linked against an LLVM 23 built with the ci-workflows llvm-wasm recipe (WebAssembly target only, -Oz, LTO: 23.7 MB on the wire instead of 32.5, cell times unchanged within noise); tools/build-wasm-kernel.sh takes such an install through LLVM_WASM_DIR, and the workflow builds against the conda package until that install is published as a cell. The kernel is built from CppInterOp, cppjit and xeus-cpp main with compiler-research/CppInterOp #1167 and Clad at 90a623d6 (vgvassilev/clad#2453: a failed input's declarations are no longer replayed into CodeGen on clang >= 22, which hung the kernel after any parse error), with the patches in lite/kernel/: #1167's clad cross-build flags and static link interface under emsdk 4.0.9, cppjit as a wasm side module and with Clad on request. These are candidates for upstream. Verified under nektos/act: the cppjit job, the Pages build with the real assets and the kernel-and-wheel build. Two cells of 07_stl_modern_cpp.ipynb ran lambdas that write to std::cout inside top-level statements; clang-repl 21 (the xeus-cpp kernel) crashes in the Itanium mangler on that pattern (compiler-research/xeus-cpp#476) depending on heap layout, which is why the C++ job was green on master in June and dead on every runner since. Both cells now define a named function and call it, with the same outputs.
1 parent 6a7b384 commit 974f033

55 files changed

Lines changed: 4563 additions & 20 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/build-kernel.yml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: Build kernel and wheel
2+
# Rebuild the two binaries the site and the cppjit CI job install from a release of this repository: the browser kernel
3+
# (xeus-cpp + cppjit + CppInterOp with Clad on wasm32, as kernel-overlay-<tag>.tar.gz) and the native cppjit wheel with
4+
# Clad inside. Everything comes from tools/build-wasm-kernel.sh, the locks and the patches in the repository; the runner's
5+
# 16 GB cover binaryen's wasm-opt pass (about 11.5 GB). About an hour. The digests go back as a pull request, because the
6+
# Pages and cppjit jobs verify the assets against lite/kernel/SHA256SUMS in the repository, not against a file beside them.
7+
on:
8+
workflow_dispatch:
9+
inputs:
10+
tag:
11+
description: release tag to create or update (kernel-YYYY-MM-DD)
12+
required: true
13+
default: kernel-2026-11-03
14+
permissions:
15+
contents: write
16+
pull-requests: write
17+
jobs:
18+
build:
19+
runs-on: ubuntu-24.04
20+
timeout-minutes: 300
21+
env:
22+
SP: ${{ github.workspace }}/.kernel-build
23+
steps:
24+
- uses: actions/checkout@v4
25+
- name: micromamba (pinned, digest checked, the same as tools/setup-lite-env.sh)
26+
run: |
27+
mkdir -p "$SP/bin"
28+
curl -sSL -o "$SP/bin/micromamba" https://github.com/mamba-org/micromamba-releases/releases/download/2.9.0-0/micromamba-linux-64
29+
(cd "$SP/bin" && echo "366cd9cd8be14df1ab8ed50352a82111082a36686b2d389fdb79a92c3fafb3e3 micromamba" | sha256sum -c -) && chmod +x "$SP/bin/micromamba"
30+
- name: emsdk 4.0.9
31+
run: |
32+
git clone -q --depth 1 https://github.com/emscripten-core/emsdk.git "$SP/emsdk"
33+
(cd "$SP/emsdk" && ./emsdk install 4.0.9 >/dev/null && ./emsdk activate 4.0.9 >/dev/null)
34+
echo "EMSDK=$SP/emsdk" >> "$GITHUB_ENV"
35+
- name: Environments (wasm32 from the lock, native LLVM 23 tools)
36+
run: bash tools/build-wasm-kernel.sh envs
37+
- name: Sources at the pins, patches applied
38+
run: bash tools/build-wasm-kernel.sh sources
39+
- name: Kernel (CppInterOp with Clad, cppjit, xeus-cpp; linked at -O1)
40+
run: bash tools/build-wasm-kernel.sh build
41+
- name: wasm-opt -Oz (binaryen, about 11.5 GB)
42+
run: bash tools/build-wasm-kernel.sh wasm-opt
43+
- name: Kernel overlay
44+
run: bash tools/build-wasm-kernel.sh overlay "${{ inputs.tag }}"
45+
- name: cppjit wheel with Clad
46+
run: bash tools/build-wasm-kernel.sh wheel
47+
- name: Digests
48+
run: |
49+
mkdir -p out && cp "$SP"/kernel-overlay-*.tar.gz "$SP"/cppjit-*.whl out/
50+
(cd out && sha256sum kernel-overlay-*.tar.gz cppjit-*.whl | tee SHA256SUMS)
51+
- uses: actions/upload-artifact@v4
52+
with:
53+
name: kernel-and-wheel-${{ inputs.tag }}
54+
path: out/
55+
- name: Release assets
56+
if: ${{ !env.ACT }}
57+
env:
58+
GH_TOKEN: ${{ github.token }}
59+
run: |
60+
gh release view "${{ inputs.tag }}" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1 \
61+
|| gh release create "${{ inputs.tag }}" --repo "$GITHUB_REPOSITORY" --title "Browser kernel and cppjit wheel with Clad, ${{ inputs.tag }}" --notes "Built by the Build kernel and wheel workflow from tools/build-wasm-kernel.sh; see lite/kernel/README.md."
62+
gh release upload "${{ inputs.tag }}" --repo "$GITHUB_REPOSITORY" --clobber out/kernel-overlay-*.tar.gz out/cppjit-*.whl out/SHA256SUMS
63+
- name: Pull request with the new digests
64+
if: ${{ !env.ACT }}
65+
env:
66+
GH_TOKEN: ${{ github.token }}
67+
run: |
68+
cp out/SHA256SUMS lite/kernel/SHA256SUMS
69+
sed -i "s/KERNEL_RELEASE: kernel-[0-9-]*/KERNEL_RELEASE: ${{ inputs.tag }}/" .github/workflows/deploy-pages.yml
70+
sed -i "s/gh release download kernel-[0-9-]* /gh release download ${{ inputs.tag }} /" .github/workflows/check-notebooks.yml
71+
git config user.name "github-actions[bot]"; git config user.email "github-actions[bot]@users.noreply.github.com"
72+
git checkout -b "digests/${{ inputs.tag }}"
73+
git commit -am "Kernel and wheel digests for release ${{ inputs.tag }}"
74+
git push -u origin "digests/${{ inputs.tag }}"
75+
gh pr create --repo "$GITHUB_REPOSITORY" --base "${{ github.ref_name }}" --head "digests/${{ inputs.tag }}" \
76+
--title "Kernel and wheel digests for ${{ inputs.tag }}" \
77+
--body "Built by the Build kernel and wheel workflow (run ${{ github.run_id }}); the assets are on release ${{ inputs.tag }}. Merging this makes the Pages and cppjit jobs use them."

‎.github/workflows/check-notebooks.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,3 +122,42 @@ jobs:
122122
name: executed-cuda-notebooks
123123
path: executed/
124124
if-no-files-found: ignore
125+
126+
execute-notebooks-cppjit:
127+
name: cppjit
128+
runs-on: ubuntu-latest
129+
defaults:
130+
run:
131+
shell: bash -el {0}
132+
steps:
133+
- name: Checkout repository
134+
uses: actions/checkout@v4
135+
- name: Set up micromamba environment
136+
uses: mamba-org/setup-micromamba@v2
137+
with:
138+
environment-file: environment.yml
139+
cache-environment: true
140+
- name: List available kernels
141+
run: jupyter kernelspec list
142+
- name: cppjit wheel with Clad from the release (digest checked against lite/kernel/SHA256SUMS)
143+
# Under nektos/act (ACT=true) the wheel is expected in lite/kernel-artifacts already; see lite/README-build.md.
144+
env:
145+
GH_TOKEN: ${{ github.token }}
146+
run: |
147+
mkdir -p lite/kernel-artifacts
148+
if [ -z "$ACT" ]; then
149+
gh release download kernel-2026-11-03 --repo "$GITHUB_REPOSITORY" --dir lite/kernel-artifacts --pattern 'cppjit-*.whl'
150+
fi
151+
(cd lite/kernel-artifacts && sha256sum -c --ignore-missing ../kernel/SHA256SUMS)
152+
$CONDA_PREFIX/bin/pip install lite/kernel-artifacts/cppjit-*.whl
153+
- name: Run cppjit notebook tests (Python kernel, C++ compiled by cppjit)
154+
run: |
155+
mkdir -p executed
156+
$CONDA_PREFIX/bin/pytest tests/test_notebooks.py::CppjitNotebookTests -sv
157+
- name: Upload executed cppjit notebooks as artifact
158+
if: always()
159+
uses: actions/upload-artifact@v4
160+
with:
161+
name: executed-cppjit-notebooks
162+
path: executed/
163+
if-no-files-found: ignore

‎.github/workflows/deploy-pages.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: Pages
2+
# Build the JupyterLite site (the C++ series and the cppjit series running in the browser) and publish it on GitHub Pages.
3+
# The kernel binaries come from a release of this repository (see lite/kernel/README.md); everything else is rebuilt here.
4+
on:
5+
push:
6+
branches: [master]
7+
paths: ['cppjit/**', 'xeus-cpp/**', 'lite/**', 'tools/**', '.github/workflows/deploy-pages.yml']
8+
workflow_dispatch:
9+
permissions:
10+
contents: read
11+
pages: write
12+
id-token: write
13+
concurrency:
14+
group: pages
15+
cancel-in-progress: false
16+
env:
17+
KERNEL_RELEASE: kernel-2026-11-03
18+
jobs:
19+
build:
20+
runs-on: ubuntu-24.04
21+
steps:
22+
- uses: actions/checkout@v4
23+
- uses: actions/setup-python@v5
24+
with:
25+
python-version: '3.12'
26+
- name: Kernel binaries from the release
27+
# Under nektos/act (ACT=true) the assets are expected in lite/kernel-artifacts already; see lite/README-build.md.
28+
if: ${{ !env.ACT }}
29+
env:
30+
GH_TOKEN: ${{ github.token }}
31+
run: |
32+
mkdir -p lite/kernel-artifacts
33+
gh release download "$KERNEL_RELEASE" --repo "$GITHUB_REPOSITORY" --dir lite/kernel-artifacts \
34+
--pattern 'kernel-overlay-*.tar.gz'
35+
- name: Verify the kernel binaries against the digests committed in lite/kernel/SHA256SUMS
36+
run: (cd lite/kernel-artifacts && sha256sum -c --ignore-missing ../kernel/SHA256SUMS) && ls -la lite/kernel-artifacts
37+
- name: Build environment (micromamba, host venv, wasm32 env from the lock, kernel overlay)
38+
run: SP="$RUNNER_TEMP/lite" bash tools/setup-lite-env.sh
39+
- name: Build the site
40+
run: SP="$RUNNER_TEMP/lite" OUT="$PWD/lite/dist" bash tools/build-lite-site.sh
41+
- name: The lazy packages are all there
42+
run: test "$(ls lite/dist/lazy/*.tar.gz | wc -l)" -eq "$(grep -v '^#' lite/env/lazy-remove.txt | wc -w)"
43+
- name: Smoke test in a headless browser (cppjit notebooks 1 and 3, the second one with SciPy, matplotlib and a figure)
44+
run: |
45+
"$RUNNER_TEMP/lite/site-venv/bin/pip" install -q playwright==1.63.0
46+
"$RUNNER_TEMP/lite/site-venv/bin/playwright" install --with-deps chromium
47+
# serve one level up so the site sits under /dist/, a sub-path like the Pages URL, not the domain root
48+
(cd lite && python3 -m http.server 8942 >/dev/null 2>&1 &)
49+
sleep 2
50+
for nb in 1_call_it 3_fit_it; do
51+
python3 - "$nb" <<'PY'
52+
import json, sys; nb = json.load(open(f"lite/content/cppjit/{sys.argv[1]}.ipynb"))
53+
json.dump([''.join(c['source']) for c in nb['cells'] if c['cell_type'] == 'code'], open("cells.json", "w"))
54+
PY
55+
"$RUNNER_TEMP/lite/site-venv/bin/python" tools/lite_run.py --site http://127.0.0.1:8942/dist --cells cells.json --timeout 600 | tee run.log
56+
grep -q '"text": "Traceback' run.log && exit 1 || true
57+
done
58+
- uses: actions/upload-pages-artifact@v3
59+
if: ${{ !env.ACT }}
60+
with:
61+
path: lite/dist
62+
deploy:
63+
needs: build
64+
if: ${{ !github.event.act }}
65+
runs-on: ubuntu-latest
66+
environment:
67+
name: github-pages
68+
url: ${{ steps.deployment.outputs.page_url }}
69+
steps:
70+
- id: deployment
71+
uses: actions/deploy-pages@v4

‎.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
lite/dist/
2+
lite/content/
3+
lite/labext/
4+
lite/lazy-packages/
5+
lite/kernel-artifacts/
6+
lite/.jupyterlite.doit.db
7+
executed/
8+
__pycache__/

‎README.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,29 @@ An introduction to modern C++ through live, interactive examples. Topics include
2020
### OpenMP (`openmp/`)
2121
CPU parallelism using OpenMP pragmas. Covers `#pragma omp parallel`, loop-level parallelism, reductions, and thread synchronization using the OpenMP-enabled `xeus-cpp` kernel. The notebooks guide you step-by-step from foundational concepts like spawning threads and managing shared vs. private data variables, all the way to advanced patterns like task-based parallelism, critical sections, and optimizing performance loops directly in an interactive environment.
2222

23+
### cppjit (`cppjit/`)
24+
The other direction: a Python kernel with a C++ compiler inside it. `cppjit` carries Clang and LLVM, reads a C++ header
25+
the way a compiler does, and lets Python call what it finds. Five notebooks from the SC26 tutorial: call a header from
26+
Python, use the derivative Clad wrote for it, fit a C++ simulator with SciPy on that gradient, fit the Higgs mass on CMS
27+
open data without ROOT, and calibrate a sixty-line tumour twin. one wheel is the whole setup (the release's `cppjit-*.whl`, Clad inside; see `cppjit/README.md`).
28+
2329
### CUDA (`cuda/`)
2430
GPU programming from first principles using CUDA C++. This section explores how to accelerate compute-heavy workloads by utilizing graphics cards. It covers essential GPU architecture concepts, kernel declarations, memory allocation (`cudaMalloc`), host-to-device data transfers, and thread organization across grids and blocks to give you hands-on experience writing raw parallel code optimized for parallel hardware.
2531

32+
## Licences
33+
34+
This repository is Apache-2.0. The cppjit notebooks use [Clad](https://github.com/vgvassilev/clad) (LGPL-3.0) through
35+
the compiler they run on: the browser kernel and the cppjit wheel of this repository's release carry it inside
36+
CppInterOp; nothing of Clad is kept here.
37+
38+
## In the browser
39+
40+
The C++ and cppjit series also run with nothing installed, on GitHub Pages:
41+
https://compiler-research.org/live-cpp-tutorials/ . The kernel is xeus-cpp compiled to WebAssembly (LLVM 23), hosting
42+
Python through cppjit for the cppjit series, with Clad built in: a C++ cell can call `clad::gradient` and a Python cell
43+
can do the same through `cppjit.cppdef`. The first visit downloads the compiler once (about 42 MB), after that
44+
everything runs in the tab. OpenMP and CUDA need the native kernels below. How the site is built: `lite/README-build.md`.
45+
2646
## Installation
2747
To ensure a clean setup, install `live-cpp-tutorials` inside a fresh environment using Micromamba.
2848

0 commit comments

Comments
 (0)