Ci/workflows into scripts and i225 support - #1700
Merged
Merged
Conversation
DawidWesierski4
requested review from
Sakoram,
awilczyns,
moleksy and
soopel
as code owners
August 28, 2026 14:11
DawidWesierski4
changed the title
DawidWesierski4
force-pushed
the
ci/workflows-into-scripts
branch
from
August 28, 2026 14:23
276858b to
d25abd5
Compare
Every workflow held its own inline shell. The same host preparation, the same cache key and the same build step were written more than one time, so a repair had to be made in each copy, and no step could be run on a development host. This change puts one copy of each step in .github/scripts/ci/, and each workflow calls it. Taskfile.yml gives the same tasks a name on a development host, so a step can be run and read before a run is started. What is new: * .github/workflows/pr-gate.yml -- one required check. It waits for the build workflow, then starts the test legs. A cancelled or a skipped check is not a result, so the gate drops those runs before it decides. Without that, a commit that reached the branch through a push and a pull_request event left a cancelled check that the gate read as a failure. * .github/workflows/provision-runner.yml -- repairs a build host from the same apt list that setup_environment.sh holds. A build job installs nothing, so before this no commit could repair a host that missed a package. * .github/workflows/unit_tests.yml -- the unit tier, which needs no NIC. * .github/actions/validate-host and .github/actions/wait-for-workflow -- host state and cross-workflow waiting, with the wait logic in wait.js. What goes away: validation-tests.yml and run_validation_tests.sh, replaced by custom-pytest.yml and the scripts; upstream_sync.yml, which no longer runs. Host and cache rules that a test now depends on: * activate-ice.sh loads the cached ice and iavf modules always. A srcversion match only says the source was the same; it does not say the driver is in a state a test can use. So the VFs go down, irdma and ice are removed, and the cached modules take their place. * configure-host.sh takes the seven MTL GStreamer plugins out of the system plugin directories. GST_PLUGIN_PATH adds a directory; it does not take one away, so a plugin an earlier install left behind was a second plugin of the same name. GST_REGISTRY now names a file of the job. * The host prerequisite check asks python3 for the elftools module, which DPDK's buildtools/meson.build needs. A missing module used to stop `meson setup` after the whole patch series had been applied, with a message that names no package and no host. Test suites, so a job can run on a fleet host: * tests/integration_tests/tests.cpp -- the host octet of each port after the primary one counted up from p_ip[0], the network octet. With the --p_sip that gtest.sh now passes, p_ip[0] is 192, so the redundant port always took .193 and a primary address of .193 stopped the run. The octet now counts up from the primary host octet and stays in 1..254. * noctx/run.sh and run_pf.sh -- the test list was read through a pipeline, so the status was awk's and a crashed enumerator read as zero tests. * The acceptance suite reaches a host as an ordinary account with sudo, not as root: the VFIO group node of each new VF is handed to that account, ldconfig and the ffmpeg links run under sudo, and netsniff-ng captures under sudo and is reaped by argv, because sudo does not pass a signal to its child. * The EBU LIST password goes to upload_pcap.py over standard input. mfd_connect logs every command it runs, and those logs are the artifact of a public run. * A `low_bandwidth` marker names each case that stays under 2 Gbps, so a 2.5 GbE card (i225/i226) can carry a smoke leg. gen_config.py gained --interface_type, because a card with no SR-IOV has no VF, and a single-port card takes the kernel-socket path. .gitignore holds the new build and report artifacts of all four branches of this work, so each one does not have to change the same file. Signed-off-by: Wesierski, Dawid <dawid.wesierski@intel.com>
The scripts under `script/` are what the jobs call. Three changes here let
the workflows of the commit before this one do their work.
build_ebpf_xdp.sh checks the host, and can check only
These dependencies are invisible until something far away breaks:
`libdpdk.pc` names libelf and zlib in `Requires.private`, so a host
without `libelf-dev` fails inside a `pkg-config` call minutes into a
build, with nothing in the log that points at the cause. The script now
collects everything it finds missing and prints the apt line for each one,
rather than stopping at the first.
`--check` runs that check alone and installs nothing. `--check build`
narrows it to what consuming DPDK asks of the host. The two sets are not
the same, and treating them as one stopped the fleet: the build job builds
DPDK, MTL, the plugins, FFmpeg and GStreamer, and never builds xdp-tools,
so when `cap-ng.h` joined one shared list every build on every host that
had not been reprovisioned failed on a header nothing it builds includes.
The caller now says which set it needs.
The build scope holds `libelf.h`, `zlib.h`, `make`, and the python3
`elftools` module. `elftools` is there because DPDK's
`buildtools/meson.build` asks python3 for it and stops `meson setup` with
"python3 is missing modules: elftools" — the stop that ended the build job
of run 33168999034 on mtl-runner-12, after the job had downloaded,
unpacked and patched the whole of DPDK. The message names no package and
no host, so the cause was not readable from the log. mtl-runner-12 was the
one host of the fleet without `python3-pyelftools`.
The full scope adds xdp-tools' own prerequisites — `cap-ng.h`,
`pcap/pcap.h`, m4, clang, llvm-strip, wget, unzip — and the three kernel
config symbols AF_XDP needs, which no package can supply. The kernel check
is skipped where `/boot/config-$(uname -r)` is unreadable, as in a
container.
The install path also picks its library directory by planting a probe
`.pc` file and asking `pkg-config` whether it reads it. Both xdp-tools and
libbpf default `LIBDIR` to `${PREFIX}/lib64`, which Debian's pkg-config
does not search, and the multiarch directory Debian does search does not
exist on the RHEL family, so neither upstream default is portable.
`pkg-config --variable pc_path pkg-config` cannot answer this either: it
is a string in a file the distribution ships, and on Rocky 9 it leaves out
`/usr/local/lib64/pkgconfig`, which pkgconf's compiled-in default does
search. Getting this wrong was silent — `make install` succeeded,
pkg-config then reported the library absent, and MTL configured itself
without AF_XDP. If no directory under the prefix is read at all, the `.pc`
files are linked into one that is, because a `.pc` file records its own
libdir and so answers correctly from wherever it is read.
build_drivers.sh produces a bundle, and honors the prefix
`build-dependencies.sh` had two gates and two producers. A `standard_miss`
loop repeated the `any_miss` gate that the workflow already applies to the
step, and it left ICE out, so an ICE-only miss took a second code path
into `build-ice.sh`. That script did by hand what every other component
does through `MTL_INSTALL_PREFIX`.
So `build_drivers.sh` now honors the prefix as `build_dpdk.sh` does. With
one set, the two `.ko` files go to `<prefix parent>/ice/<kernel
release>/<architecture>` and nothing is installed and nothing is loaded,
because a kernel module cannot go in a prefix and be loaded. ICE then maps
to `SETUP_BUILD_AND_INSTALL_DRIVERS_ICE` and `_IAVF` like every other
component maps to its own flag, so the second gate and the second producer
go, and `build-ice.sh` goes with them.
The bundle is `ice.ko` and `iavf.ko` and nothing else, like every other
stash. The 19-field `metadata.env` it used to carry, and the cache key
built from a hash of every kernel header plus a producer-compiler
contract, decided nothing: a bundle that reaches a host is already keyed
on the patch hash, the kernel release and the architecture, and the rest
is answered by the files themselves. Validation reads `vermagic` and the
`ice_vc_cfg_q_bw` symbol — the kernel it loads into, and that it is the
Kahawai build. Activation compares `srcversion`. Dropping the header ABI
and compiler hashes from the key also stops a bundle built for the fleet
kernel from missing on a fleet host whose kernel build tree differs in a
file that has no bearing on the module.
The kernel header check moves here with the work, so a build stops with
the package to install instead of with a missing make target deep in the
driver tree.
`build_drivers.sh` can also build the out-of-tree IAVF module, from
`--iavf-version` / `IAVF_VER` and `patches/iavf_drv/<version>/`. The
version pin and that patch folder come with the driver pull request; until
it lands, only the ICE half of the bundle can be produced.
hash_sources.sh keys JPEG XS and ICE
The waterfall gains two entries. `jpegxs` hashes its own paths over the
`mtl` checksum, and `ffmpeg` now hangs off `jpegxs` rather than off `mtl`,
because the FFmpeg plugin consumes the JPEG XS bundle. `ice` hashes its
own paths and nothing else, because the module does not depend on DPDK or
on MTL. `script/hash_sources_ice.env` and
`script/hash_sources_jpegxs.env` name the paths, one file per component
like the five that were already there.
The ICE key covers `patches/ice_drv/` and `patches/iavf_drv/` together,
because both modules leave in one cached package, so a change to either
one's patches must rebuild it.
nicctl.sh
The usage text goes to standard error and the exit code becomes 2, so a
job that calls the script with the wrong arguments fails instead of
reading an empty success. `create_dcf_vf` goes: no test, no job and no
document names it.
`doc/xdp.md` records `--check`, `task ebpf:check`, the libpcap package
that xdp-tools' configure calls required, and how the install directory is
chosen.
Signed-off-by: Wesierski, Dawid <dawid.wesierski@intel.com>
awilczyns
force-pushed
the
ci/workflows-into-scripts
branch
from
August 31, 2026 14:05
d25abd5 to
4e922b7
Compare
IAVF 4.13.35 is not an independent version pin: it is the VF half of the ICE 2.6.7 virtchnl MAP_QUEUE_VECTOR size fix, which has to be applied to the PF and the VF driver together or VF traffic stalls on opcode 111. The matching patch pair lives in the DPDK 26.07 / ICE 2.6.7 change, so the build flow that consumes it belongs there too, not here. Split across two changes, neither half worked. This one carried build_iavf() and the bundle, cache-key and activation plumbing while versions.env had no IAVF_VER, so build_drivers.sh aborted under set -u on an unbound variable before it built anything. The other carries the version keys and both virtchnl patches with no code that builds them. So drop IAVF here and leave it whole in the change that owns the patches. The ICE bundle becomes ice.ko alone, and this change goes back to being a CI refactor with no driver-version content of its own. The DPDK iavf PMD references stay: iavf_tm_node_add and vf->qos_cap are the stock-driver segfault symptom, unrelated to the out-of-tree module. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>
The example is an exhaustive struct literal, so it stops compiling every time the header gains a field. port_packet_loss arrived in 29d416e and MTL_DMA_DEV_MAX went from 8 to 32 in 101dd49, and neither reached here, so the example has not built since. cargo skips examples and clippy runs without --all-targets, so nothing in CI noticed. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>
awilczyns
force-pushed
the
ci/workflows-into-scripts
branch
from
September 1, 2026 12:01
a32e489 to
db53016
Compare
Sakoram
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.