From d01ae00c9d0b043ad19665e378b18036deb014d7 Mon Sep 17 00:00:00 2001 From: Zeeshan Lakhani Date: Tue, 11 Aug 2026 03:40:02 +0000 Subject: [PATCH 1/5] [multicast] Host-side plumbing for externally sourced multicast This work gets externally sourced multicast into a running rack. voxel can now provide plumbing for: - a route for each group pointed at `ce` - a mirror on the transit router carrying each group's flood toward every scrimlet - a membership so that the router's NIC accepts the group's frames to begin with This work adds all of these steps via `voxel network multicast {up,down,check}` commands. The mirror rests on two main properties: 1. external multicast ingresses at whichever switch holds the group's external NAT entry, elected by the rack and not visible to the host (so that every scrimlet has to receive a copy). 2. flower classification ends at the first matching filter, so the targets are chained `action mirred` clauses inside one filter per group rather than one filter per scrimlet. The mirror also only sees frames the NIC accepts, and a NIC accepts a multicast group only after a group join (subscription). Nothing on the router joins these groups (voxel's FRR runs only unicast BGP) on its own, so the `up` command pins a static link-layer membership for each group's Ethernet address (the RFC 1112 section 6.4 mapping) on the host-facing NIC. This mapping folds 28 group bits into 23 (RFC 7042 section 2.1.1), meaning that 32 groups alias each Ethernet address and teardown drops a membership only when no remaining group still maps to it. Each membership `up` adds is recorded under `/run` on the router, and `down` removes only recorded ones, so a membership the router already held is never voxel's to delete. The record dies with the router, exactly as the memberships do. Host routes are the one piece that outlives `voxel destroy`, and the host's route table is shared with every other Falcon environment. So `up` records what it installs in `.falcon/multicast-.json`, and `down` and `check` read that record instead of scanning. The record, not the nexthop, is what makes a route owned by this specific environment: in isolated mode `ce`'s address derives from `[network]` rather than from the environment name, so two environments would share a nexthop and it proves nothing. That record is written ahead of the route, so a crash between the two leaves a record with no route rather than a route nothing can prove is ours, and both directions fail closed: an unrecorded route is neither swept by `up` nor removed by `down`. Teardown also has to work against a rack that has already gone away. `ssh_try_capture` separates an unreachable node (ssh exit 255) from a command that ran and failed, so `down` clears the host routes and the record when `cr1` is unreachable while still reporting a genuine failure on a live router. The `check` command also prints each group's control-plane mapping onto the underlay, read from `swadm multicast list` in every switch zone, naming the NAT target an external group replicates onto. This work also repins sidecar-lite and omicron to the leaves of their respective multicast branches, `zl/multicast` and `zl/mcast-build`. Most importantly, we add `docs/multicast.md`, covering the API side (pools, groups, members, probes), this plumbing, and the static production equivalents of each piece. Also here: - `voxel commtest --traffic multicast` refuses to start when the setup is missing, with `--setup-mcast` to plumb it in place. It also defaults `--icmp-loss-tolerance` to 500, the value omicron's a4x2 CI passes, and rejects an inverted `--ip-pool-begin`/`--ip-pool-end` pair up front. - `[falcon].ssh_pubkey` stages a public key into every node, so `ssh root@` authenticates by key rather than by the images' empty root password. - `voxel status` reports each sled's rpool usage and `zpool status -x` health. - Format arguments are inlined across the workspace, and `commtest`'s `mod test` is renamed to `mod tests`. --- Cargo.lock | 91 +- Cargo.toml | 9 +- README.md | 225 ++- docs/multicast.md | 898 +++++++++++ docs/parameters.md | 3 +- voxel-config/src/config.rs | 52 +- voxel-image/README.md | 2 +- voxel-init/src/gimlet.rs | 49 +- voxel-init/src/router.rs | 24 +- voxel-init/src/sys.rs | 53 + voxel/Cargo.toml | 2 + voxel/src/access.rs | 5 +- voxel/src/commission.rs | 5 +- voxel/src/commtest.rs | 344 ++++- voxel/src/config_cmd.rs | 13 +- voxel/src/cpbuild.rs | 42 +- voxel/src/image.rs | 16 +- voxel/src/main.rs | 130 +- voxel/src/multicast.rs | 1913 ++++++++++++++++++++++++ voxel/src/net.rs | 82 +- voxel/src/network.rs | 2 +- voxel/src/patch.rs | 7 +- voxel/src/rack.rs | 81 +- voxel/src/rss_request.rs | 15 +- voxel/src/sp_cmd.rs | 30 +- voxel/src/testdata/tc-filter-show.json | 98 ++ voxel/src/topo.rs | 124 +- voxel/src/wicket_setup.rs | 2 +- 28 files changed, 4031 insertions(+), 286 deletions(-) create mode 100644 docs/multicast.md create mode 100644 voxel/src/multicast.rs create mode 100644 voxel/src/testdata/tc-filter-show.json diff --git a/Cargo.lock b/Cargo.lock index 39f1543..d302041 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -122,7 +122,7 @@ checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" [[package]] name = "api_identity" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "omicron-workspace-hack", "proc-macro2", @@ -409,7 +409,7 @@ dependencies = [ [[package]] name = "bootstore" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "bytes", "camino", @@ -438,7 +438,7 @@ dependencies = [ [[package]] name = "bootstrap-agent-lockstep-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "anyhow", "chrono", @@ -453,6 +453,8 @@ dependencies = [ "sled-hardware-types", "slog", "strum 0.27.2", + "thiserror 2.0.18", + "trust-quorum-types", "wicketd-commission-types", ] @@ -1549,7 +1551,7 @@ dependencies = [ [[package]] name = "gateway-types-versions" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "daft", "dropshot", @@ -1617,7 +1619,7 @@ dependencies = [ [[package]] name = "gfss" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "digest 0.10.7", "omicron-workspace-hack", @@ -2764,7 +2766,7 @@ dependencies = [ [[package]] name = "omicron-common" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "anyhow", "api_identity", @@ -2786,6 +2788,7 @@ dependencies = [ "omicron-ledger", "omicron-uuid-kinds", "omicron-workspace-hack", + "oxide-generation", "oxnet", "parse-display", "progenitor-client 0.14.0", @@ -2810,7 +2813,7 @@ dependencies = [ [[package]] name = "omicron-ledger" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "async-trait", "atomicwrites", @@ -2827,7 +2830,7 @@ dependencies = [ [[package]] name = "omicron-passwords" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "argon2", "omicron-workspace-hack", @@ -2842,7 +2845,7 @@ dependencies = [ [[package]] name = "omicron-uuid-kinds" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "daft", "newtype-uuid", @@ -2898,6 +2901,19 @@ version = "3.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c1b04fb49957986fdce4d6ee7a65027d55d4b6d2265e5848bbb507b58ccfdb6f" +[[package]] +name = "oxide-generation" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e130a1add81f976d13f8f3ab455de3e3fd1f1b9b796520049680dd19a385809c" +dependencies = [ + "daft", + "schemars 0.8.22", + "serde", + "serde_json", + "slog", +] + [[package]] name = "oxnet" version = "0.1.6" @@ -3390,10 +3406,10 @@ dependencies = [ [[package]] name = "propolis-api-types-versions" version = "0.0.0" -source = "git+https://github.com/oxidecomputer/propolis?rev=046f74302e2ea09a75b0a6810645d42c7df6644a#046f74302e2ea09a75b0a6810645d42c7df6644a" +source = "git+https://github.com/oxidecomputer/propolis?rev=3c07d60a77d528018fded2f0a434cafe724f3423#3c07d60a77d528018fded2f0a434cafe724f3423" dependencies = [ "crucible-client-types 0.1.0 (git+https://github.com/oxidecomputer/crucible?rev=ad8a31742adc45e925e63443a5b43c8e30604022)", - "propolis_types 0.0.0 (git+https://github.com/oxidecomputer/propolis?rev=046f74302e2ea09a75b0a6810645d42c7df6644a)", + "propolis_types 0.0.0 (git+https://github.com/oxidecomputer/propolis?rev=3c07d60a77d528018fded2f0a434cafe724f3423)", "schemars 0.8.22", "serde", "thiserror 1.0.69", @@ -3424,15 +3440,6 @@ dependencies = [ "uuid", ] -[[package]] -name = "propolis_api_types" -version = "0.0.0" -source = "git+https://github.com/oxidecomputer/propolis?rev=046f74302e2ea09a75b0a6810645d42c7df6644a#046f74302e2ea09a75b0a6810645d42c7df6644a" -dependencies = [ - "crucible-client-types 0.1.0 (git+https://github.com/oxidecomputer/crucible?rev=ad8a31742adc45e925e63443a5b43c8e30604022)", - "propolis-api-types-versions", -] - [[package]] name = "propolis_api_types" version = "0.0.0" @@ -3446,10 +3453,19 @@ dependencies = [ "uuid", ] +[[package]] +name = "propolis_api_types" +version = "0.0.0" +source = "git+https://github.com/oxidecomputer/propolis?rev=3c07d60a77d528018fded2f0a434cafe724f3423#3c07d60a77d528018fded2f0a434cafe724f3423" +dependencies = [ + "crucible-client-types 0.1.0 (git+https://github.com/oxidecomputer/crucible?rev=ad8a31742adc45e925e63443a5b43c8e30604022)", + "propolis-api-types-versions", +] + [[package]] name = "propolis_types" version = "0.0.0" -source = "git+https://github.com/oxidecomputer/propolis?rev=046f74302e2ea09a75b0a6810645d42c7df6644a#046f74302e2ea09a75b0a6810645d42c7df6644a" +source = "git+https://github.com/oxidecomputer/propolis?rev=36f20be9bb4c3b362029237f5feb6377c982395f#36f20be9bb4c3b362029237f5feb6377c982395f" dependencies = [ "schemars 0.8.22", "serde", @@ -3458,7 +3474,7 @@ dependencies = [ [[package]] name = "propolis_types" version = "0.0.0" -source = "git+https://github.com/oxidecomputer/propolis?rev=36f20be9bb4c3b362029237f5feb6377c982395f#36f20be9bb4c3b362029237f5feb6377c982395f" +source = "git+https://github.com/oxidecomputer/propolis?rev=3c07d60a77d528018fded2f0a434cafe724f3423#3c07d60a77d528018fded2f0a434cafe724f3423" dependencies = [ "schemars 0.8.22", "serde", @@ -3570,7 +3586,7 @@ checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" [[package]] name = "rack-init-config" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "bootstrap-agent-lockstep-types", "iddqd", @@ -4538,7 +4554,7 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "sled-agent-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "anyhow", "async-trait", @@ -4568,7 +4584,7 @@ dependencies = [ [[package]] name = "sled-agent-types-versions" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "anyhow", "async-trait", @@ -4588,7 +4604,7 @@ dependencies = [ "omicron-workspace-hack", "oxnet", "propolis-api-types-versions", - "propolis_api_types 0.0.0 (git+https://github.com/oxidecomputer/propolis?rev=046f74302e2ea09a75b0a6810645d42c7df6644a)", + "propolis_api_types 0.0.0 (git+https://github.com/oxidecomputer/propolis?rev=3c07d60a77d528018fded2f0a434cafe724f3423)", "schemars 0.8.22", "serde", "serde_json", @@ -4607,7 +4623,7 @@ dependencies = [ [[package]] name = "sled-hardware-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "daft", "omicron-workspace-hack", @@ -5502,10 +5518,19 @@ dependencies = [ "once_cell", ] +[[package]] +name = "trust-quorum-types" +version = "0.1.0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +dependencies = [ + "omicron-workspace-hack", + "trust-quorum-types-versions", +] + [[package]] name = "trust-quorum-types-versions" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "byte-wrapper", "daft", @@ -5860,11 +5885,13 @@ dependencies = [ "clap", "futures", "indoc", + "itertools 0.14.0", "libc", "libfalcon", "oxnet", "rack-init-config", "reqwest 0.13.4", + "serde", "serde_json", "slog", "sprockets-tls-test-utils", @@ -6091,7 +6118,7 @@ dependencies = [ [[package]] name = "wicketd-commission-client" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "iddqd", "omicron-uuid-kinds", @@ -6109,7 +6136,7 @@ dependencies = [ [[package]] name = "wicketd-commission-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "omicron-workspace-hack", "wicketd-commission-types-versions", @@ -6118,7 +6145,7 @@ dependencies = [ [[package]] name = "wicketd-commission-types-versions" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=7950023e5971625c2f5fe7c2d808558bfc2653c0#7950023e5971625c2f5fe7c2d808558bfc2653c0" +source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" dependencies = [ "gateway-types-versions", "iddqd", @@ -6134,6 +6161,7 @@ dependencies = [ "slog-error-chain 0.1.0 (git+https://github.com/oxidecomputer/slog-error-chain?branch=main)", "thiserror 2.0.18", "uuid", + "zeroize", ] [[package]] @@ -6523,6 +6551,7 @@ version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" dependencies = [ + "serde", "zeroize_derive", ] diff --git a/Cargo.toml b/Cargo.toml index 63b22fe..8091f13 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -10,10 +10,10 @@ members = [ [workspace.dependencies] anyhow = "1.0.102" # The omicron commit voxel is pinned to. Bumping this rev is the new-omicron -# motion; build.rs surfaces it to voxel image create. Keep both revs identical. -rack-init-config = { git = "https://github.com/oxidecomputer/omicron", rev = "7950023e5971625c2f5fe7c2d808558bfc2653c0" } -wicketd-commission-types-versions = { git = "https://github.com/oxidecomputer/omicron", rev = "7950023e5971625c2f5fe7c2d808558bfc2653c0" } -wicketd-commission-client = { git = "https://github.com/oxidecomputer/omicron", rev = "7950023e5971625c2f5fe7c2d808558bfc2653c0" } +# motion; build.rs surfaces it to voxel image create. Keep all revs identical. +rack-init-config = { git = "https://github.com/oxidecomputer/omicron", rev = "521e1b3903a3e4a1652a15395e0922723fc90cdc" } +wicketd-commission-types-versions = { git = "https://github.com/oxidecomputer/omicron", rev = "521e1b3903a3e4a1652a15395e0922723fc90cdc" } +wicketd-commission-client = { git = "https://github.com/oxidecomputer/omicron", rev = "521e1b3903a3e4a1652a15395e0922723fc90cdc" } clap = { version = "4.6.1", features = ["derive", "env"] } expectorate = "1.2.0" # Our crates use only #[tokio::main] + tokio::time; libfalcon pulls its own @@ -26,6 +26,7 @@ slog = "2.7" serde = "1" camino = { version = "1.2.2", features = ["serde1"] } indoc = "2.0.7" +itertools = "0.14.0" toml_edit = "0.25" # sprockets/trust quorum related deps diff --git a/README.md b/README.md index eb6b518..12971be 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # voxel -**V**irtual **OX**ide **E**mulation **L**ab. A tool for standing up emulated Oxide -rack deployments on a single Helios host. +**V**irtual **OX**ide **E**mulation **L**ab. A tool for standing up emulated +Oxide rack deployments on a single Helios host. -Voxel emulates an Oxide rack's control plane; -[omicron] software on [falcon]-managed propolis VMs, with +Voxel emulates an Oxide rack's control plane: +[Omicron] software on [falcon]-managed propolis VMs, with SoftNPU switches and FRR routers. Pick a platform version and a topology (sled count, multi-rack, BGP/static) and launch. It succeeds the `a4x2` testbed topology, reworked around a first-class CLI and on-the-fly config @@ -13,14 +13,16 @@ generation. ## Layout - **`voxel/`**: CLI and launcher -- **`voxel-config/`**: the `VoxelConfig` model (`voxel.toml`) and all per-topology - config generation (sled-agent, RSS, FRR, MGS/SP-sim). -- **`voxel-init/`**: the in-guest bring-up agent baked into the images (gimlet/router - roles). -- **`voxel-image/`**: image build machinery (`voxel image create`) and the install - scripts that bake a control-plane image from an omicron commit. +- **`voxel-config/`**: the `VoxelConfig` model (`voxel.toml`) and all + per-topology config generation (sled-agent, RSS, FRR, MGS/SP-sim). +- **`voxel-init/`**: the in-guest bring-up agent baked into the images + (gimlet/router roles). +- **`voxel-image/`**: image build machinery (`voxel image create`) and the + install scripts that bake a control-plane image from an omicron commit. -See [`docs/parameters.md`](docs/parameters.md) for the `voxel.toml` reference, there are a LOT of tuning knobs. +See [parameters] for the `voxel.toml` reference and its many tuning knobs. +See [multicast] for the multicast API (pools, groups, members, probes, omdb) +and the host plumbing that carries externally sourced multicast into a rack. ## Building @@ -29,33 +31,45 @@ cargo build ``` `voxel` links omicron's own RSS config types (the `rack-init-config` crate in -omicron, pinned to a commit), so `config-rss.toml` is rendered in-process and schema -drift surfaces at voxel compile time. +omicron, pinned to a commit), so `config-rss.toml` is rendered in-process and +schema drift surfaces at voxel compile time. ## Quickstart 1. `cargo build` builds voxel. -2. `voxel image create 43bb5af` builds omicron (v21) and bakes `voxel-cp-43bb5af` - (30-45 min). -3. `voxel image create-frr proto` bakes `voxel-frr-proto` (omicron-independent; - build once, reuse for any commit). +2. `pfexec voxel image create` builds the workspace's pinned omicron commit and + bakes `voxel-cp-` (30-45 min). Pass a commit to build another version, + e.g. `pfexec voxel image create 43bb5af`. Image builds boot a builder VM, + so they need `pfexec` (see [Privileges](#privileges)). +3. `pfexec voxel image create-frr proto` bakes `voxel-frr-proto` + (omicron-independent; build once, reuse for any commit). 4. Configure: ``` -voxel config set image.cp voxel-cp-43bb5af voxel config set image.frr voxel-frr-proto ``` + An unset `image.cp` follows the workspace pin, the image a commitless + `voxel image create` bakes, so a repin needs no config edit. Set it only + to select a different image: `voxel config set image.cp voxel-cp-43bb5af`. + 5. `pfexec voxel launch` -A few notes: by default, this will all happen under $HOME. If you don't like that or need -to improve performance by using a separate disk, there are some knobs set via `voxel config set`: +A few notes: by default, this will all happen under $HOME. If you don't like +that or need to improve performance by using a separate disk, there are some +knobs set via `voxel config set`: * falcon.dataset: Location for built control plane snapshots, exported as `FALCON_DATASET`, with images and topo zvols under `/img/...` -* falcon.build_root: Location where omicron will clone and compile for new images, - exported as `BUILD_ROOT`, holding the omicron checkout -* falcon.workdir: Location where voxel will do its configuration and setup for new launches +* falcon.build_root: Location where omicron will clone and compile for new + images, exported as `BUILD_ROOT`, holding the omicron checkout +* falcon.workdir: Location where voxel will do its configuration and setup for + new launches +* falcon.ssh_pubkey: SSH public key staged into every node, so `ssh root@` + authenticates by key rather than the images' empty root password. Defaults to + the first of `~/.ssh/id_ed25519.pub`, `id_ecdsa.pub`, `id_rsa.pub`; set it + when the key has a non-standard name: + `voxel config set falcon.ssh_pubkey ~/.ssh/github_ed25519.pub` ## Privileges @@ -63,25 +77,35 @@ Voxel commands need different privileges: - `voxel launch`, `voxel destroy`, and image builds run under `pfexec`. They manage zfs datasets, data links, and zones, so they need full root. -- `voxel network external ...` runs unprivileged. Voxel escalates each - mutating host command through `pfexec` itself, and `--dry-run` prints those - as `+ pfexec ...` lines. +- `voxel network external ...` and `voxel network multicast ...` run + unprivileged. Voxel escalates each mutating host command through `pfexec` + itself, and `--dry-run` prints those as `+ pfexec ...` lines (plus + `+ ssh root@...` for the commands multicast runs on a router). - `voxel commtest` runs as your login user, with the `net_icmpaccess` privilege described below. It refuses effective uid 0 so a root run cannot leave root-owned files in the build worktrees and reports. `--allow-root` overrides that where a per-user grant is impractical, at the cost of root-owned artifacts under the build root. -## omicron commtest +Host-side multicast plumbing brackets the rack's lifetime rather than just +sharing it. Run `voxel network multicast up` after `voxel launch`, since it +reaches the running `ce` and `cr1`, and run it again after every launch +because the mirror and the memberships live inside `cr1`. **Run +`voxel network multicast down`** before `voxel destroy`. "Destroy" leaves that +state alone because router-state cleanup requires the explicit router target, +as the per-environment host-route record remains available for the later +`down`. + +## Omicron commtest -`voxel commtest` builds and runs omicron's `commtest` binary against a launched -rack. The source is the omicron checkout matching the configured control-plane +`voxel commtest` builds and runs Omicron's `commtest` binary against a launched +rack. The source is the Omicron checkout matching the configured control-plane image, an explicit commit or tag, or the latest upstream `main`. Voxel derives the selected rack's Nexus API address and takes a test IP pool from the range directly above the configured service pool. ```sh -# Configured image's omicron commit (unicast is the default). +# Configured image's Omicron commit (unicast is the default). voxel commtest # A specific commit (older unicast-only versions are supported). @@ -104,22 +128,31 @@ voxel commtest 43bb5af -- cleanup `--traffic` accepts `unicast`/`uni`, `multicast`/`multi`, or `both`. Voxel detects whether the selected commit supports multicast and refuses the -multicast modes on older, unicast-only versions. `--api URL` overrides the -derived Nexus API address, and `--no-build` runs an existing -`/target/debug/commtest`. - -Voxel only injects the arguments commtest has no default for, so everything +multicast modes on older, unicast-only versions. The detection inspects the +selected checkout, not the commit baked into the running rack's image. A rack +image that predates probe multicast silently drops a probe's +`multicast_groups` and surfaces later as no-delivery, so be sure to keep the +two commits aligned. `--api URL` overrides the derived Nexus API address, and +`--no-build` runs an existing `/target/debug/commtest`. + +Voxel injects the arguments commtest has no usable default for, so everything after `--` reaches it unchanged: -- `--ip-pool-begin` and `--ip-pool-end` override the derived pool. Pass both. - Passing one alone is refused, because the half voxel derives would overlap - the service pool. +- `--ip-pool-begin` and `--ip-pool-end` override the derived pool. Pass both, + as voxel rejects a lone bound: pairing a caller's address with one derived + from `[network]` yields a range that either overlaps the service pool or + becomes inverted. - `--mcast-group` (repeatable, `GROUP[@SRC,...]`) replaces voxel's default group of `239.1.1.1`. `--mcast-deny-group` on its own, the source-filter negative test, also runs the multicast phase, so voxel adds no default group when it is present. -- `--test-duration`, `--warmup`, `--packet-rate`, and `--icmp-loss-tolerance` - keep commtest's defaults of `100s`, `0s`, `10`, and `0`. +- `--icmp-loss-tolerance` overrides voxel's default of `500`, the value + omicron's a4x2 CI uses. Commtest's own default of `0` suits real hardware, + but a virtual rack shares one host across every sled VM and sheds a few + packets at the virtio rings under burst. Pass `--icmp-loss-tolerance 0` to + restore the strict threshold. +- `--test-duration`, `--warmup`, and `--packet-rate` keep commtest's defaults + of `100s`, `0s`, and `10`. - `--api-timeout` (default `60m`) is a top-level argument, so it goes before the `run` subcommand. @@ -134,7 +167,7 @@ pfexec usermod -K defaultpriv=basic,net_icmpaccess "$USER" Start a new login session afterward and confirm that `ppriv $$` lists `net_icmpaccess` in the effective set. -Voxel keeps its omicron mirror under `$BUILD_ROOT/commtest` (or +Voxel keeps its Omicron mirror under `$BUILD_ROOT/commtest` (or `~/voxel-builds/commtest`) and checks each commit out into a detached [Git worktree][Git worktrees], so the checkouts, Cargo output, and commtest reports stay owned by the invoking user. `--source` builds the given checkout @@ -145,7 +178,7 @@ in place, without fetching or changing its Git state. By default (`[external] mode = "lan"`), every node's external NIC lands on the host's default-route interface (or `$EXT_INTERFACE`) and leases an address from whatever DHCP serves the network that link attaches to. That is option 1 ("an -existing IPv4 network") of omicron's [how-to-run external networking]. +existing IPv4 network") of Omicron's [how-to-run external networking]. On a host without such a network, voxel can instead build the whole external segment itself, option 2 ("an external network that only exists on your test machine") of the same doc, which a4x2 required the user to plumb by hand. @@ -162,15 +195,16 @@ nodes' external NICs correctly), a host VNIC `voxel_ext0` holding the gateway address (`[external].host_ip`, default `172.30.199.199`), and IPv4 forwarding plus an ipnat rule out `uplink`. Node addresses are static because voxel numbers every sled and router -deterministically from `[external].ip_start` (default `172.30.199.10`) and stages -`/` + gateway + DNS into that node's cargo-bay (`external-net`). +deterministically from `[external].ip_start` (default `172.30.199.10`) and +stages `/` + gateway + DNS into that node's cargo-bay +(`external-net`). The in-guest agent (`voxel-init`) applies the staged address on both sleds and routers. No DHCP server runs on the segment. The nodes' addresses stay in use -after bring-up (RSS progress is polled over SSH to them, each router NATs rack -egress out its own external address, and the host route to each rack points at -the customer-edge router, `ce`), which is why the segment must exist before -boot. +after bring-up (the RSS watch polls sleds over SSH at those addresses, each +router NATs rack egress out its own external address, and the host route to +each rack points at the customer-edge router, `ce`), which is why the segment +must exist before boot. Operator commands (the same code paths launch uses): @@ -185,7 +219,7 @@ Notes: the matching rules, so unrelated rules survive. ipv4-forwarding stays enabled, as it is a host-global setting. - Unlike the how-to-run recipe, voxel never persists the NAT rules to - `/etc/ipf/ipnat.conf`: they are loaded at runtime only, so voxel doesn't + `/etc/ipf/ipnat.conf`: the rules live in the kernel only, so voxel doesn't own a shared system file. They don't survive a reboot, and the next `launch` (or `up`) reloads them. - `[external].mtu` must stay below 9000: voxel-init classifies a sled NIC as @@ -210,12 +244,85 @@ Notes: - A manually plumbed fake network (`fake_external0` etc.) can coexist because voxel's link names are distinct, and `$EXT_INTERFACE` always wins. +## Multicast host plumbing (optional) + +This is scaffolding for the emulated environment, not something a rack needs in +production. A real rack sits behind a customer network that already routes +multicast (static multicast routes, or PIM upstream with IGMP toward hosts), so +an externally sourced group reaches the rack's uplinks on its own and the rack +takes it from there. Voxel's customer network is a few FRR boxes carrying +unicast BGP, so the host has to stand in for that upstream. Per [RFD 488], the +rack signals nothing upstream by design in v1: assignment is static and +API-driven, with IGMP host-proxying ([RFC 4605]) proposed atop it. The +[multicast] doc records the production equivalent of each piece below, along +with the TODO for reworking this scaffolding into a listening upstream once +host-proxying lands. + +Standing in for it needs three things the rack cannot arrange itself: + +- a host route pointing the group at the customer-edge router. +- a mirror on the transit router that copies the group's flood from its + host-facing NIC onto every scrimlet-facing NIC. The routers run FRR for + unicast BGP and no multicast routing daemon, so without the mirror a group's + frames stop at `ce`. We use a mirror rather than a daemon because the only + job here is getting frames onto the switch ports. The rack replicates to + members itself, so a router that forwarded properly would duplicate that + work, and the FRR image stays as it is. +- a static link-layer membership for the group's Ethernet address on the + router's host-facing NIC. Nothing on the router joins the group, so without + it, the NIC drops the frames before the mirror ever sees them. This exact + `ip maddress` entry is solely a workaround for voxel's mirror-only Linux + router. In actual environments, customers would have to arrange upstream + delivery toward both rack uplinks, whether via PIM/IGMP, static multicast + routes and joins on the upstream routers, or an equivalent mechanism. + +``` +voxel network multicast up # route + tc/mirred mirror + membership (--dry-run to preview) +voxel network multicast check # one line per item, PASS/FAIL overall +voxel network multicast down # remove the mirror, memberships, and routes +``` + +`up` defaults to `239.1.1.1`, the group `voxel commtest --traffic multicast` +uses; pass `--group` (repeatable) for others. `check` and `down` default to +whatever is currently plumbed for this Falcon environment, discovered from its +`.falcon/` state and voxel's own `tc` filters on the mirror router. A host route +belonging to another environment is left alone. An unreachable router with a +known address narrows that to this environment's host routes alone, with a +warning. In `lan` mode, the address must first be read from the running router +over the falcon console. If that lookup fails, `check` reports an error and +live `down` stops after host-route cleanup rather than treating the router as +gone; a dry-run skips the router preview. `check` closes with each group's +underlay mapping, read from `swadm multicast list` in both switch zones; the +lines are informational and appear once the rack has programmed the group. All +three work in whichever `[external]` mode is set: isolated mode derives the +mirror router's address from config, while `lan` mode reads its DHCP lease over +the falcon console. + +Notes: +- Every scrimlet is a mirror target because external multicast ingresses at + whichever switch holds the group's external NAT entry, which the rack elects + and the host cannot see. The switches without the entry drop their copy, so + there is no duplicate replication. +- The mirror is one `tc` filter per group with the targets chained as `action + mirred` clauses. One filter per scrimlet does not work: the first matching + filter ends flower classification, whereas chained actions all run. +- `voxel commtest --traffic multicast` (and `both`) refuses to start when a + route or `tc` filter is missing, since the symptom is otherwise a receive + timeout minutes into the run. `--setup-mcast` plumbs the missing pieces + for the groups that run uses instead of refusing. +- `check` asserts only the host side, the route, the filter, and the + membership. Proving delivery past the switch needs a member in the group. + The rack side (pools, groups, members, probes) is in [multicast]. Short of + a full `commtest` run, the cheapest confirmation is a probe joined to the + group plus `ping -s ` from the host, which prints a reply line per + responder. + ## Emulated SPs and RoTs (sp-emu, optional) -By default voxel backs each SP with omicron's `sp-sim`. To run real SP and RoT +By default voxel backs each SP with Omicron's `sp-sim`. To run real SP and RoT firmware, voxel uses [sp-emu], which boots unmodified Hubris on emulated -STM32H7 and LPC55 cores. sp-emu is a separate binary -run inside the switch zone, not a Cargo dependency, so build it and point voxel at it. +STM32H7 and LPC55 cores. sp-emu runs as its own binary inside the switch zone +rather than as a Cargo dependency, so build it and point voxel at it. 1. Build sp-emu: @@ -247,13 +354,17 @@ run inside the switch zone, not a Cargo dependency, so build it and point voxel `--wicket-setup` runs rack setup through wicketd instead of the file-based sled-agent auto-init. -When you build a cp image, voxel bakes the sp-emu binary and per-role firmware into -the image from `[sp]`, so a launched rack is self-contained and `emu_bin` can be -left unset at launch. Setting `emu_bin` at launch stages it on the fly instead, -which is useful for iterating on sp-emu without rebaking. +When you build a cp image, voxel bakes the sp-emu binary and per-role firmware +into the image from `[sp]`, so a launched rack is self-contained and `emu_bin` +can be left unset at launch. Setting `emu_bin` at launch stages it on the fly +instead, which is useful for iterating on sp-emu without rebaking. -[omicron]: https://github.com/oxidecomputer/omicron +[multicast]: docs/multicast.md +[parameters]: docs/parameters.md +[Omicron]: https://github.com/oxidecomputer/omicron [falcon]: https://github.com/oxidecomputer/falcon [how-to-run external networking]: https://github.com/oxidecomputer/omicron/blob/main/docs/how-to-run.adoc#external-networking [sp-emu]: https://github.com/oxidecomputer/sp-emu [Git worktrees]: https://git-scm.com/docs/git-worktree +[RFD 488]: https://rfd.shared.oxide.computer/rfd/0488 +[RFC 4605]: https://www.rfc-editor.org/rfc/rfc4605 diff --git a/docs/multicast.md b/docs/multicast.md new file mode 100644 index 0000000..366d738 --- /dev/null +++ b/docs/multicast.md @@ -0,0 +1,898 @@ +# Multicast on a voxel rack + +Three API objects express multicast on an Oxide rack: a multicast IP pool, a +group, and its members. An operator creates only the pool directly. Nexus +implicitly materializes a group when the first member joins an address the pool +covers, and reaps it when the last member leaves. This document is a reference +for those objects as they behave on a voxel rack, then covers the host-side +topology management that gets externally sourced multicast traffic into the +rack at all. + +The traffic here originates on the **host**, not in a guest, so these runs +exercise the external-to-underlay ingress path, from host through to switch and +to sled for each subscribing member. Nothing described below drives the +guest-sourced egress path (or OPTE's sled-side next-hop selection). Exercising +that path would take a sender from inside the rack, where, here, the members +only answer: a probe's echo reply is unicast, so no guest originates multicast +data traffic. A joining guest does emit an IGMPv3 or MLDv2 membership report, +which is a multicast frame, but those go to the protocol's own link-local +address (224.0.0.22, ff02::16) rather than to the group, and OPTE encapsulates +only what its multicast-to-physical table maps. That table holds admin-scoped +underlay addresses for materialized groups alone, so a report is denied rather +than encapsulated and the sled-side next-hop selection never runs. The rack +drives forwarding from the API subscription, not from the report. Consuming it +instead is what [RFD 488]'s dynamic group identification (IGMP snooping and +querying) proposes, and nothing here exercises that as of yet. + +`voxel commtest --traffic multi` performs every API step below automatically. +The API sections document what it creates, and how to create the same objects +by hand when taking a run apart. + +## Prerequisites + +- A launched rack with RSS complete and the external API answering. Multicast + needs no configuration knobs of its own, as nothing in `voxel.toml` enables + it. +- The host-side setup below (routes, mirror, membership). `voxel network + multicast up` installs it in whichever `[external]` mode is set, and `voxel + commtest --setup-mcast` does the same before a run. Isolated mode resolves + ce's and cr1's addresses from config alone, while `lan` mode reads their DHCP + leases from the running nodes over the falcon console. The plumbing lives in + [`multicast.rs`], the run wrapper that invokes it in [`commtest.rs`]. +- A control-plane image whose Omicron carries multicast. `voxel commtest + --traffic multi` detects an older, unicast-only commit and fails with that + diagnosis before building. The detection inspects the `--source` checkout, + not the commit baked into the running rack's image. A rack image that + predates probe multicast silently drops a probe's `multicast_groups` and + surfaces later as no-delivery, so be sure to keep the two commits aligned. +- The commtest privilege setup from the [README's Privileges section]: the run + needs `net_icmpaccess` in the effective set and refuses uid 0. +- `curl` and `jq` for the API examples below. Voxel installs neither, and + neither is needed for a `voxel commtest` run. + +### Dependencies + +> TODO: the branches below are in flight and will collapse into their main +> branches as they land upstream. Until then, the Omicron pin is the pushed leaf +> of `zl/mcast-build`, held by the workspace `rack-init-config` dependency in +> `Cargo.toml`; `build.rs` surfaces this bookmark to `voxel image create`, +> so that a commitless `voxel image create` builds the pin. + +The multicast stack spans multiple repositories, but voxel itself pins only two +of them: the Omicron commit handed to `voxel image create` (`OMICRON_REPO` +selects the clone source) and the sidecar-lite artifact rev the build fetches +(`SIDECAR_LITE_REV`, already defaulted to the multicast rev). Everything else +rides the chosen Omicron commit's own pins, so pointing the image at the right +Omicron commit pulls the whole set. + +| Repository | Branch / rev | PR | Carried by | Trajectory | +| --- | --- | --- | --- | --- | +| omicron | `zl/mcast-build` leaf | #11128 open, top of the PR stack below | workspace `rack-init-config` pin, via `voxel image create` | stack lands bottom-up into `main`, starting at #9912 | +| dendrite | `multicast-e2e`, `3d49b131` | #224 open | Omicron `tools/dendrite_*` pins | #224 to `main` | +| maghemite | `zl/ddm-mcast`, `96a2f153` | #696 open, stacked on `zl/mrib`; related #402 (`zl/mgd-ddm-meta`) | Omicron `tools/maghemite_*` pins | #696 to `main` after `zl/mrib` | +| opte | `master`, `0525f2f95` (0.41.506) | #1012 merged 2026-07-25 | Omicron `Cargo.toml` / `tools/opte_version` | landed; Omicron pins 0.41.506 intentionally (master's later #1040 is a comment-only xde change) | +| propolis | `zl/multicast`, `3c07d60a` | [#1093] open | Omicron `package-manifest.toml` (guest propolis-server) | [#1093] to `master`; host side also needs the viona V7 tables below | +| thundermuffin | `zl/multicast-joiner`, `486559bc` | #14 open | Omicron `package-manifest.toml` (probe zone, prebuilt) | #14 to `main` | +| sidecar-lite | `zl/multicast`, `461cbe19` | #152 open | `voxel image create` (`SIDECAR_LITE_REV`) | #152 to `main` | +| softnpu | `zl/multicast`, `284c6830` | #183 open | Omicron `tools/softnpu_version` (xtask `SOFTNPU_COMMIT`) | #183 to `main` | +| p4 | `zl/multicast` (`p4rs`) | #240 open | transitive, via softnpu and sidecar-lite lockfiles | #240 to `main`, then #183/#152 repoint | + +The omicron work is a stack of PRs, each based on the branch below it. Voxel +pins the leaf, so one commit carries the whole chain: + +| PR | Branch | Base | +| --- | --- | --- | +| [omicron#11128] | `zl/mcast-build` (voxel's pin) | `zl/mcast-e2e-commtest` | +| [omicron#11118] | `zl/mcast-e2e-commtest` | `zl/probe-multicast` | +| [omicron#10520] | `zl/probe-multicast` | `zl/multicast-mgd-ddm` | +| [omicron#10346] | `zl/multicast-mgd-ddm` | `zl/multicast-m2p-forwarding` | +| [omicron#10070] | `zl/multicast-m2p-forwarding` | `multicast-e2e` | +| [omicron#9912] | `multicast-e2e` | `main` | + +Thundermuffin's [multicast joiner] is the receiver-side prerequisite for probe +tests, shipped in the image by Omicron's probe package. It runs inside each +probe zone and holds the ASM or SSM socket membership that lets the zone's IP +stack accept and answer multicast, standing in for a guest application. +Voxel's `ip maddress` entry on `cr1` is independent sender-path plumbing: it +admits the group's Ethernet address to the mirror router so `tc` can copy the +frame toward the rack. Probe-based commtest requires both. + +Two pieces sit outside the image, on the host itself: + +- The **host propolis** needs propolis `zl/multicast`'s viona MAC-filter + wiring (PR [#1093] above), or the sled VMs receive no multicast at all. + This is the falcon VM boundary, not the rack's guest instances: each sled + is itself a propolis VM whose illumos `vioif` negotiates + `VIRTIO_NET_F_CTRL_RX` but never programs a multicast table, so viona + narrows the link to no-multicast at feature negotiation and drops every + group frame. It also needs the SMBIOS type 1 fix from [#1200], which is + not multicast-specific. Without it falcon's a4x2 identity never reaches + the sled VM and RSS fails trust quorum validation on any voxel rack. The + `mcast-smbios-test` branch (`8bb6a90b`) merges both. + + Build it with the `falcon` feature: + + ```sh + git checkout mcast-smbios-test # zl/multicast + propolis#1200 + cargo build --release --bin propolis-server --features falcon + ``` + + and point `[falcon].propolis_binary` at the resulting + `target/release/propolis-server`. +- The **host viona kernel module** needs the MAC-filter table ioctls from + stlouis#986, Gerrit change [775] on illumos-gate, merged into `stlouis` as + `5ffff4b8` on 2026-08-19. Run the host on an illumos build at or past that + commit. From a [helios] checkout, pull the latest `stlouis`, build it, and + install it onto a new boot environment (see [`helios-build onu`]): + + ```sh + git -C projects/illumos pull # latest stlouis, includes 5ffff4b8 + ./helios-build build-illumos -q + ./helios-build onu -t viona-mac-filters + ``` + + then reboot into the new BE. + + Hosts without the change fall back to unfiltered RX, which still delivers + a single copy but without filtering. + + Verify a host with `nm /usr/kernel/drv/amd64/viona | grep set_mac_filters`. + +## Reaching the API + +Nexus answers on the rack's `[network].service_pool` addresses that external DNS +does not hold. With the defaults (pool `198.51.100.20-.29`, external DNS on +`.20` and `.21`), probe upward from `.22`: + +```sh +for ip in $(seq 22 29); do + curl -sf -m 2 -o /dev/null "http://198.51.100.$ip/v1/ping" \ + && echo "198.51.100.$ip" +done +``` + +`voxel commtest` derives `--api` from the same address sweep, though with +plain TCP connection checks on ports 80 and 443 (external DNS addresses as a +fallback) rather than an HTTP ping. + +Authentication for these examples uses the recovery silo. Its `voxel.toml` +defaults are silo `recovery`, user `recovery`, and password `oxide`, matching +the login `commtest` performs. `oxide auth login` runs a device flow and stores +a token the later calls reuse. On a headless host, `--no-browser` prints a URL +that can be opened elsewhere. + +```sh +API=http://198.51.100.23 +oxide auth login --host "$API" +oxide api /v1/multicast-groups +``` +The typed CLI makes the same call: +```sh +oxide experimental multicast-group list +``` + +*Note*: every multicast endpoint currently carries the `experimental` tag, so +the typed commands land under `oxide experimental ...` in a CLI build whose +spec includes them. Even then the structured request fields, a join's +`source_ips` and a probe's `multicast_groups` and `pool_selector`, get no +flag of their own and are reachable only through `--json-body `. The +examples below use `oxide api`, the raw passthrough, which takes that same +JSON on stdin and works regardless of how much of the surface the installed +CLI knows about. The typed form follows each call where one exists. + +Without a CLI, `curl` does the same work. A local login returns a session cookie +that stands in for the token: + +```sh +SESSION=$(curl -si -X POST "$API/v1/login/recovery/local" \ + -H 'content-type: application/json' \ + -d '{"username":"recovery","password":"oxide"}' \ + | sed -n 's/^set-cookie: \(session=[^;]*\).*/\1/p') + +curl -s -H "Cookie: $SESSION" "$API/v1/multicast-groups" | jq +``` + +## IP pools + +A pool carries a `pool_type` discriminator, `unicast` (the default) or +`multicast`. Multicast pools are further constrained: + +- One IP version per pool (`ip_version`, `v4` or `v6`). +- Every range in the pool must be entirely Any-Source Multicast (ASM) or + entirely Source-Specific Multicast (SSM), never both. SSM is `232.0.0.0/8` + for IPv4 and the per-scope `ff3x::/32` blocks for IPv6 ([RFC 4607]); + everything else is ASM. An ASM group set and an SSM group set therefore + need two pools. The split is about address space, not filtering: joins on + ASM addresses may still carry `source_ips` (see [Join forms](#join-forms)). +- A silo may hold at most one default pool per (pool type, IP version) pair, + four in total. A multicast pool linked non-default is still usable; the group + is then resolved by address rather than by the silo default. + +```sh +oxide api /v1/system/ip-pools --method POST --input - <<'JSON' +{ "name": "mcast-v4-asm", "description": "ASM multicast pool", + "ip_version": "v4", "pool_type": "multicast" } +JSON + +oxide api /v1/system/ip-pools/mcast-v4-asm/silos --method POST --input - <<'JSON' +{ "silo": "recovery", "is_default": false } +JSON + +oxide api /v1/system/ip-pools/mcast-v4-asm/ranges/add --method POST --input - <<'JSON' +{ "first": "239.100.0.1", "last": "239.100.0.2" } +JSON +``` +The typed equivalents live under plain `oxide ip-pool`, since pools are stable +CLI surface, not experimental: +```sh +oxide ip-pool create --name mcast-v4-asm --description "ASM multicast pool" \ + --ip-version v4 --pool-type multicast +oxide ip-pool silo link --pool mcast-v4-asm --silo recovery --is-default false +oxide ip-pool range add --pool mcast-v4-asm \ + --first 239.100.0.1 --last 239.100.0.2 +``` + +The SSM pool, `mcast-v4-ssm`, is the same three calls with `232.100.0.1` as +both ends of the range. Members also need an ordinary unicast pool for their +external addresses. `commtest` creates one named `default` over its +`--ip-pool-begin/--ip-pool-end` range and links it to the silo as the default. + +Both pool list endpoints filter by type, which is how to pick the multicast +pools out of a mixed fleet. The installed CLI's `oxide ip-pool list` predates +the filter flags, so this one stays raw: + +```sh +oxide api "/v1/system/ip-pools?pool_type=multicast" +oxide api "/v1/ip-pools?pool_type=multicast" # silo-scoped view +``` + +## Groups + +A group is not created directly. There is no `POST /v1/multicast-groups`, for +example. A group comes into existence when the first member joins an address +(or a name) that a linked multicast pool covers, and Nexus reaps it once its +last member leaves. The `multicast_reconciler` background task drives its +transitions and the switch programming behind them. + +``` + ip pool pool_type = multicast, one ip_version, ASM xor SSM + mcast-v4-asm : 239.100.0.1 - 239.100.0.2 + | + | linked to the silo + v + first join of an address the pool covers + (instance PUT .../multicast-groups/G, or probe create) + | + v + group 239.100.0.1 + Creating --[multicast_reconciler]--> Active + ^ | + | later joins attach as members | + | v + members myvm (*,G), probe0@g0 (S,G) + Joining --[multicast_reconciler]--> Joined --> Left + | + | last member leaves + v + group empty + Deleting --[multicast_reconciler]--> Deleted +``` + +The read side is `GET /v1/multicast-groups`, `/v1/multicast-groups/{group}`, and +`/v1/multicast-groups/{group}/members`, where `{group}` is a name, a UUID, or +the multicast IP. + +```sh +oxide api /v1/multicast-groups \ + | jq -r '.items[] | "\(.name) \(.multicast_ip) \(.state)"' +oxide api /v1/multicast-groups/239.100.0.1/members +``` +The typed CLI makes the same calls: +```sh +oxide experimental multicast-group list +oxide experimental multicast-group view --multicast-group 239.100.0.1 +oxide experimental multicast-group member list --multicast-group 239.100.0.1 +``` + +A group view carries `multicast_ip`, `ip_pool_id`, `state`, the deduplicated +union of its members' `source_ips`, and `has_any_source_member`. The union is +contributed to only by members that joined with an explicit source list, so a +non-empty `source_ips` does not imply that every member filters by source; +`has_any_source_member` is what answers that. + +### Join forms + +Every join takes the same body: an optional `source_ips` array and an optional +`ip_version` (needed only when a join creates a group by name and both an +IPv4 and an IPv6 default multicast pool are linked). The forms differ in +what the source list means: + +- **ASM**, e.g. `239.100.0.1` with no sources. An any-source `(*, G)` join. +- **SSM**, e.g. `232.100.0.1` with `source_ips`. SSM builds no shared `(*, G)` + tree, so every join must carry a source list, and Nexus rejects a bare SSM + join. +- **Source-bound ASM**, e.g. `239.100.0.2` with `source_ips`. An ASM group + joined `(S, G)`, which exercises source filtering ([RFC 3376]) on an address + that does not require it. + +A member's source list has a defined maximum of 32 entries +(`MAX_SOURCE_IPS_PER_MEMBER`), and the union across a group's members is capped +at 256 (`MAX_SOURCE_IPS_PER_GROUP`). Neither bound comes from the protocol. +IGMPv3 ([RFC 3376]) and MLDv2 ([RFC 3810]) leave per-group source-list size +implementation-defined, and implementations diverge accordingly: Linux defaults +to 10 (`igmp_max_msf`), FreeBSD to 128 (`maxsocksrc`). 32 covers the typical +one to eight sources per channel while keeping a single member's fan-out from +dominating the shared `(S, G)` forwarding state, and 256 bounds what one group +can install in the dataplane. For any source-filtered join, the list must +include whatever sends the verification traffic, or the dataplane correctly +drops it. That is also the recipe for the negative case: +join with a source list that excludes the host and assert nothing arrives +(`commtest --mcast-deny-group GROUP@SRC`). The examples below write the +sending address as `$SRC`, resolved under [Sending traffic](#sending-traffic). + +### Instances + +An instance joins and leaves by group identifier, and lists its own +memberships. The first join implicitly creates the group unless the +identifier is a UUID, which must name an existing group. + +```sh +oxide api "/v1/instances/myvm/multicast-groups/239.100.0.1?project=classone" \ + --method PUT --input - <<'JSON' +{} +JSON + +oxide api "/v1/instances/myvm/multicast-groups/232.100.0.1?project=classone" \ + --method PUT --input - < ssm-join.json </dev/null || true + pfexec route add -host "$group" "$CE" +done +``` + +The route table belongs to the Helios host rather than to any one, specific +Falcon environment, so voxel records each group's gateway in +`.falcon/multicast-.json` and treats that record, not the +gateway address, as proof of ownership. `up` writes the record before adding +the route. An interrupted run then leaves a record with no route, which the +next `up` overwrites and a groupless `down` reads as nothing to remove. A +reverse ordering would leave a route that nothing afterwards could prove was +voxel's to begin with. Both commands stop rather than guess when a group's +route is not in the record: `up` refuses the group instead of displacing the +route, and `down` leaves it alone and names it. Deleting the record while its +routes exist, therefore, locks voxel out of those groups until they are removed +by hand with `pfexec route delete -host `. + +Two isolated-mode environments sharing an `[external]` subnet are the one case +the record cannot separate. They place a given group on the same `ce` address, +so the two records describe a single kernel route, and a `down` in either +environment takes it away from both. + +### The `cr1` mirror + +Host routes only put the frames on the external segment, addressed toward +`ce`, and no router forwards them across the transit path into a switch. The +naming matches the customer-edge and transit split of [RFC 4364]: `ce` is the +edge, and only the `cr*` transit routers hold switch-facing links. `cr1` sits +on the same host-facing segment and sees the flood directly, so rather than +introducing a multicast routing daemon, a stock iproute2 `tc` ingress filter +mirrors each group from `cr1`'s host-facing NIC to both switch-facing NICs +via the [`mirred`][tc-mirred] action. + +Router NIC names follow falcon's link ordering, the same derivation +`VoxelConfig::router_ext_iface` encodes: a fabric router's links are `ce` first, +then every scrimlet across every rack, then its own external NIC. For the stock +single-rack, this is the four-sled topology that makes up `cr1`: + +- `enp0s8`, toward `ce` +- `enp0s9`, toward `g0` (switch0) +- `enp0s10`, toward `g3` (switch1) +- `enp0s11`, host-facing, where the multicast flood arrives + +The filter mirrors both switch-facing NICs. External multicast ingresses at +whichever switch holds the group's external NAT entry, and Omicron's +designated-forwarder election ([omicron#11128]) gates that entry to a single +switch, a designation made in the control plane, not visible to the mirror. The +elected switch ingests and replicates to the underlay, while the other has no +entry and drops its copy, so mirroring to both costs only the second copy and +guarantees the elected one is reached. The mirror does not choose the ingress +switch. + +`voxel network multicast up` installs these filters itself, and `voxel +commtest --setup-mcast` does the same before a run. The manual equivalent, +run on `cr1` (`voxel host login cr1`; `host exec` covers sleds only): + +```sh +IIF=enp0s11 + +# Ensure the shared clsact qdisc exists without recreating it. Deleting it +# would drop every ingress filter on the device, including ones this guide +# does not own. Per-group filters below use `replace`, which is idempotent. +tc qdisc add dev $IIF clsact 2>/dev/null || true + +# Both sidecars must be chained actions within one filter per group. Separate +# per-sidecar filters do not work: the first matching filter ends flower +# classification, so the second never fires and a group elected to that switch +# goes undelivered. Chained actions all run, since mirred's default control is +# pipe. +pref=100 +for group in 239.100.0.1 239.100.0.2 232.100.0.1; do + # The explicit handle keeps `replace` idempotent. Left at 0, the kernel + # treats a re-run as a fresh insert and flower returns EEXIST for the + # duplicate key. + tc filter replace dev $IIF ingress handle 1 pref $pref protocol ip \ + flower dst_ip $group \ + action mirred egress mirror dev enp0s9 \ + action mirred egress mirror dev enp0s10 + pref=$((pref + 1)) +done + +tc filter show dev $IIF ingress +``` + +### Group membership on `cr1` + +The mirror only sees frames the NIC accepts, so until the NIC accepts a +group's frames, the filter above matches nothing. An IPv4 group's frames +arrive under a derived Ethernet address: the group's low-order 23 bits placed +into `01:00:5e:00:00:00` ([RFC 1112] section 6.4). The prefix is IANA's OUI, +`00-00-5E`, and of the 2^24 multicast identifiers one OUI provides, only the +lower half is allotted to IPv4 ([RFC 7042] section 2.1.1), so a group's 28 +significant bits fold onto 23 and 32 groups alias each Ethernet address. The +example groups above demonstrate the aliasing: `239.100.0.1` and `232.100.0.1` +both map to `01:00:5e:64:00:01`. + +A NIC accepts a multicast address only while some form of membership holds it. +Voxel's FRR router speaks no multicast protocol directly and nothing else on +that emulated router joins, so without a membership the NIC drops the flood +before the tc ingress hook and the mirror counters stay at zero. A static +link-layer membership stands in for the missing join: + +```sh +ip maddress add 01:00:5e:64:00:01 dev $IIF # 239.100.0.1 and 232.100.0.1 +ip maddress add 01:00:5e:64:00:02 dev $IIF # 239.100.0.2 +``` + +`voxel network multicast up` derives and pins these itself, one per distinct +Ethernet address, recording each one it adds under `/run` on `cr1`. `down` +removes a membership only when no remaining group still maps onto it and the +record shows `up` created it, so one the router already held (the kernel's +all-hosts mapping, or a join inside the router) is left alone. The record +lives and dies with the router, exactly like the memberships themselves. + +This static membership is part of voxel's test scaffolding, not a customer or +rack configuration requirement. Customers do not run these Linux `ip maddress` +commands on the rack. They do still configure the upstream network to deliver +the group toward both rack uplinks, whether via PIM/IGMP, static multicast +routes and joins on the upstream routers, or an equivalent mechanism. + +### Verifying the host path + +`voxel network multicast check` reads all of the above back from the live +state, printing one line per item (host route, mirror filter, link-layer +membership) per group and `PASS`/`FAIL` overall. With no `--group`, it covers +everything this Falcon environment has plumbed thus far, using its `.falcon/` +state and the selected router's owned filters. A route belonging to another +environment is left alone. This is the same set a groupless `down` tears down. + +If the router address is known but SSH cannot reach it, `check` and `down` +count or verify host routes only and warn that router state could not be +inspected. In `lan` mode, the address must first be read from the running +router over the falcon console. If that lookup fails, `check` reports an error +and live `down` stops after host-route cleanup instead of assuming that router +state is gone; a dry-run skips the router preview. + +``` +ok: host route 232.100.0.1 -> 172.30.199.14 +ok: host route 239.100.0.1 -> 172.30.199.14 +ok: mirror of 232.100.0.1 on enp0s11 -> enp0s9 enp0s10 +ok: mirror of 239.100.0.1 on enp0s11 -> enp0s9 enp0s10 +ok: membership 01:00:5e:64:00:01 (232.100.0.1) on enp0s11 +ok: membership 01:00:5e:64:00:01 (239.100.0.1) on enp0s11 +underlay: 232.100.0.1 -> ff04::e864:1 on switch0 (g0) +underlay: 239.100.0.1 -> ff04::ef64:1 on switch0 (g0) +underlay: 232.100.0.1 not programmed on switch1 (g3) +underlay: 239.100.0.1 not programmed on switch1 (g3) +check: PASS +``` + +*Note* that the two groups above share a membership address. [RFC 1112]'s +mapping keeps only the low 23 bits of the group address, so 32 groups alias +each Ethernet address and these two differ only in the discarded bits. The +aliasing is confined to the link layer on `cr1`, where one `ip maddress` +entry admits both. The underlay groups stay distinct (`ff04::e864:1` against +`ff04::ef64:1`), because that mapping embeds the full v4 address, so nothing +downstream of the switch conflates them. Teardown accounts for the aliasing +as well, dropping a membership only when no remaining group still maps to it. + +Trailing `underlay:` lines connect that external path to the rack. Each +switch zone's `swadm multicast list` names the NAT target an external group +maps onto, the admin-scoped underlay group the switch replicates toward the +members. Those entries exist only once the group exists in the control plane +(a commtest run or an API join creates it), so `not programmed` on freshly +plumbed groups just means not yet, and the lines' output never affects the +check's result. + +### Sending traffic + +The source address the members must permit, `$SRC` above, is the host's address +on the external segment: `[external].host_ip` in isolated mode (default +`172.30.199.199`), or the host's LAN address otherwise. + +`voxel commtest` drives the whole thing, creating the pools, project, and probes +before pinging each group and asserting every member replies within tolerance. +`--mcast-deny-group` adds the negative case from the join forms above: the +member's source list excludes the host, so the run asserts nothing arrives. +Every group needs the host-side setup first, deny groups included. No delivery +is also what a missing route or mirror produces, meaning that a deny-only run +would otherwise pass without ever reaching the dataplane, and voxel's +preflight refuses to start until the pieces are in place. Run +`voxel network multicast up` over the full set, or pass `--setup-mcast` to +have commtest do it: + +```sh +voxel network multicast up --group 239.100.0.1 --group 239.100.0.2 \ + --group 232.100.0.1 --group 239.100.0.9 + +voxel commtest --source /oxide/workspace/omicron --traffic multi -- run \ + --test-duration 200s --warmup 10s --packet-rate 10 \ + --mcast-group 239.100.0.1 \ + --mcast-group 239.100.0.2@172.30.199.199 \ + --mcast-group 232.100.0.1@172.30.199.199 \ + --mcast-deny-group 239.100.0.9@172.30.199.198 +``` + +The deny source is any address other than the host's, here `172.30.199.198`, +so the joined filter excludes the actual sender and the dataplane must drop +the traffic. + +With no `--mcast-group`, voxel supplies `239.1.1.1`, which then needs its own +host route, mirror filter, and link-layer membership. See the commtest +section of the [README] for the build and privilege details. + +Against probes created manually, `ping` is enough. illumos `ping -s` to a group +address prints a reply line per responder, so every probe address from the join +above should answer. `-t` raises the multicast TTL past the default of 1, which +otherwise expires the request before it clears `cr1` (see the TTL caveat +below): + +```sh +for group in 239.100.0.1 239.100.0.2 232.100.0.1; do + echo "== $group ==" + pfexec ping -s -t 16 "$group" 56 10 +done +``` + +### Teardown + +Run `voxel network multicast down` before `voxel destroy`, while the rack is +reachable. The host routes outlive `voxel destroy` and otherwise have to be +removed explicitly. Voxel keeps a per-environment host-route record under +`.falcon/`, while router-state discovery remains tied to the selected router +VM. Destroy leaves multicast state alone because router-state cleanup requires +that explicit target. The mirror and memberships normally go with `cr1`, since +they are runtime state inside the router. `voxel network multicast down` covers +all of the host-side state and takes the same repeated `--group` as `up`: + +```sh +voxel network multicast down \ + --group 239.100.0.1 \ + --group 239.100.0.2 \ + --group 232.100.0.1 +``` + +A `commtest` run leaves its `classone` project and the IP pools in place. +The separate cleanup pass (`voxel commtest -- cleanup`) deletes the project +but not the pools. Manually created objects unwind in dependency order: probes, +then the project's default subnet and VPC (a project cannot be deleted while +it holds a VPC), then the project, then each multicast pool after unlinking +it from the silo. Groups need no step of their own, as Nexus reaps a group +once its last member leaves. + +```sh +for probe in $(oxide api "/experimental/v1/probes?project=classone" | jq -r '.items[].name'); do + oxide api "/experimental/v1/probes/$probe?project=classone" --method DELETE + # typed: oxide experimental system probe delete \ + # --probe "$probe" --project classone +done + +oxide api "/v1/vpc-subnets/default?project=classone&vpc=default" --method DELETE +oxide api "/v1/vpcs/default?project=classone" --method DELETE +oxide api /v1/projects/classone --method DELETE + +for pool in mcast-v4-asm mcast-v4-ssm; do + oxide api "/v1/system/ip-pools/$pool/silos/recovery" --method DELETE + oxide api "/v1/system/ip-pools/$pool" --method DELETE +done +``` + +## Static multicast assignment and delivery without PIM or IGMP + +The `cr1` arrangement above is voxel's answer to a question customer networks +face too: how does externally sourced multicast reach the rack when nothing on +the path runs PIM and no receiver sends IGMP reports? Per [RFD 488], v1 uses +static, API-driven assignment: the control plane programs the rack's multicast +state, but the rack neither signals membership upstream nor learns it from +guests. [RFD 488] proposes one addition per direction, each toggleable per +availability zone and usable alone or together. IGMP host-proxying has the rack +advertise its membership upstream per [RFC 4605]. Dynamic group identification +has the rack snoop and query guest IGMP or MLD, deriving membership from +reports rather than from the API. The external NAT entry accepts a group's +traffic at whichever uplink delivers it. Getting the traffic to an uplink is the +customer network's job, and every mechanism voxel uses has a static production +equivalent. [RFD 488] also notes that a network with pre-configured static +multicast routes to the rack needs neither of those additions. + +**Across a flat L2 segment.** A switch with no IGMP snooping floods multicast +out every port, which delivers without any configuration. A snooping switch +constrains flooding to reported ports, but [RFC 4541] section 2.1.2 requires +unregistered groups be forwarded toward router ports and permits forwarding +on all ports, so vendor defaults often still deliver. To pin it down +deterministically, most switches accept static snooping entries binding a group +address to the rack-facing ports, the same shape as voxel's static membership +plus mirror. + +**Across a routed hop.** A router between source and rack needs forwarding +state it would normally learn from PIM or IGMP. Two static substitutes: + +- A static group membership on the rack-facing interface (Cisco's + `ip igmp static-group`, with counterparts on most vendors). The router then + forwards the group out that interface as if a receiver had joined there. +- A static `(S, G)` forwarding-cache entry. On a Linux router, [smcroute] + manages exactly this, writing static multicast routes into the kernel's + multicast forwarding cache (MFC) with no signaling protocol at all, the + routed analogue of voxel's tc mirror. + +**Caveats that apply to any static setup.** + +- Static delivery takes the receiver out of the loop because the state exists + whether or not anyone subscribes, so that any active source reaches the rack. + Idle groups cost nothing, but a live one keeps flowing until the operator + removes the configuration, since no leave or prune ever fires, and a + `(*, G)` entry takes every source sending to that group. +- The sender's TTL must clear every routed hop. [RFC 1112] specifies a default + TTL of 1 for multicast, so a source that works on its own segment silently + dies at the first router until the application raises it. +- Deliver to **both** rack uplinks. The rack elects which switch holds a + group's external NAT entry, and the election can move. The non-elected + switch drops its copy in the dataplane, so duplicating toward both costs + only the second uplink's bandwidth and is correct, exactly as the `cr1` + mirror does. The bandwidth-saving alternative is dynamic signaling: with + [RFD 488]'s proposed IGMP host-proxying ([RFC 4605]), the rack would + advertise its membership upstream and traffic would follow the election + instead of being duplicated. + +What voxel does not model is the dynamic case: a customer network where PIM +and IGMP are running end to end, with the upstream building its distribution +tree from receiver membership. That is the direction [RFD 488]'s IGMP host-proxying +(future work) targets, and nothing here exercises or validates that signaling. + +> TODO: when [RFD 488]'s host-proxying lands, exercising it here means +> replacing the static scaffolding for that mode. The emulated upstream would +> have to listen rather than be pinned: an IGMP querier on the external +> segment and forwarding driven by the rack's proxied reports (a snooping +> bridge or [smcroute] on `cr1`), in place of a membership and mirror that +> deliver whether or not anyone signals. + +## Troubleshooting + +| Symptom | Where to look | +| --- | --- | +| Group absent from `GET /v1/multicast-groups` | The join never resolved a pool. Confirm a multicast pool is linked to the silo and its range covers the address. | +| Group stuck in "Creating", or members in "Joining" | `omdb nexus background-tasks show multicast_reconciler`, then activate it. | +| "Active" group, "Joined" members, no replies | Underlay or ingress. Check `omdb nexus multicast ddm-peers --mcast`, then the host route and `tc filter show` on `cr1`. | +| One group delivers, another does not | A per-group artifact: its host route, its `tc` filter, or its membership. | +| Frames reach `cr1`'s host-facing NIC (tcpdump sees them) but the mirror counters stay zero | The group's link-layer membership is absent, so the NIC drops the frames before the tc hook. `voxel network multicast check` reports it; `ip maddress show dev enp0s11` should hold the group's `01:00:5e` address. Note that tcpdump masks this by putting the NIC in promiscuous mode. | +| SSM group silent, ASM groups fine | The join's `source_ips` must contain the sending host address. | +| Some members reply, others do not | Per-sled. `omdb db multicast members --group-ip ` shows which sled each member landed on. | +| `up` reports a host route "is not recorded for Falcon environment" | The route predates this environment's `.falcon/` record, either from another environment or from a record that was deleted while its routes remained. Voxel will not displace it. Remove it with `pfexec route delete -host ` if it is stale. | +| Every per-group artifact checks out and delivery still fails | The sled dataplane. Run opte's [`opte-mcast-delivery.d`] in a sled's global zone, where `xde` is loaded. `NOFWD` names a missing forwarding entry, `FILTERED` a source-filter drop, and the delivery matrix reports which ports took a copy. The script is not in the sled image, so copy it from an opte checkout. | + +*TODO*: IPv6 multicast is not wired up in `commtest` yet, and the isolated external +segment voxel creates is v4-only. The API objects are not the gap: a `v6` +multicast pool and its groups work like their v4 counterparts. `lan` mode +also inherits whatever v6 the LAN carries, so what is missing is voxel's +wiring, not the rack or the topology. Until then, v6 groups are out of reach +from this host-sourced path. + +[#1093]: https://github.com/oxidecomputer/propolis/pull/1093 +[#1200]: https://github.com/oxidecomputer/propolis/pull/1200 +[omicron#11128]: https://github.com/oxidecomputer/omicron/pull/11128 +[omicron#11118]: https://github.com/oxidecomputer/omicron/pull/11118 +[omicron#10520]: https://github.com/oxidecomputer/omicron/pull/10520 +[omicron#10346]: https://github.com/oxidecomputer/omicron/pull/10346 +[omicron#10070]: https://github.com/oxidecomputer/omicron/pull/10070 +[omicron#9912]: https://github.com/oxidecomputer/omicron/pull/9912 +[`helios-build onu`]: https://github.com/oxidecomputer/helios#installing-locally-on-your-build-machine +[helios]: https://github.com/oxidecomputer/helios +[775]: https://code.oxide.computer/c/illumos-gate/+/775 +[`multicast.rs`]: ../voxel/src/multicast.rs +[`commtest.rs`]: ../voxel/src/commtest.rs +[`opte-mcast-delivery.d`]: https://github.com/oxidecomputer/opte/blob/master/dtrace/opte-mcast-delivery.d +[README]: ../README.md +[README's Privileges section]: ../README.md#privileges +[RFC 1112]: https://datatracker.ietf.org/doc/html/rfc1112 +[RFC 3376]: https://datatracker.ietf.org/doc/html/rfc3376 +[RFC 3810]: https://datatracker.ietf.org/doc/html/rfc3810 +[RFC 4364]: https://datatracker.ietf.org/doc/html/rfc4364 +[RFC 4541]: https://datatracker.ietf.org/doc/html/rfc4541 +[RFC 4605]: https://datatracker.ietf.org/doc/html/rfc4605 +[RFC 4607]: https://datatracker.ietf.org/doc/html/rfc4607 +[RFC 7042]: https://datatracker.ietf.org/doc/html/rfc7042 +[RFD 488]: https://rfd.shared.oxide.computer/rfd/0488 +[smcroute]: https://github.com/troglobit/smcroute +[tc-mirred]: https://man7.org/linux/man-pages/man8/tc-mirred.8.html +[multicast joiner]: https://github.com/oxidecomputer/thundermuffin/pull/14 diff --git a/docs/parameters.md b/docs/parameters.md index c0c9798..7a8fb97 100644 --- a/docs/parameters.md +++ b/docs/parameters.md @@ -33,7 +33,7 @@ Falcon settings resolve as: flag, then `voxel.toml`, then env, then built-in. | Key | Type | Default | Notes | |-----|------|---------|-------| | `version` | string | `"proto"` | Shorthand suffix for both images (`voxel-cp-`, `voxel-frr-`). Ignored when `cp`/`frr` are set. | -| `cp` | string | unset | Full cp image name. Overrides `version`. Keep the `voxel-cp-` form so the matching omicron checkout is found. | +| `cp` | string | unset | Full cp image name. Overrides `version`. Unset follows the workspace's omicron pin (`voxel-cp-`, the image a commitless `voxel image create` bakes). Keep the `voxel-cp-` form so the matching omicron checkout is found. | | `frr` | string | unset | Full frr image name. Overrides `version`. | | `data_links_schema` | enum | unset | `list` or `tagged`. Unset auto-detects from the image. | | `disks_schema` | enum | unset | `vdevs`, `external_disks`, or `hardcoded` (omicron#10948). Unset auto-detects from the image. | @@ -102,6 +102,7 @@ Runtime paths. Each unset value resolves via env then built-in default. | `workdir` | string | directory of `voxel.toml` | Root that `cargo-bay/` and `.falcon/` live under. Absolute. | | `build_root` | string | `$BUILD_ROOT`, else `$HOME/voxel-builds` | Root for `voxel image create` (omicron checkouts). | | `propolis_binary` | string | unset | `propolis-server` the host runs each node under. Unset leaves falcon's own binary, which it downloads on demand. Set it to run a locally built propolis, e.g. for a device-model fix that has not reached a release. Rack nodes only, as the image-build VM keeps falcon's binary. | +| `ssh_pubkey` | string | first of `~/.ssh/id_ed25519.pub`, `id_ecdsa.pub`, `id_rsa.pub` | SSH public key staged into every node's cargo-bay as `root_authorized_keys`; voxel-init appends it to root's `authorized_keys`, so `ssh root@` authenticates by key instead of the empty password. Content-validated before staging (a private key is refused). When unset and no default key exists, staging is skipped. | ## [sp] diff --git a/voxel-config/src/config.rs b/voxel-config/src/config.rs index e40d9e9..4a423f3 100644 --- a/voxel-config/src/config.rs +++ b/voxel-config/src/config.rs @@ -31,7 +31,7 @@ pub const SLED_SERIAL_PREFIX: &str = "2FAKE00"; /// Part number shared by all fake sleds. pub const SLED_PART_NUMBER: &str = "913-0000019"; -/// Top-level Voxel configuration (voxel.toml). +/// Top-level voxel configuration (`voxel.toml`). #[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct VoxelConfig { @@ -197,6 +197,15 @@ pub struct Falcon { /// Applies to rack nodes only. The image-build VM keeps falcon's own /// binary. pub propolis_binary: Option, + /// SSH public key staged into every node's cargo-bay as + /// `root_authorized_keys`, which voxel-init appends to root's + /// `authorized_keys`, so plain `ssh root@` authenticates by key + /// instead of the empty password. `None` -> the first of + /// `~/.ssh/id_ed25519.pub`, `id_ecdsa.pub`, `id_rsa.pub` that exists; + /// staging is skipped when none do. The file's content is validated as a + /// public key before staging (the cargo-bay is mounted into every guest, + /// so a private key must never land there). + pub ssh_pubkey: Option, } /// SP provider selection: which SPs run on the real-firmware emulator sp-emu @@ -378,7 +387,7 @@ impl Topology { for local in 0..self.sleds { let index = rack * self.sleds + local; let name = format!("g{index}"); - let serial_number = format!("{SLED_SERIAL_PREFIX}{}", index); + let serial_number = format!("{SLED_SERIAL_PREFIX}{index}"); let part_number = SLED_PART_NUMBER.to_string(); out.push(SledDesc { rack, @@ -596,7 +605,7 @@ pub struct Network { /// IPv6 /56. Empty -> not emitted. pub rack_subnet: String, /// Service IP pool (single range). Rendered as the rack's sole - /// service_ip_pools entry. + /// `service_ip_pools` entry. pub service_pool_first: String, pub service_pool_last: String, pub bgp_asn: u32, @@ -910,6 +919,24 @@ impl VoxelConfig { format!("enp0s{n}") } + /// A fabric router's scrimlet-facing NIC names, in `sleds()` order. Empty + /// for `ce`, which links only fabric routers. + /// + /// Same derivation as [`Self::router_ext_iface`]: a fabric router links + /// `ce` at `FRR_IFACE_BASE`, then every scrimlet across every rack, so + /// scrimlet `k` sits at `enp0s{FRR_IFACE_BASE + 1 + k}`. A falcon change + /// that shifts the base moves these names too, along with the external + /// NIC voxel-init verifies at bring-up. These are the ports a host-side + /// mirror feeds to reach the rack's switches. + pub fn router_scrimlet_ifaces(&self, router: &str) -> Vec { + if router == "ce" { + return Vec::new(); + } + (0..self.sleds().into_iter().filter(|s| s.scrimlet).count()) + .map(|k| format!("enp0s{}", FRR_IFACE_BASE + 1 + k)) + .collect() + } + /// Each customer router's frr.conf. cr* peer ce plus every scrimlet across /// all racks and originate nothing; ce originates the default route. pub fn to_frr(&self) -> Vec<(String, FrrRouter)> { @@ -1778,6 +1805,25 @@ mod tests { assert_eq!(cfg.router_ext_iface("cr2"), "enp0s13"); } + #[test] + fn router_scrimlet_ifaces_slots() { + // Scrimlet-facing NICs follow ce at enp0s8, so they start at enp0s9 and + // must stop exactly where `router_ext_iface` picks up. + let cfg = VoxelConfig::default(); + assert_eq!(cfg.router_scrimlet_ifaces("cr1"), ["enp0s9", "enp0s10"]); + assert_eq!(cfg.router_ext_iface("cr1"), "enp0s11"); + assert!(cfg.router_scrimlet_ifaces("ce").is_empty()); + + let multi = + VoxelConfig::from_toml("[topology]\nracks = 2\nsleds = 3\n") + .unwrap(); + assert_eq!( + multi.router_scrimlet_ifaces("cr1"), + ["enp0s9", "enp0s10", "enp0s11", "enp0s12"] + ); + assert_eq!(multi.router_ext_iface("cr1"), "enp0s13"); + } + #[test] fn image_names() { let mut img = Image::default(); diff --git a/voxel-image/README.md b/voxel-image/README.md index 4840029..29f5384 100644 --- a/voxel-image/README.md +++ b/voxel-image/README.md @@ -1,6 +1,6 @@ # voxel-image (prototype) -Machinery to build pre-built Voxel images via the snapshot-first path: boot one +Machinery to build pre-built voxel images via the snapshot-first path: boot one node, install baked software onto it, then capture its disk as a falcon base image (optionally a distributable `_0.raw.xz`). Topologies later boot from these images and apply topology-specific config at launch, so one image serves every diff --git a/voxel-init/src/gimlet.rs b/voxel-init/src/gimlet.rs index cfef206..1648f56 100644 --- a/voxel-init/src/gimlet.rs +++ b/voxel-init/src/gimlet.rs @@ -10,7 +10,8 @@ //! kicks RSS on the RSS node). use crate::sys::{ - note, read_external_net, replace_in_file, run, run_env, run_quiet, warn, + append_authorized_keys, note, read_external_net, replace_in_file, run, + run_env, run_quiet, warn, }; use anyhow::{Context, Result, bail}; use camino::Utf8Path; @@ -49,7 +50,7 @@ fn wait_until(max_s: u32, mut f: impl FnMut() -> bool) -> bool { pub fn bring_up() -> Result<()> { setup_ssh(); - crash_dump(); + disable_crash_dump(); maybe_load_sidecar(); // The omicron CLI tools are baked into the image at /opt/oxide/omicron, and @@ -87,25 +88,7 @@ pub fn bring_up() -> Result<()> { /// function). illumos sshd defaults differ from debian's, hence the explicit /// config edits. fn setup_ssh() { - let authorized = format!("{CARGO_BAY}/root_authorized_keys"); - if Utf8Path::new(&authorized).exists() { - let _ = fs::create_dir_all("/root/.ssh"); - if let Ok(keys) = fs::read(&authorized) { - use std::io::Write; - match fs::OpenOptions::new() - .create(true) - .append(true) - .open("/root/.ssh/authorized_keys") - { - Ok(mut f) => { - if let Err(e) = f.write_all(&keys) { - warn(format!("authorized_keys: {e}")); - } - } - Err(e) => warn(format!("authorized_keys: {e}")), - } - } - } + append_authorized_keys(&format!("{CARGO_BAY}/root_authorized_keys")); run("ssh-keygen", &["-A"]); replace_in_file( "/etc/ssh/sshd_config", @@ -118,9 +101,23 @@ fn setup_ssh() { run("svcadm", &["restart", "svc:/network/ssh:default"]); } -fn crash_dump() { - run("zfs", &["create", "-p", "-V", "8G", "rpool/dump"]); - run("dumpadm", &["-d", "/dev/zvol/dsk/rpool/dump"]); +/// Disable kernel crash dumps and reclaim the `rpool/dump` zvol. +/// +/// The guest rpool is about 96G while the sparse U.2/M.2 vdev backing files +/// staged in /var/tmp hold 140G of combined potential, so the pool is +/// deliberately overcommitted and only ever fills (file vdevs never return +/// freed blocks). An 8G dump reservation buys nothing on a debug VM and +/// giving it back delays the ENOSPC cliff that otherwise kills svc.configd +/// and leaves dendrite in the switch zone unresponsive. +/// +/// The destroy also reaps a zvol persisted by an image baked before this change, +/// since falcon keeps the sled disk across destroy/relaunch. +/// `dumpadm -d none` must come first because zfs refuses to destroy the active +/// dump device, and `run_quiet` swallows the expected failure on a fresh disk +/// with no zvol. +fn disable_crash_dump() { + run("dumpadm", &["-d", "none"]); + run_quiet("zfs", &["destroy", "rpool/dump"]); } /// Scrimlets load the baked SoftNPU sidecar P4 program. Gimlets have no softnpu @@ -533,8 +530,8 @@ fn open_switch_zone_ssh() { /// with one contract-daemon instance per SP (startd supervises + restarts each, /// survives reboots). No-op when nothing's staged; idempotent once imported. fn setup_sp_emu() { - // The emu fleet (binary + per-role hubris archives + rot image) is STAGED in - // the cargo-bay (dev: [sp].emu_bin set) or BAKED at /opt/oxide/sp-emu. Staged + // The emu fleet (binary + per-role hubris archives + rot image) is staged in + // the cargo-bay (dev: [sp].emu_bin set) or baked at /opt/oxide/sp-emu. Staged // wins; baked is the fallback. The SP set, per-SP role, VPD identity, and // --emu-rot come from the `ports` manifest topo always stages. const BAKED: &str = "/opt/oxide/sp-emu"; diff --git a/voxel-init/src/router.rs b/voxel-init/src/router.rs index 4bae202..b706d80 100644 --- a/voxel-init/src/router.rs +++ b/voxel-init/src/router.rs @@ -9,8 +9,8 @@ //! its upstream). use crate::sys::{ - ExternalNet, capture, note, read_external_net, replace_in_file, run, - run_quiet, warn, + ExternalNet, append_authorized_keys, capture, note, read_external_net, + replace_in_file, run, run_quiet, warn, }; use anyhow::{Context, Result, bail}; use camino::Utf8Path; @@ -54,25 +54,7 @@ pub fn bring_up() -> Result<()> { /// sshd_config: voxel authenticates as root with the rack's empty password, and /// Debian's stock `PermitRootLogin prohibit-password` refuses that. fn setup_ssh() { - let authorized = "/opt/cargo-bay/root_authorized_keys"; - if Utf8Path::new(authorized).exists() { - let _ = fs::create_dir_all("/root/.ssh"); - if let Ok(keys) = fs::read(authorized) { - use std::io::Write; - match fs::OpenOptions::new() - .create(true) - .append(true) - .open("/root/.ssh/authorized_keys") - { - Ok(mut f) => { - if let Err(e) = f.write_all(&keys) { - warn(format!("authorized_keys: {e}")); - } - } - Err(e) => warn(format!("authorized_keys: {e}")), - } - } - } + append_authorized_keys("/opt/cargo-bay/root_authorized_keys"); // Debian's stock sshd_config keeps PasswordAuthentication yes but defaults // PermitRootLogin to prohibit-password. Flip it so serial-first debugging diff --git a/voxel-init/src/sys.rs b/voxel-init/src/sys.rs index 93e029c..e5c621d 100644 --- a/voxel-init/src/sys.rs +++ b/voxel-init/src/sys.rs @@ -7,6 +7,7 @@ //! every step is visible and best-effort steps log a warning instead of //! aborting. Mirror that—`run`/`run_quiet` never panic and return success. +use std::fs; use std::process::{Command, Stdio}; /// Parsed `/opt/cargo-bay/external-net` (voxel-managed isolated segment). All @@ -70,6 +71,58 @@ pub fn warn(msg: impl AsRef) { println!("[voxel-init] WARN: {}", msg.as_ref()); } +/// Append the staged operator key(s) at `staged` (the cargo-bay's +/// `root_authorized_keys`, if present) to root's `authorized_keys`, skipping +/// lines already there. Both role agents run this on every boot and the node +/// disk survives destroy/relaunch, so a plain append would accumulate +/// duplicates. +/// Permissions are pinned to 0700/0600 because sshd's default `StrictModes` +/// rejects looser ones. +/// +/// This is best-effort, like the rest of bring-up. +pub fn append_authorized_keys(staged: &str) { + use std::os::unix::fs::{DirBuilderExt, PermissionsExt}; + if !std::path::Path::new(staged).exists() { + return; + } + let keys = match fs::read_to_string(staged) { + Ok(k) => k, + Err(e) => { + warn(format!("authorized_keys: read {staged}: {e}")); + return; + } + }; + let dir = "/root/.ssh"; + let path = "/root/.ssh/authorized_keys"; + if let Err(e) = + fs::DirBuilder::new().recursive(true).mode(0o700).create(dir) + { + warn(format!("authorized_keys: {dir}: {e}")); + return; + } + let existing = fs::read_to_string(path).unwrap_or_default(); + let mut out = existing.clone(); + if !out.is_empty() && !out.ends_with('\n') { + out.push('\n'); + } + for line in keys.lines() { + let line = line.trim_end_matches('\r'); + if line.is_empty() || existing.lines().any(|l| l == line) { + continue; + } + out.push_str(line); + out.push('\n'); + } + if out != existing + && let Err(e) = fs::write(path, &out) + { + warn(format!("authorized_keys: write: {e}")); + return; + } + let _ = fs::set_permissions(dir, fs::Permissions::from_mode(0o700)); + let _ = fs::set_permissions(path, fs::Permissions::from_mode(0o600)); +} + /// Apply literal `(from, to)` substitutions to `path` in one rewrite. Both role /// agents use it to relax sshd_config, where the patterns are the distro's /// shipped lines, commented or not. A pattern that does not match is silently a diff --git a/voxel/Cargo.toml b/voxel/Cargo.toml index cc4c9ce..15e4ded 100644 --- a/voxel/Cargo.toml +++ b/voxel/Cargo.toml @@ -19,6 +19,7 @@ futures.workspace = true clap.workspace = true camino.workspace = true indoc.workspace = true +itertools.workspace = true oxnet.workspace = true voxel-config = { path = "../voxel-config" } # Omicron's own RackInitializeRequest types and config-rss serialization, @@ -29,6 +30,7 @@ wicketd-commission-client.workspace = true # Reshape the generated config-rss.toml into wicketd's RSS config JSON for # `launch --wicket-setup` (drive setup through wicketd). toml = "0.9" +serde = { workspace = true, features = ["derive"] } serde_json = "1" reqwest = { version = "0.13", default-features = false, features = ["rustls"] } libc = "0.2" diff --git a/voxel/src/access.rs b/voxel/src/access.rs index f982908..201b603 100644 --- a/voxel/src/access.rs +++ b/voxel/src/access.rs @@ -113,8 +113,9 @@ pub(crate) async fn cmd_host_login( node: &str, ) -> anyhow::Result<()> { let topo = build_topo(cfg, name)?; - // Routers accept the same root SSH login (the FRR image bakes in sshd with - // the operator key), so `host login` covers them too. + // Routers accept the same root SSH login (the FRR image ships sshd and + // voxel-init relaxes its config, plus any staged operator key), so + // `host login` covers them too. let (n, is_router) = topo .sleds .iter() diff --git a/voxel/src/commission.rs b/voxel/src/commission.rs index 346db9a..f2f4fa8 100644 --- a/voxel/src/commission.rs +++ b/voxel/src/commission.rs @@ -101,6 +101,9 @@ fn uplink_port( enforce_first_as: false, allowed_import: Default::default(), allowed_export: Default::default(), + // Source addresses apply to numbered sessions only (ours is + // unnumbered). + src_addr: None, vlan_id: None, }], ), @@ -343,7 +346,7 @@ pub(crate) async fn drive( .await .map_err(|e| anyhow!("upload cert: {e}"))?; client - .post_rss_config_key(&types::PrivateKeyPem(key)) + .post_rss_config_key(&types::PrivateKeyPem(key.into())) .await .map_err(|e| anyhow!("upload key: {e}"))?; client diff --git a/voxel/src/commtest.rs b/voxel/src/commtest.rs index 873858f..45d875b 100644 --- a/voxel/src/commtest.rs +++ b/voxel/src/commtest.rs @@ -2,7 +2,7 @@ // License, v. 2.0. If a copy of the MPL was not distributed with this // file, You can obtain one at https://mozilla.org/MPL/2.0/. -//! Build and run Omicron's `commtest` against a Voxel rack. +//! Build and run Omicron's `commtest` against a voxel rack. //! //! The source defaults to the checkout matching the configured control-plane //! image. An explicit git ref can select another Omicron era (including the @@ -28,7 +28,15 @@ const RUN_SUBCOMMAND: &str = "run"; /// TODO: IPv4 only, since commtest rejects v6 groups during validation. Pick a /// v6 default once its own `validate_mcast` TODO to add the v6 pool buckets and /// a v6 arm in `test_mcast_connectivity` is discharged. -const DEFAULT_MCAST_GROUP: &str = "239.1.1.1"; +pub(crate) const DEFAULT_MCAST_GROUP: &str = "239.1.1.1"; +/// Loss tolerance appended when the caller does not pass +/// `--icmp-loss-tolerance`. Commtest defaults to zero, which is the right +/// threshold for real hardware, but a virtual rack shares one host across +/// every sled VM and sheds a few packets at the virtio rings under burst. +/// Omicron's a4x2 CI passes the same value +/// (`.github/buildomat/jobs/a4x2-deploy.sh`). +/// Pass `--icmp-loss-tolerance 0` through to restore the strict threshold. +const DEFAULT_ICMP_LOSS_TOLERANCE: u32 = 500; const HELIOS_RUSTFLAGS: &str = "--cfg svcadm_autoclear \ -C link-arg=-R/usr/platform/oxide/lib/amd64 \ -C link-arg=-Wl,-znocompstrtab --cfg tokio_unstable"; @@ -74,12 +82,14 @@ pub(crate) struct Options<'a> { pub api_override: Option<&'a str>, pub traffic: Traffic, pub no_build: bool, + pub setup_mcast: bool, pub allow_root: bool, pub passthrough: &'a [String], } -pub(crate) fn run( +pub(crate) async fn run( cfg: &VoxelConfig, + name: &str, options: Options<'_>, ) -> anyhow::Result<()> { let Options { @@ -88,6 +98,7 @@ pub(crate) fn run( api_override, traffic, no_build, + setup_mcast, allow_root, passthrough, } = options; @@ -120,9 +131,11 @@ pub(crate) fn run( passthrough, supports_multicast(&source)?, )?; + let groups = preflight_groups(traffic, &args); + ensure_mcast_plumbing(cfg, name, &groups, setup_mcast).await?; if !no_build { - eprintln!("[voxel] building Omicron commtest from {}", source); + eprintln!("[voxel] building Omicron commtest from {source}"); let mut cargo = Command::new("cargo"); cargo.current_dir(&source).args([ "build", @@ -135,9 +148,8 @@ pub(crate) fn run( require_success(cargo.status(), "cargo build commtest")?; } else if !bin.is_file() { bail!( - "{} does not exist; omit --no-build or build it with \ - `cargo build -p end-to-end-tests --bin commtest`", - bin + "{bin} does not exist; omit --no-build or build it with \ + `cargo build -p end-to-end-tests --bin commtest`" ); } @@ -174,13 +186,13 @@ fn run_streamed( .stdout(Stdio::piped()) .stderr(Stdio::piped()) .spawn() - .with_context(|| format!("run {}", bin))?; + .with_context(|| format!("run {bin}"))?; let out = child.stdout.take().expect("stdout piped above"); let err = child.stderr.take().expect("stderr piped above"); let t_out = std::thread::spawn(move || tee(out, std::io::stdout(), log)); let t_err = std::thread::spawn(move || tee(err, std::io::stderr(), log_err)); - let status = child.wait().with_context(|| format!("wait for {}", bin))?; + let status = child.wait().with_context(|| format!("wait for {bin}"))?; join_tee(t_out, "stdout")?; join_tee(t_err, "stderr")?; Ok(status) @@ -250,10 +262,10 @@ fn resolve_source( fn validate_source(path: &Utf8Path) -> anyhow::Result { let path = path .canonicalize_utf8() - .with_context(|| format!("resolve Omicron source {}", path))?; + .with_context(|| format!("resolve Omicron source {path}"))?; let commtest = path.join("end-to-end-tests/src/bin/commtest.rs"); if !path.join("Cargo.toml").is_file() || !commtest.is_file() { - bail!("{} is not an Omicron checkout with {}", path, commtest); + bail!("{path} is not an Omicron checkout with {commtest}"); } Ok(path) } @@ -348,9 +360,9 @@ fn checkout(reference: &str) -> anyhow::Result { std::env::var("OMICRON_REPO").unwrap_or_else(|_| DEFAULT_REPO.into()); std::fs::create_dir_all(&root) - .with_context(|| format!("create commtest cache {}", root))?; + .with_context(|| format!("create commtest cache {root}"))?; if !repository.exists() { - eprintln!("[voxel] creating Omicron Git cache in {}", repository); + eprintln!("[voxel] creating Omicron Git cache in {repository}"); let mut clone = Command::new("git"); clone.args(["clone", "--mirror", "--", &repo]).arg(&repository); require_success(clone.status(), "git clone Omicron")?; @@ -372,13 +384,13 @@ fn checkout(reference: &str) -> anyhow::Result { validate_worktree(&source, &wanted)?; } else { std::fs::create_dir_all(&worktrees).with_context(|| { - format!("create worktree directory {}", worktrees) + format!("create worktree directory {worktrees}") })?; require_success( git_dir_command(&repository).args(["worktree", "prune"]).status(), "git worktree prune", )?; - eprintln!("[voxel] creating detached Omicron worktree {}", source); + eprintln!("[voxel] creating detached Omicron worktree {source}"); require_success( git_dir_command(&repository) .args(["worktree", "add", "--detach", "--"]) @@ -398,7 +410,7 @@ fn validate_repository( if git_dir_output(repository, &["rev-parse", "--is-bare-repository"])? != "true" { - bail!("{} exists but is not a bare Git repository", repository); + bail!("{repository} exists but is not a bare Git repository"); } // `remote get-url` applies the user's `url..insteadOf` rewrites and // can false-mismatch the configured URL. Read the raw remote instead. @@ -406,11 +418,8 @@ fn validate_repository( git_dir_output(repository, &["config", "--get", "remote.origin.url"])?; if actual_remote != expected_remote { bail!( - "{} uses origin '{}', but OMICRON_REPO is '{}'; use a different \ - BUILD_ROOT for the other repository", - repository, - actual_remote, - expected_remote + "{repository} uses origin '{actual_remote}', but OMICRON_REPO is '{expected_remote}'; use a different \ + BUILD_ROOT for the other repository" ); } Ok(()) @@ -442,8 +451,7 @@ fn resolve_reference( match matches.as_slice() { [commit_id] => Ok(commit_id.clone()), [] => bail!( - "Omicron commit or ref '{reference}' was not found in {}", - repository + "Omicron commit or ref '{reference}' was not found in {repository}" ), _ => bail!( "Omicron ref '{reference}' is ambiguous; use a full refs/heads/... \ @@ -505,9 +513,8 @@ fn validate_worktree(source: &Utf8Path, wanted: &str) -> anyhow::Result<()> { )?; if !dirty.is_empty() { bail!( - "{} has tracked local changes; move them to a separate checkout and \ - use --source, or restore this cached worktree", - source + "{source} has tracked local changes; move them to a separate checkout and \ + use --source, or restore this cached worktree" ); } Ok(()) @@ -581,7 +588,7 @@ fn target_dir(source: &Utf8Path) -> Utf8PathBuf { } } -/// Reproduce `voxel-image/build-cp.sh`'s build environment, so commtest links +/// Reproduce `voxel image create`'s build environment, so commtest links /// against the same Helios runtime as the image it tests. /// /// Caller-supplied flags win out: cargo ignores `RUSTFLAGS` once @@ -675,7 +682,7 @@ fn api_candidates(network: &Network) -> Vec { fn supports_multicast(source: &Utf8Path) -> anyhow::Result { let source_file = source.join("end-to-end-tests/src/bin/commtest.rs"); let text = std::fs::read_to_string(&source_file) - .with_context(|| format!("read {}", source_file))?; + .with_context(|| format!("read {source_file}"))?; Ok(text.contains("skip_unicast") && text.contains("mcast_group")) } @@ -698,13 +705,28 @@ fn commtest_args_for( apply_traffic(source, traffic, supports_multicast, &mut args)?; - let has_begin = args - .iter() - .any(|a| a == "--ip-pool-begin" || a.starts_with("--ip-pool-begin=")); - let has_end = args - .iter() - .any(|a| a == "--ip-pool-end" || a.starts_with("--ip-pool-end=")); + if !has_arg(&args, "--icmp-loss-tolerance") { + args.push("--icmp-loss-tolerance".into()); + args.push(DEFAULT_ICMP_LOSS_TOLERANCE.to_string()); + } + + let has_begin = has_arg(&args, "--ip-pool-begin"); + let has_end = has_arg(&args, "--ip-pool-end"); if has_begin && has_end { + let begin = arg_value(&args, "--ip-pool-begin") + .and_then(|v| v.parse::().ok()); + let end = arg_value(&args, "--ip-pool-end") + .and_then(|v| v.parse::().ok()); + // commtest hands the pair to Nexus as-is, so an inverted range fails + // minutes in as a pool creation error rather than here. + if let (Some(begin), Some(end)) = (begin, end) + && begin > end + { + bail!( + "--ip-pool-begin {begin} is above --ip-pool-end {end}; \ + the range is inverted." + ); + } return Ok(args); } // Deriving the missing half of a partial override would pair a caller's @@ -715,7 +737,7 @@ fn commtest_args_for( bail!( "pass both --ip-pool-begin and --ip-pool-end, or neither. Voxel \ derives the pair from [network], and mixing the two produces a \ - range that overlaps the service pool." + range that either overlaps the service pool or becomes inverted." ); } @@ -747,9 +769,8 @@ fn apply_traffic( } Traffic::Multicast | Traffic::Both if !supports_multicast => { bail!( - "{} does not support multicast commtest; select --traffic unicast \ - or use an Omicron commit containing multicast commtest support", - source + "{source} does not support multicast commtest; select --traffic unicast \ + or use an Omicron commit containing multicast commtest support" ); } Traffic::Multicast => { @@ -780,6 +801,21 @@ fn has_arg(args: &[String], name: &str) -> bool { args.iter().any(|a| a == name || a.starts_with(&format!("{name}="))) } +/// The value following `name`, in either `--flag value` or `--flag=value` +/// spelling, or `None` when the flag is absent or trails without a value. +fn arg_value<'a>(args: &'a [String], name: &str) -> Option<&'a str> { + let mut rest = args.iter(); + while let Some(arg) = rest.next() { + if let Some(value) = arg.strip_prefix(&format!("{name}=")) { + return Some(value); + } + if arg == name { + return rest.next().map(String::as_str); + } + } + None +} + fn add_default_mcast_group(args: &mut Vec) { if !has_arg(args, "--mcast-group") && !has_arg(args, "--mcast-deny-group") { args.push("--mcast-group".into()); @@ -787,6 +823,92 @@ fn add_default_mcast_group(args: &mut Vec) { } } +/// The groups the multicast preflight covers. Empty here when the run skips +/// the multicast phase (`--traffic unicast`, or a passed-through `--skip-mcast`), +/// even if an explicit `--mcast-group` rides along in the arguments, so the +/// preflight cannot block, or under `--setup-mcast` plumb, a phase commtest +/// never enters. +fn preflight_groups(traffic: Traffic, args: &[String]) -> Vec { + if matches!(traffic, Traffic::Unicast) || has_arg(args, "--skip-mcast") { + return Vec::new(); + } + mcast_groups(args) +} + +/// Collect the group addresses from the assembled commtest arguments, +/// defaulted or passed through, in either `--flag value` or `--flag=value` +/// spelling. +/// +/// Deny groups count. They expect no delivery, which is also what missing +/// plumbing produces, so a deny-only run would otherwise pass without ever +/// reaching the dataplane it is meant to exercise. +fn mcast_groups(args: &[String]) -> Vec { + const FLAGS: [&str; 2] = ["--mcast-group", "--mcast-deny-group"]; + let mut out = Vec::new(); + let mut rest = args.iter(); + while let Some(arg) = rest.next() { + if let Some(group) = + FLAGS.iter().find_map(|f| arg.strip_prefix(&format!("{f}="))) + { + out.push(group.to_string()); + } else if FLAGS.contains(&arg.as_str()) + && let Some(group) = rest.next() + { + out.push(group.clone()); + } + } + out +} + +/// Refuse to start a multicast run before the host plumbing is in place, or +/// plumb it here under `setup`. +/// +/// Without the mirror and host routes the traffic never leaves the host, and +/// commtest reports that as a receive timeout minutes into the run rather than +/// as missing setup. +async fn ensure_mcast_plumbing( + cfg: &VoxelConfig, + name: &str, + groups: &[String], + setup: bool, +) -> anyhow::Result<()> { + if groups.is_empty() { + return Ok(()); + } + let missing = crate::multicast::missing_plumbing(cfg, name, groups).await?; + if missing.is_empty() { + return Ok(()); + } + if setup { + crate::multicast::up(cfg, name, groups, false).await?; + let still_missing = + crate::multicast::missing_plumbing(cfg, name, groups).await?; + + if still_missing.is_empty() { + return Ok(()); + } + + bail!( + "host multicast plumbing still incomplete after setup:\n {}", + still_missing.join("\n ") + ); + } + + // A groupless `up` already covers the default group, but explicit --group + // flags suppress that default, so any other list is spelled out in full. + let flags: String = if groups == [DEFAULT_MCAST_GROUP] { + String::new() + } else { + groups.iter().map(|g| format!(" --group {g}")).collect() + }; + + bail!( + "host multicast plumbing is incomplete:\n {}\nrun `voxel network multicast up{flags}` \ + first, or pass --setup-mcast", + missing.join("\n ") + ); +} + fn derive_pool( network: &Network, sleds: usize, @@ -827,7 +949,7 @@ fn derive_pool( } #[cfg(test)] -mod test { +mod tests { use super::*; #[test] @@ -858,6 +980,8 @@ mod test { .unwrap(), [ "run", + "--icmp-loss-tolerance", + "500", "--ip-pool-begin", "198.51.100.30", "--ip-pool-end", @@ -873,6 +997,7 @@ mod test { "--api-timeout".into(), "5m".into(), "run".into(), + "--icmp-loss-tolerance=0".into(), "--ip-pool-begin=203.0.113.10".into(), "--ip-pool-end".into(), "203.0.113.20".into(), @@ -905,6 +1030,35 @@ mod test { ); } + #[test] + fn defaults_loss_tolerance_alongside_explicit_pool() { + let network = Network::default(); + let explicit = vec![ + "run".to_string(), + "--ip-pool-begin=203.0.113.10".into(), + "--ip-pool-end=203.0.113.20".into(), + ]; + let args = commtest_args_for( + Utf8Path::new("/tmp/old-omicron"), + &network, + 4, + Traffic::Unicast, + &explicit, + false, + ) + .unwrap(); + assert_eq!( + args, + [ + "run", + "--ip-pool-begin=203.0.113.10", + "--ip-pool-end=203.0.113.20", + "--icmp-loss-tolerance", + "500", + ] + ); + } + #[test] fn rejects_partial_pool_override() { let network = Network::default(); @@ -930,6 +1084,37 @@ mod test { } } + #[test] + fn rejects_inverted_pool_override() { + let network = Network::default(); + for inverted in [ + vec![ + "run".to_string(), + "--ip-pool-begin=203.0.113.20".into(), + "--ip-pool-end=203.0.113.10".into(), + ], + vec![ + "run".to_string(), + "--ip-pool-begin".into(), + "203.0.113.20".into(), + "--ip-pool-end".into(), + "203.0.113.10".into(), + ], + ] { + assert!( + commtest_args_for( + Utf8Path::new("/tmp/old-omicron"), + &network, + 4, + Traffic::Unicast, + &inverted, + false + ) + .is_err() + ); + } + } + #[test] fn selects_multicast_phases() { let network = Network::default(); @@ -959,6 +1144,87 @@ mod test { assert!(both.contains(&DEFAULT_MCAST_GROUP.into())); } + #[test] + fn preflight_reads_back_the_groups_commtest_uses() { + // The preflight has to cover passed-through groups, in either spelling, + // not just the default this module appends. + let network = Network::default(); + let unicast = commtest_args_for( + Utf8Path::new("/tmp/new-omicron"), + &network, + 4, + Traffic::Unicast, + &[], + true, + ) + .unwrap(); + assert!(mcast_groups(&unicast).is_empty()); + + let defaulted = commtest_args_for( + Utf8Path::new("/tmp/new-omicron"), + &network, + 4, + Traffic::Multicast, + &[], + true, + ) + .unwrap(); + assert_eq!(mcast_groups(&defaulted), [DEFAULT_MCAST_GROUP]); + + let passthrough = [ + "--mcast-group".to_string(), + "224.0.2.5".to_string(), + "--mcast-group=224.0.2.6".to_string(), + ]; + let explicit = commtest_args_for( + Utf8Path::new("/tmp/new-omicron"), + &network, + 4, + Traffic::Multicast, + &passthrough, + true, + ) + .unwrap(); + assert_eq!(mcast_groups(&explicit), ["224.0.2.5", "224.0.2.6"]); + } + + #[test] + fn unicast_never_preflights_a_passthrough_group() { + let network = Network::default(); + let passthrough = vec![ + "run".to_string(), + "--mcast-group".to_string(), + "224.0.2.5".to_string(), + ]; + let args = commtest_args_for( + Utf8Path::new("/tmp/new-omicron"), + &network, + 4, + Traffic::Unicast, + &passthrough, + true, + ) + .unwrap(); + // The group rides through to commtest, which ignores it under the + // appended --skip-mcast, and the preflight must ignore it too. + assert!(args.contains(&"--skip-mcast".into())); + assert_eq!(mcast_groups(&args), ["224.0.2.5"]); + assert!(preflight_groups(Traffic::Unicast, &args).is_empty()); + + // The gate is the phase selection, not the group spelling: the same + // passthrough under --traffic both is preflighted. + let both = commtest_args_for( + Utf8Path::new("/tmp/new-omicron"), + &network, + 4, + Traffic::Both, + &passthrough, + true, + ) + .unwrap(); + assert_eq!(preflight_groups(Traffic::Both, &both), ["224.0.2.5"]); + } + #[test] fn rejects_multicast_on_old_commtest() { assert!( diff --git a/voxel/src/config_cmd.rs b/voxel/src/config_cmd.rs index a588a9a..7e5368c 100644 --- a/voxel/src/config_cmd.rs +++ b/voxel/src/config_cmd.rs @@ -33,18 +33,17 @@ pub(crate) fn cmd_config( vcfg::set(&text, key, value).map_err(|e| anyhow!(e))?; ensure_parent_dir(path)?; fs::write(path, &updated) - .with_context(|| format!("write {}", path))?; + .with_context(|| format!("write {path}"))?; println!("{key} = {value}"); } ConfigCmd::Load { file } => { let text = fs::read_to_string(file) - .with_context(|| format!("read {}", file))?; + .with_context(|| format!("read {file}"))?; VoxelConfig::from_toml(&text) - .map_err(|e| anyhow!("invalid config {}: {e}", file))?; + .map_err(|e| anyhow!("invalid config {file}: {e}"))?; ensure_parent_dir(path)?; - fs::write(path, &text) - .with_context(|| format!("write {}", path))?; - println!("loaded {} -> {}", file, path); + fs::write(path, &text).with_context(|| format!("write {path}"))?; + println!("loaded {file} -> {path}"); } } Ok(()) @@ -56,7 +55,7 @@ fn ensure_parent_dir(path: &Utf8Path) -> anyhow::Result<()> { if let Some(dir) = path.parent() && !dir.as_os_str().is_empty() { - fs::create_dir_all(dir).with_context(|| format!("create {}", dir))?; + fs::create_dir_all(dir).with_context(|| format!("create {dir}"))?; } Ok(()) } diff --git a/voxel/src/cpbuild.rs b/voxel/src/cpbuild.rs index b75005e..23ff90c 100644 --- a/voxel/src/cpbuild.rs +++ b/voxel/src/cpbuild.rs @@ -16,8 +16,10 @@ use crate::imagebuild::{ BakeOpts, bake, builder_network, repo_root, toolchain_bin, }; -/// sidecar-lite pinned rev. TODO repin to main once zl/multicast merges. -const SIDECAR_LITE_REV: &str = "6f3311e8acd7e7e95c167aab61188355a93afe72"; +/// sidecar-lite pinned rev, the zl/multicast tip. +/// +/// TODO: repin to main once zl/multicast merges. +const SIDECAR_LITE_REV: &str = "461cbe1926b93b20c2f43ad5cd9007b193db61a6"; const SIDECAR_URL: &str = "https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release"; /// Build flags matching the validated recipe for building omicron on Helios. @@ -44,7 +46,8 @@ pub(crate) struct CpBuild<'a> { /// The omicron sha voxel's own rack-init-config dependency is pinned to. Empty while /// the dependency is a path dep. Set by build.rs from Cargo.lock. -const PINNED_OMICRON_REV: &str = env!("RACK_INIT_CONFIG_OMICRON_REV"); +pub(crate) const PINNED_OMICRON_REV: &str = + env!("RACK_INIT_CONFIG_OMICRON_REV"); /// `voxel image create`: resolve where the omicron source is and what the image /// is called, then build. `--src ` builds that checkout as-is with @@ -157,31 +160,22 @@ pub(crate) async fn create_cp(b: CpBuild<'_>) -> Result<()> { let repo = std::env::var("OMICRON_REPO").unwrap_or_else(|_| { "https://github.com/oxidecomputer/omicron".into() }); - run( - Command::new("git").arg("clone").arg(&repo).arg(src), - "git clone omicron", - )?; + run(git().arg("clone").arg(&repo).arg(src), "git clone omicron")?; } eprintln!("[voxel] checking out {commit}"); // A fetch failure is tolerable: the commit may already be local. - let _ = Command::new("git") + let _ = git() .arg("-C") .arg(src) .args(["fetch", "--all", "--tags", "-q"]) .status(); run( - Command::new("git") - .arg("-C") - .arg(src) - .args(["checkout", "-q", commit]), + git().arg("-C").arg(src).args(["checkout", "-q", commit]), "git checkout", )?; // Drop leftover local edits so the pinned commit builds pristine. run( - Command::new("git") - .arg("-C") - .arg(src) - .args(["checkout", "-q", "--", "."]), + git().arg("-C").arg(src).args(["checkout", "-q", "--", "."]), "git restore tracked files", )?; } @@ -363,6 +357,22 @@ fn fetch_sidecar(voxel_image: &Utf8Path, dest: &Utf8Path) -> Result<()> { Ok(()) } +/// A `git` command that ignores the invoking user's git config. `voxel image +/// create` runs under pfexec, which keeps the caller's HOME, so root's git +/// would otherwise read the user's `~/.gitconfig`. An `insteadOf` rewrite +/// there turns the anonymous HTTPS clone into SSH that root has no host keys +/// for, and root reading a user-owned `OMICRON_REPO` checkout trips git's +/// `safe.directory` ownership check. Scrubbing both config files and passing +/// `safe.directory` back in command scope (taken into effect since git 2.37's +/// protected configuration) covers both. +pub(crate) fn git() -> Command { + let mut c = Command::new("git"); + c.env("GIT_CONFIG_GLOBAL", "/dev/null") + .env("GIT_CONFIG_SYSTEM", "/dev/null") + .args(["-c", "safe.directory=*"]); + c +} + /// A command run inside the omicron checkout, with the PATH and RUSTFLAGS the /// omicron build needs. `install_builder_prerequisites.sh` ci-downloads /// cockroach/clickhouse/dpd into `out/` and then fails unless they are on PATH - diff --git a/voxel/src/image.rs b/voxel/src/image.rs index 52e6b50..9548048 100644 --- a/voxel/src/image.rs +++ b/voxel/src/image.rs @@ -39,7 +39,7 @@ pub(crate) fn ensure_image(image: &str) -> anyhow::Result<()> { /// The checkout's short HEAD sha (the default `--src` image label). pub(crate) fn head_short_sha(src: &Utf8Path) -> anyhow::Result { - let out = std::process::Command::new("git") + let out = crate::cpbuild::git() .arg("-C") .arg(src) // Under pfexec the checkout is usually owned by the invoking user, @@ -48,7 +48,7 @@ pub(crate) fn head_short_sha(src: &Utf8Path) -> anyhow::Result { .arg(format!("safe.directory={src}")) .args(["rev-parse", "--short", "HEAD"]) .output() - .with_context(|| format!("run git in {}", src))?; + .with_context(|| format!("run git in {src}"))?; if !out.status.success() { bail!( "git rev-parse HEAD failed in {src}: {}", @@ -93,9 +93,9 @@ pub(crate) fn render_smf( for (rel, text) in writes { let path = omicron_root.join(rel); let dir = path.parent().expect("smf path has a parent"); - fs::create_dir_all(dir).with_context(|| format!("mkdir {}", dir))?; - fs::write(&path, text).with_context(|| format!("write {}", path))?; - println!("rendered {}", path); + fs::create_dir_all(dir).with_context(|| format!("mkdir {dir}"))?; + fs::write(&path, text).with_context(|| format!("write {path}"))?; + println!("rendered {path}"); } Ok(()) } @@ -294,7 +294,7 @@ pub(crate) fn cmd_image( }; let out = out.clone().unwrap_or_else(|| Utf8PathBuf::from(default_out)); - eprintln!("[voxel] exporting {snap} -> {}", out); + eprintln!("[voxel] exporting {snap} -> {out}"); let status = std::process::Command::new("bash") .arg("-c") .arg(format!("{pipe} > {}", shell_quote(out.as_str()))) @@ -306,7 +306,7 @@ pub(crate) fn cmd_image( if *raw { "xz" } else { "zstd" } ); } - println!("exported {}", out); + println!("exported {out}"); Ok(()) } ImageCmd::Import { file } => { @@ -330,7 +330,7 @@ pub(crate) fn cmd_image( ); }; let dst = format!("{dataset}/img/{name}"); - eprintln!("[voxel] importing {} -> {dst}", file); + eprintln!("[voxel] importing {file} -> {dst}"); let status = std::process::Command::new("bash") .arg("-c") .arg(format!("{decomp} | zfs recv {dst}")) diff --git a/voxel/src/main.rs b/voxel/src/main.rs index acdebbc..919c25f 100644 --- a/voxel/src/main.rs +++ b/voxel/src/main.rs @@ -31,6 +31,7 @@ mod cpbuild; mod image; mod imagebuild; mod isolated_external; +mod multicast; mod net; mod network; mod patch; @@ -125,7 +126,8 @@ enum Cmd { #[arg(long)] dry_run: bool, }, - /// Destroy the rack. + /// Destroy the rack. Run `voxel network multicast down` first when + /// host-side multicast plumbing is in use. Destroy, /// Open a serial console to a node (^q to exit). Serial { node: String }, @@ -199,6 +201,12 @@ enum Cmd { #[arg(long)] no_build: bool, + /// Plumb any missing host multicast route or router mirror instead of + /// refusing to start. This is equivalent to + /// `voxel network multicast up` for the groups this run uses. + #[arg(long)] + setup_mcast: bool, + /// Permit running with effective uid 0. Build artifacts and reports /// under the build root become root-owned, which later unprivileged /// runs may trip over. @@ -372,6 +380,15 @@ enum NetworkCmd { #[command(subcommand)] cmd: ExternalCmd, }, + /// Plumb host-sourced multicast into a running rack: a host route per + /// group, the transit router's `tc` mirror, and the link-layer membership + /// that lets its NIC accept the group. + /// + /// This works in either `[external]` mode. + Multicast { + #[command(subcommand)] + cmd: MulticastCmd, + }, } #[derive(Subcommand)] @@ -393,6 +410,48 @@ enum ExternalCmd { Check, } +#[derive(Subcommand)] +enum MulticastCmd { + /// Point each group's host route at `ce`, install the mirror, and pin the + /// router NIC's link-layer membership. + Up { + /// Group to plumb (pass the flag once per group). + /// + /// This accepts commtest's `GROUP[@SRC,...]` form, of which only the + /// address matters here. + /// + /// Defaults to the group `voxel commtest --traffic multicast` sends. + #[arg(long = "group", value_name = "GROUP", default_value = commtest::DEFAULT_MCAST_GROUP)] + groups: Vec, + /// Print the host and router commands instead of running them. + #[arg(long)] + dry_run: bool, + }, + /// Remove the mirror, the memberships, and the per-group host routes. + Down { + /// Group to tear down (pass the flag once per group). + /// + /// Same form as `up`. Defaults to everything this Falcon environment + /// has recorded or still owns on its router. Routes belonging to + /// another environment are left alone. + #[arg(long = "group", value_name = "GROUP")] + groups: Vec, + /// Print the host and router commands instead of running them. + #[arg(long)] + dry_run: bool, + }, + /// Assert the routes, mirror, and memberships are in place, listing + /// anything that's missing. + Check { + /// Group to assert (pass the flag once per group). + /// + /// Same form as `up`. Defaults to everything voxel has plumbed, + /// the same set a groupless `down` tears down. + #[arg(long = "group", value_name = "GROUP")] + groups: Vec, + }, +} + #[derive(Subcommand)] enum RackCmd { /// Swap a single component on the running rack at a ref, then restart it. @@ -584,8 +643,7 @@ enum TpCmd { fn config_text(path: &Utf8Path) -> anyhow::Result { if path.exists() { - Ok(fs::read_to_string(path) - .with_context(|| format!("read {}", path))?) + Ok(fs::read_to_string(path).with_context(|| format!("read {path}"))?) } else { Ok(VoxelConfig::default().to_toml()) } @@ -593,9 +651,17 @@ fn config_text(path: &Utf8Path) -> anyhow::Result { fn load_config(path: &Utf8Path) -> anyhow::Result { let text = config_text(path)?; - let cfg = VoxelConfig::from_toml(&text) - .with_context(|| format!("parse {}", path))?; + let mut cfg = VoxelConfig::from_toml(&text) + .with_context(|| format!("parse {path}"))?; cfg.topology.validate().map_err(|e| anyhow::anyhow!("{path}: {e}"))?; + // An unset image.cp follows the workspace's omicron pin, the image a + // commitless `voxel image create` bakes, so a repin and rebuild-relaunch + // cycle gets by without a config edit. An explicit cp still selects any + // image. + if cfg.image.cp.is_none() && !cpbuild::PINNED_OMICRON_REV.is_empty() { + cfg.image.cp = + Some(format!("voxel-cp-{}", cpbuild::PINNED_OMICRON_REV)); + } Ok(cfg) } @@ -737,7 +803,7 @@ fn anchor_workdir( && root.is_dir() { std::env::set_current_dir(&root) - .with_context(|| format!("chdir to workdir {}", root))?; + .with_context(|| format!("chdir to workdir {root}"))?; } Ok(()) } @@ -803,25 +869,31 @@ async fn main() -> Result<(), Error> { api, traffic, no_build, + setup_mcast, allow_root, args, - } => commtest::run( - &load_config(&config_path)?, - commtest::Options { - // clap rejects together with --source (conflicts_with). - source: match (source.as_deref(), reference.as_deref()) { - (Some(path), _) => commtest::Source::Local(path), - (None, Some(r)) => commtest::Source::Reference(r), - (None, None) => commtest::Source::Image, + } => { + commtest::run( + &load_config(&config_path)?, + &cli.name, + commtest::Options { + // clap rejects together with --source (conflicts_with). + source: match (source.as_deref(), reference.as_deref()) { + (Some(path), _) => commtest::Source::Local(path), + (None, Some(r)) => commtest::Source::Reference(r), + (None, None) => commtest::Source::Image, + }, + rack: *rack, + api_override: api.as_deref(), + traffic: *traffic, + no_build: *no_build, + setup_mcast: *setup_mcast, + allow_root: *allow_root, + passthrough: args, }, - rack: *rack, - api_override: api.as_deref(), - traffic: *traffic, - no_build: *no_build, - allow_root: *allow_root, - passthrough: args, - }, - ), + ) + .await + } Cmd::Config { cmd } => config_cmd::cmd_config(&config_path, cmd), Cmd::Image { cmd } => match cmd { ImageCmd::Patch { component, reference, image, out } => { @@ -933,6 +1005,20 @@ async fn main() -> Result<(), Error> { } } } + NetworkCmd::Multicast { cmd } => { + let cfg = load_config(&config_path)?; + match cmd { + MulticastCmd::Up { groups, dry_run } => { + multicast::up(&cfg, &cli.name, groups, *dry_run).await + } + MulticastCmd::Down { groups, dry_run } => { + multicast::down(&cfg, &cli.name, groups, *dry_run).await + } + MulticastCmd::Check { groups } => { + multicast::check(&cfg, &cli.name, groups).await + } + } + } }, Cmd::Rack { cmd } => match cmd { RackCmd::Patch { component, reference, list, dry_run } => { diff --git a/voxel/src/multicast.rs b/voxel/src/multicast.rs new file mode 100644 index 0000000..26cabcf --- /dev/null +++ b/voxel/src/multicast.rs @@ -0,0 +1,1913 @@ +//! Host-side plumbing that gets host-sourced multicast into a running rack. +//! +//! The edge and transit routers are plain Linux boxes running FRR for unicast +//! BGP, with no multicast routing daemon, so a host route carries a group's +//! frames as far as `ce` and no further. In place of a daemon, a stock iproute2 +//! `tc`/`mirred` ingress filter on the transit router mirrors each group from +//! the router's host-facing NIC onto its scrimlet-facing NICs, which is what +//! puts the frames on a switch. +//! +//! Two properties of that filter matter here. Every scrimlet is a mirror +//! target in a single filter per group, because external multicast ingresses +//! at whichever switch holds the group's external NAT entry, an election the +//! rack makes internally. The switches without the entry drop their copy, so +//! there is no duplicate replication. And the targets are chained actions +//! within one filter rather than one filter each: the first matching filter +//! ends flower classification, so a per-scrimlet filter would reach only the +//! first target unless every one of them carried an explicit `continue`, +//! whereas chained `mirred` actions all run under mirred's default `pipe` +//! control. +//! +//! The election is the only guard against a duplicate at this level. Viona's +//! ownership split between its classified and promiscuous receive callbacks +//! (viona_rx.c, stlouis#986) dedupes a different overlap, two local delivery +//! paths for one wire arrival, so a second copy forwarded by the other switch +//! would reach a guest as a second frame. +//! +//! The mirror only sees frames the router's NIC accepts, and a NIC accepts a +//! multicast group only after a join. Nothing on the router joins these groups +//! (FRR speaks no multicast protocol), so the flooded frames are dropped +//! before they reach the `tc` ingress hook. A static link-layer membership for +//! each group's Ethernet address (the RFC 1112 section 6.4 mapping) on the +//! host-facing NIC stands in for the join. Each membership `up` adds is +//! recorded under `/run` on the router, so `down` never removes one the +//! router already held, and the record dies with the router just as the +//! memberships do. This is voxel-only scaffolding for its mirror-based +//! emulated upstream. Customers do not add this Linux `ip maddress` entry to +//! a rack. Their upstream network must still deliver each group toward the +//! rack uplinks. +//! +//! The host route table lives in the global zone and is shared by Falcon +//! environments. Voxel records each environment's group and gateway under +//! `.falcon/`, so setup and teardown can leave routes owned by another +//! environment alone. +//! +//! This runs on the host unprivileged, escalating each mutating command +//! through `pfexec` (routes) or `ssh root@` (the router), the same path +//! `voxel commtest --setup-mcast` reuses before a run. +//! +//! See [RFC 1112 section 6.4](https://www.rfc-editor.org/rfc/rfc1112#section-6.4) +//! for the group-to-Ethernet mapping and +//! [RFC 7042 section 2.1.1](https://www.rfc-editor.org/rfc/rfc7042#section-2.1.1) +//! for the IANA OUI allotment behind it. + +use anyhow::{Context, bail, ensure}; +use itertools::Itertools; +use std::net::{IpAddr, Ipv4Addr}; +use std::path::PathBuf; +use std::process::Command; +use voxel_config::VoxelConfig; + +use crate::net::{ + ROUTE, RouteEntry, SshFailure, resolve_external_ip, route_entries, + serial_bounded, ssh_capture, ssh_output, ssh_try_capture, zlogin, +}; +use crate::network::SWADM; +use crate::topo::build_topo; +use crate::util::shell_quote; + +/// The lowest `tc` filter priority voxel will claim. A group keeps whichever +/// pref its filter already holds, so repeated `up` runs replace in place, and +/// new groups take the next free one above this rather than displacing anything +/// else attached to the same ingress. +const PREF_BASE: u32 = 100; + +/// The handle voxel gives every filter it installs. We make it explicit so +/// `replace` is idempotent (see `filter_cmd`) and, as part of `Filter::owned`, +/// so the filter a group resolves to is the same one `down`'s delete +/// addresses. +const HANDLE: u64 = 1; + +/// The action voxel installs, as tc renders it in JSON: kind `mirred`, action +/// `mirror`, direction `egress`. This is shared between the install command +/// (`filter_cmd`) and detection (`TcAction::egress_mirror`) so the two cannot +/// drift apart. +const MIRRED: &str = "mirred"; +const MIRROR: &str = "mirror"; +const EGRESS: &str = "egress"; +/// The netstat flag illumos uses for a host route. +const HOST_ROUTE_FLAG: char = 'H'; + +/// A host route installed for one Falcon environment. +#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)] +struct MulticastRoute { + group: Ipv4Addr, + gateway: String, +} + +/// Host-side multicast state persisted per Falcon environment. +#[derive(Debug, serde::Deserialize, serde::Serialize)] +struct MulticastState { + environment: String, + routes: Vec, +} + +impl MulticastState { + fn new(environment: &str) -> Self { + Self { environment: environment.to_string(), routes: Vec::new() } + } + + fn route(&self, group: &Ipv4Addr) -> Option<&MulticastRoute> { + self.routes.iter().find(|route| route.group == *group) + } + + fn groups(&self) -> impl Iterator + '_ { + self.routes.iter().map(|route| route.group) + } + + fn set_route(&mut self, group: Ipv4Addr, gateway: String) { + if let Some(route) = self.routes.iter_mut().find(|r| r.group == group) { + route.gateway = gateway; + } else { + self.routes.push(MulticastRoute { group, gateway }); + } + } + + fn remove_groups(&mut self, groups: &[Ipv4Addr]) { + self.routes.retain(|route| !groups.contains(&route.group)); + } +} + +/// The local state file is keyed by the Falcon environment name. Hex encoding +/// keeps arbitrary names out of the path while retaining one file per +/// environment. +fn multicast_state_path(name: &str) -> PathBuf { + use std::fmt::Write as _; + let mut key = String::with_capacity(name.len() * 2); + for byte in name.bytes() { + write!(&mut key, "{byte:02x}") + .expect("writing to a String cannot fail"); + } + PathBuf::from(".falcon").join(format!("multicast-{key}.json")) +} + +fn read_multicast_state(name: &str) -> anyhow::Result> { + let path = multicast_state_path(name); + let body = match std::fs::read_to_string(&path) { + Ok(body) => body, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(e) => { + return Err(e) + .with_context(|| format!("reading {}", path.display())); + } + }; + let state: MulticastState = serde_json::from_str(&body) + .with_context(|| format!("parsing {}", path.display()))?; + ensure!( + state.environment == name, + "{} belongs to Falcon environment '{}', not '{name}'", + path.display(), + state.environment, + ); + Ok(Some(state)) +} + +fn write_multicast_state( + name: &str, + state: &MulticastState, +) -> anyhow::Result<()> { + ensure!( + state.environment == name, + "multicast state environment '{}' does not match '{name}'", + state.environment, + ); + let path = multicast_state_path(name); + if state.routes.is_empty() { + match std::fs::remove_file(&path) { + Ok(()) => {} + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} + Err(e) => { + return Err(e) + .with_context(|| format!("removing {}", path.display())); + } + } + return Ok(()); + } + + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("creating {}", parent.display()))?; + } + let tmp = path.with_extension("json.tmp"); + let body = serde_json::to_vec_pretty(state) + .with_context(|| format!("serializing {}", path.display()))?; + std::fs::write(&tmp, body) + .with_context(|| format!("writing {}", tmp.display()))?; + // A failed rename leaves the old record intact, which is what makes the + // update atomic, but it also leaves the temporary behind to be mistaken + // for a record later. + if let Err(e) = std::fs::rename(&tmp, &path) { + let _ = std::fs::remove_file(&tmp); + return Err(e).with_context(|| { + format!("renaming {} to {}", tmp.display(), path.display()) + }); + } + Ok(()) +} + +/// A group as passed to commtest: the bare address, or `GROUP@SRC,...` for a +/// source-filtered join. Only the address matters to the route and the mirror +/// (the source list is a property of the join), so strip any suffix. +/// +/// TODO: IPv4 only, matching commtest's `validate_mcast`. Once that accepts +/// v6 groups, this file needs v6 throughout: the `33:33` membership mapping +/// (RFC 2464 section 7) in `group_mac` and `protocol ipv6` in `filter_cmd`. +/// +/// The switch side already takes external IPv6 groups, and none of this +/// waits on new addressing: multicast frames carry the group MAC, so the +/// route through `ce` only pins the egress interface, which for v6 can be +/// `ce`'s link-local once the host plumbs addrconf on the segment (or the +/// sender selects the interface itself via `IPV6_MULTICAST_IF`). +fn group_addr(group: &str) -> anyhow::Result { + let addr = group.split_once('@').map_or(group, |(addr, _)| addr); + let addr: Ipv4Addr = addr + .parse() + .with_context(|| format!("multicast group '{group}' must be IPv4"))?; + if !addr.is_multicast() { + bail!("'{addr}' is not a multicast address (224.0.0.0/4)"); + } + Ok(addr) +} + +/// The distinct addresses `groups` include. This is deduplicated because a +/// repeated `--group` would otherwise install a second filter for the same +/// group at another pref, which `down` then cannot fully remove. +fn group_addrs(groups: &[String]) -> anyhow::Result> { + groups + .iter() + .map(|g| group_addr(g)) + .process_results(|addrs| addrs.unique().collect()) +} + +/// The Ethernet address `group`'s frames carry on the wire: the group's +/// low-order 23 bits placed into `01:00:5e:00:00:00` (RFC 1112 section 6.4). +/// +/// The prefix is IANA's OUI (`00-00-5E`), of whose 2^24 multicast identifiers +/// only the lower half is allotted to IPv4 (RFC 7042 section 2.1.1), so a +/// 28-bit group address maps onto 23 bits and 32 groups alias each Ethernet +/// address. Teardown, therefore, drops a membership only when no remaining +/// group still maps to it. +/// +/// Note: Oxide's own OUI (`A8:40:25`) plays no part here. That prefix marks +/// Oxide-assigned unicast MACs, guest NICs included, while the multicast +/// mapping is protocol-defined and the same for every sender. +fn group_mac(addr: &Ipv4Addr) -> String { + let [_, b, c, d] = addr.octets(); + format!("01:00:5e:{:02x}:{:02x}:{:02x}", b & 0x7f, c, d) +} + +/// The distinct Ethernet addresses `addrs` map to, in order of first +/// occurrence. +fn group_macs(addrs: &[Ipv4Addr]) -> Vec { + addrs.iter().map(group_mac).unique().collect() +} + +/// Split the MACs `down` is tearing down into those voxel may delete and +/// those it must leave alone. A membership goes only when no staying group +/// still maps to it and the router-side record says `up` created it. +fn deletable_members( + addrs: &[Ipv4Addr], + staying: &[Ipv4Addr], + owned: &[String], +) -> (Vec, Vec) { + let keep = group_macs(staying); + group_macs(addrs) + .into_iter() + .filter(|m| !keep.contains(m)) + .partition(|m| owned.contains(m)) +} + +/// The router carrying the mirror: the first non-`ce` router, i.e. `cr1`. +/// +/// One mirror suffices: every router sits on the host-facing segment and sees +/// the same flood, so mirroring from a second one would only duplicate frames +/// into the same switches. +fn mirror_router(cfg: &VoxelConfig) -> anyhow::Result { + cfg.topology + .routers + .iter() + .find(|r| r.as_str() != "ce") + .cloned() + .context("topology.routers has no fabric router to mirror from") +} + +/// A node's external address as voxel assigned it. `None` outside isolated +/// mode, where addresses are leased and only discoverable from the running +/// node. +fn static_ip(cfg: &VoxelConfig, node: &str) -> Option { + cfg.external + .isolated() + .then(|| { + cfg.static_external_ips() + .into_iter() + .find_map(|(n, ip)| (n == node).then_some(ip)) + }) + .flatten() +} + +/// The mirror's shape, derived from config alone. +struct MirrorTarget { + /// The router carrying the mirror (see `mirror_router`). + router: String, + /// The host-facing NIC the filter attaches to. + iif: String, + /// The scrimlet-facing NICs the filter mirrors onto. + ifaces: Vec, +} + +impl MirrorTarget { + /// The router, ingress NIC, and mirror devices the filter commands + /// address. This is derived from config alone. The router's address is + /// resolved separately (`node_addr`), which is the only step that can + /// need the rack running. + fn new(cfg: &VoxelConfig) -> anyhow::Result { + let router = mirror_router(cfg)?; + let ifaces = cfg.router_scrimlet_ifaces(&router); + if ifaces.is_empty() { + bail!("{router} has no scrimlet-facing NIC to mirror to"); + } + let iif = cfg.router_ext_iface(&router); + Ok(Self { router, iif, ifaces }) + } +} + +/// A router's external address in either mode: isolated mode's static +/// assignment comes from config alone; otherwise, the node's DHCP lease is +/// read over the falcon console under `serial_bounded`'s two-stage deadline. +async fn node_addr( + cfg: &VoxelConfig, + name: &str, + node: &str, +) -> anyhow::Result { + if let Some(ip) = static_ip(cfg, node) { + return Ok(ip); + } + let topo = build_topo(cfg, name)?; + let n = topo + .node_ref(node) + .with_context(|| format!("{node} is not in the topology"))?; + serial_bounded( + &format!("reading {node}'s address"), + resolve_external_ip(cfg, &topo.runner, node, n, true), + ) + .await + .with_context(|| { + format!("cannot resolve {node}'s external address (is the rack up?)") + }) +} + +/// `ce`'s external address, the nexthop every group's host route points at. An +/// explicit `[topology] ce_external_ip` or isolated mode's static numbering +/// resolves without touching the guest. Otherwise `ce`'s lease is read from the +/// running node. +async fn ce_nexthop(cfg: &VoxelConfig, name: &str) -> anyhow::Result { + match crate::net::ce_static_ip(cfg) { + Some(ip) => Ok(ip), + None => node_addr(cfg, name, "ce").await, + } +} + +/// Run a command on the mirror router, or print it under `--dry-run`. Returns +/// its stdout and fails when ssh cannot reach the router or the command exits +/// non-zero. +fn router_run(ip: &str, cmd: &str, dry_run: bool) -> anyhow::Result { + if dry_run { + eprintln!("+ ssh root@{ip} {cmd}"); + return Ok(String::new()); + } + ssh_capture(ip, cmd).with_context(|| { + format!( + "`{cmd}` on {ip} (is the rack up and its external NIC addressed?)" + ) + }) +} + +/// Run a command on the mirror router, tolerating a non-zero exit. For the +/// deletes, which fail benignly when there is nothing to delete. +fn router_try(ip: &str, cmd: &str, dry_run: bool) { + if dry_run { + eprintln!("+ ssh root@{ip} {cmd}"); + return; + } + let _ = ssh_output(ip, cmd); +} + +/// Run a host command under pfexec, or print it under `--dry-run`. +/// +/// illumos `route` exits non-zero even on a successful add, so the status is +/// not checked here. `up` and `down` re-read the table instead. +fn host_route(args: &[&str], dry_run: bool) { + if dry_run { + eprintln!("+ pfexec route {}", args.join(" ")); + return; + } + let _ = Command::new("pfexec").arg(ROUTE).args(args).output(); +} + +/// The host-route gateways currently listed for `group`. +fn route_gateways(entries: &[RouteEntry], group: &Ipv4Addr) -> Vec { + let group = group.to_string(); + entries + .iter() + .filter(|entry| { + entry.dest == group + && is_host_route(entry) + && !entry.gateway.is_empty() + }) + .map(|entry| entry.gateway.clone()) + .unique() + .collect() +} + +/// Whether a netstat route entry is a host route. Illumos prints route flags +/// as a compact set, such as `UGH`, rather than as a single enum value. +fn is_host_route(entry: &RouteEntry) -> bool { + entry.flags.contains(HOST_ROUTE_FLAG) +} + +/// Drop only the host routes whose gateways this environment owns. +fn purge_route( + group: &Ipv4Addr, + entries: &[RouteEntry], + owned_gateways: &[String], + dry_run: bool, +) { + let group_text = group.to_string(); + for gateway in route_gateways(entries, group) + .into_iter() + .filter(|gateway| owned_gateways.contains(gateway)) + { + host_route(&["delete", "-host", &group_text, &gateway], dry_run); + } +} + +/// The host routes belonging to `state` that remain for `addrs`. +fn remaining_host_routes( + entries: &[RouteEntry], + addrs: &[Ipv4Addr], + state: &MulticastState, +) -> Vec { + addrs + .iter() + .filter_map(|addr| { + let owned = state.route(addr)?; + entries + .iter() + .find(|entry| { + entry.dest == addr.to_string() + && is_host_route(entry) + && entry.gateway == owned.gateway + }) + .map(|entry| format!("host route {addr} -> {}", entry.gateway)) + }) + .collect() +} + +/// Host routes for `addrs` that do not belong to `state`. +fn foreign_host_routes( + entries: &[RouteEntry], + addrs: &[Ipv4Addr], + state: Option<&MulticastState>, +) -> Vec { + addrs + .iter() + .flat_map(|addr| { + let owned = state + .and_then(|state| state.route(addr)) + .map(|route| route.gateway.as_str()); + entries + .iter() + .filter(move |entry| { + entry.dest == addr.to_string() + && is_host_route(entry) + && Some(entry.gateway.as_str()) != owned + }) + .map(move |entry| { + format!("host route {addr} -> {}", entry.gateway) + }) + }) + .collect() +} + +/// Name the host routes teardown is leaving in place, so a group that looks +/// torn down (but still resolves) is accounted for rather than silently skipped. +fn report_foreign_routes( + addrs: &[Ipv4Addr], + state: Option<&MulticastState>, + name: &str, +) { + for route in foreign_host_routes(&route_entries(), addrs, state) { + eprintln!( + "[voxel] multicast: leaving {route}; it is not owned by Falcon \ + environment '{name}'" + ); + } +} + +/// The groups recorded for a Falcon environment. This is what a groupless +/// `down` or `check` starts from rather than a scan of the host route table: +/// host routes are the one piece of the plumbing that outlives +/// `voxel destroy`, and the table is shared with every other environment, so +/// only this record says which of them are this environment's. +fn state_groups(state: Option<&MulticastState>) -> Vec { + state.map(|state| state.groups().collect()).unwrap_or_default() +} + +/// Remove the host routes recorded for a Falcon environment. +fn purge_state_routes( + addrs: &[Ipv4Addr], + state: Option<&MulticastState>, + dry_run: bool, +) { + let Some(state) = state else { + return; + }; + let entries = route_entries(); + for addr in addrs { + let Some(route) = state.route(addr) else { + continue; + }; + purge_route( + addr, + &entries, + std::slice::from_ref(&route.gateway), + dry_run, + ); + } +} + +/// Whether the host route for `group` currently resolves to `nexthop`. +fn route_ok(group: &Ipv4Addr, nexthop: &str) -> bool { + Command::new(ROUTE) + .args(["-n", "get", &group.to_string()]) + .output() + .map(|o| gateway_matches(&String::from_utf8_lossy(&o.stdout), nexthop)) + .unwrap_or(false) +} + +/// Whether `route -n get` named `nexthop` as the gateway. This reads the +/// `gateway:` field rather than the whole output, so a nexthop that is a +/// prefix of another address on the segment cannot match by accident. +fn gateway_matches(out: &str, nexthop: &str) -> bool { + out.lines() + .filter_map(|l| l.trim().strip_prefix("gateway:")) + .any(|gw| gw.trim() == nexthop) +} + +/// An installed ingress filter, including the group it matches and the +/// devices its egress-mirror actions target. +struct Filter { + pref: u32, + handle: Option, + kind: String, + protocol: String, + dst: Option, + mirrors: Vec, +} + +impl Filter { + /// Whether this filter is one voxel installs: flower, protocol ip, at + /// `HANDLE`, and in the pref range `free_pref` allocates from. `up` and + /// `down` only replace or delete filters passing this, so a pre-existing + /// filter that happens to match a group's address is left alone. + fn owned(&self) -> bool { + self.kind == "flower" + && self.protocol == "ip" + && self.handle == Some(HANDLE) + && self.pref >= PREF_BASE + } +} + +/// One entry of `tc -json filter show`, restricted to the fields voxel reads. +/// Every field is optional because tc mixes real filters with per-pref summary +/// entries that carry no `options`. Only `pref` is required of an entry: one +/// missing `kind` or `protocol` still counts in `free_pref`'s accounting, +/// while `Filter::owned` rejects it. +#[derive(serde::Deserialize)] +struct TcFilter { + pref: Option, + kind: Option, + protocol: Option, + options: Option, +} + +impl TcFilter { + /// Convert a `tc` entry into the subset of filter state voxel uses. + fn into_filter(self) -> Option { + let Self { pref, kind, protocol, options } = self; + let pref = pref?; + let (handle, dst, mirrors) = options.map_or( + (None, None, Vec::new()), + |TcOptions { handle, keys, actions }| { + ( + handle, + keys.and_then(|keys| keys.dst_ip), + actions + .into_iter() + .filter(TcAction::egress_mirror) + .filter_map(|action| action.to_dev) + .collect(), + ) + }, + ); + Some(Filter { + pref, + kind: kind.unwrap_or_default(), + protocol: protocol.unwrap_or_default(), + handle, + dst, + mirrors, + }) + } +} + +/// A handle other filter kinds may print as a string where flower prints a +/// number (compare the "ffff:" of `tc -json qdisc show`). Any non-numeric +/// form reads as `None`, keeping the entry rather than failing the read; +/// `Filter::owned` requires the numeric `HANDLE` anyway. +fn de_handle<'de, D>(d: D) -> Result, D::Error> +where + D: serde::Deserializer<'de>, +{ + use serde::Deserialize; + Ok(Option::::deserialize(d)?.and_then(|v| v.as_u64())) +} + +/// The `options` object of a `tc` filter entry. +#[derive(serde::Deserialize)] +struct TcOptions { + #[serde(default, deserialize_with = "de_handle")] + handle: Option, + keys: Option, + #[serde(default)] + actions: Vec, +} + +/// The match keys of a `tc` filter entry. Flower (a pun on "flow-er") is +/// tc's classifier that matches on named packet-header fields rather than +/// raw byte offsets; `dst_ip` is the only key voxel's filters set. See +/// tc-flower(8). +#[derive(serde::Deserialize)] +struct TcKeys { + dst_ip: Option, +} + +/// One action of a `tc` filter entry. +#[derive(serde::Deserialize)] +struct TcAction { + kind: Option, + mirred_action: Option, + direction: Option, + to_dev: Option, +} + +impl TcAction { + /// Whether this action mirrors to a device's egress. A redirect or an + /// ingress action must not count as part of an installed mirror. + fn egress_mirror(&self) -> bool { + self.kind.as_deref() == Some(MIRRED) + && self.mirred_action.as_deref() == Some(MIRROR) + && self.direction.as_deref() == Some(EGRESS) + } +} + +/// Read the router's filters from `tc -json`, rather than scraping the +/// human-readable rendering. Anything but a JSON array is an error, so a +/// garbled read cannot pass for an empty ingress. An entry that does not fit +/// `TcFilter`'s shape (a foreign kind whose fields use other types) is +/// skipped on its own rather than failing the whole read. +/// +/// The JSON keeps a filter's match under `options.keys` and each action's +/// kind, direction, and target as separate fields, so a redirect or an ingress +/// action cannot be mistaken for an egress mirror. Filters that match no +/// address (the per-pref summary entries tc emits) carry `dst: None`. +fn parse_filters(json: &str) -> anyhow::Result> { + let entries: Vec = + serde_json::from_str(json).context("parsing `tc -json filter show`")?; + Ok(entries + .into_iter() + .filter_map(|e| serde_json::from_value::(e).ok()) + .filter_map(TcFilter::into_filter) + .collect()) +} + +/// The pref holding `group`'s filter, if voxel has one installed. This lets +/// `up` and `down` address a single group without disturbing filters +/// belonging to others, including a foreign filter matching the same address +/// (see `Filter::owned`). +fn group_pref(filters: &[Filter], group: &Ipv4Addr) -> Option { + let group = group.to_string(); + filters + .iter() + .find(|f| f.owned() && f.dst.as_deref() == Some(group.as_str())) + .map(|f| f.pref) +} + +/// The multicast groups represented by filters voxel owns. +fn owned_group_addrs( + filters: &[Filter], +) -> impl Iterator + '_ { + filters + .iter() + .filter(|filter| filter.owned()) + .filter_map(|filter| filter.dst.as_deref()?.parse().ok()) +} + +/// The lowest pref at or above `PREF_BASE` that is neither installed nor +/// already claimed by this run. +fn free_pref(filters: &[Filter], taken: &mut Vec) -> u32 { + let pref = (PREF_BASE..) + .find(|p| !filters.iter().any(|f| f.pref == *p) && !taken.contains(p)) + .expect("u32 range is not exhausted"); + taken.push(pref); + pref +} + +/// Whether `group`'s own filter mirrors to every device in `ifaces`. Scoped to +/// that filter, since devices belonging to a different group would otherwise +/// let a partial install pass. +fn mirror_installed( + filters: &[Filter], + group: &Ipv4Addr, + ifaces: &[String], +) -> bool { + let group = group.to_string(); + filters + .iter() + .filter(|f| f.owned() && f.dst.as_deref() == Some(group.as_str())) + .any(|f| ifaces.iter().all(|dev| f.mirrors.contains(dev))) +} + +/// One interface of `ip -json maddress show`, restricted to the fields voxel +/// reads. +#[derive(serde::Deserialize)] +struct MaddrIface { + #[serde(default)] + maddr: Vec, +} + +/// One membership entry. +/// +/// Link-layer entries carry `link`, and the inet entries this reader skips +/// carry `family` and `address` instead. +#[derive(serde::Deserialize)] +struct MaddrEntry { + link: Option, +} + +/// Read a NIC's link-layer memberships from `ip -json maddress show`, one +/// Ethernet address per entry. As with `parse_filters`, anything but a JSON +/// array is an error, so a garbled read cannot pass for a bare NIC. +fn parse_members(json: &str) -> anyhow::Result> { + let ifaces: Vec = serde_json::from_str(json) + .context("parsing `ip -json maddress show`")?; + Ok(ifaces + .into_iter() + .flat_map(|i| i.maddr) + .filter_map(|m| m.link) + .collect()) +} + +/// Marker error: the ssh transport could not reach the router. +/// +/// `down` and the groupless `check` discovery treat only this failure as the +/// destroyed-rack case. A command that ran and failed propagates, so a broken +/// read cannot pass for a clean teardown or an empty rack. +#[derive(Debug)] +struct RouterUnreachable(String); + +impl std::fmt::Display for RouterUnreachable { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "router {} unreachable over ssh", self.0) + } +} + +impl std::error::Error for RouterUnreachable {} + +/// Whether `err`'s chain contains [`RouterUnreachable`]. +fn is_unreachable(err: &anyhow::Error) -> bool { + err.chain().any(|c| c.downcast_ref::().is_some()) +} + +/// Wrap an [`SshFailure`] for `cmd` on `ip`, keeping the unreachable case +/// downcastable via [`is_unreachable`]. +fn router_read_err(ip: &str, cmd: &str, e: SshFailure) -> anyhow::Error { + match e { + SshFailure::Unreachable => { + anyhow::Error::new(RouterUnreachable(ip.to_string())) + } + SshFailure::Failed(text) => { + anyhow::anyhow!("`{cmd}` on {ip} failed: {}", text.trim()) + } + } +} + +/// The router NIC's current link-layer memberships. +fn router_members(ip: &str, iif: &str) -> anyhow::Result> { + let cmd = format!("ip -json maddress show dev {iif}"); + match ssh_try_capture(ip, &cmd) { + Ok(out) => { + parse_members(&out).with_context(|| format!("`{cmd}` on {ip}")) + } + Err(e) => Err(router_read_err(ip, &cmd, e)), + } +} + +/// The router-side record of the memberships `up` added to `iif`, one +/// Ethernet address per line. Presence on the NIC alone cannot prove +/// ownership: the kernel maps 224.0.0.1 onto every interface, a join inside +/// the router adds its group's address, and 32 groups alias each one (see +/// `group_mac`). +/// +/// This is kept under `/run` so the record dies with the router or +/// its reboot, exactly when the memberships themselves do, and a stale +/// record can never claim a membership on a later rack. +fn record_file(iif: &str) -> String { + format!("/run/voxel-mcast-members.{iif}") +} + +/// The memberships the record names as voxel's own. A missing record reads +/// as no memberships, which fails safe: `down` then leaves them alone. A +/// failed read is an error, so it cannot pass for an empty record and let a +/// teardown quietly skip the memberships. +fn recorded_members(ip: &str, iif: &str) -> anyhow::Result> { + let record = shell_quote(&record_file(iif)); + let cmd = format!("if [ -e {record} ]; then cat {record}; fi"); + let out = + ssh_try_capture(ip, &cmd).map_err(|e| router_read_err(ip, &cmd, e))?; + Ok(out.lines().map(str::to_string).collect()) +} + +/// Add a membership and record its ownership, rolling back the membership if +/// the record cannot be updated. +fn add_recorded_membership( + ip: &str, + iif: &str, + mac: &str, + dry_run: bool, +) -> anyhow::Result<()> { + let mac_q = shell_quote(mac); + let iif_q = shell_quote(iif); + let record = shell_quote(&record_file(iif)); + let tmp = shell_quote(&format!("{}.tmp", record_file(iif))); + let add = format!("ip maddress add {mac_q} dev {iif_q}"); + router_run(ip, &add, dry_run).map(drop)?; + + // Use a temporary file so a failed append cannot leave a partial record. + // A missing record is the only non-match that should create a new one. + let record_cmd = format!( + "set -e; \ + if [ -e {record} ]; then \ + if grep -qxF {mac_q} {record}; then \ + exit 0; \ + else \ + status=$?; \ + [ \"$status\" -eq 1 ] || exit \"$status\"; \ + fi; \ + cat {record} > {tmp}; \ + else \ + : > {tmp}; \ + fi; \ + printf '%s\\n' {mac_q} >> {tmp}; \ + mv {tmp} {record}" + ); + if let Err(record_err) = router_run(ip, &record_cmd, dry_run).map(drop) { + let rollback = format!("ip maddress del {mac_q} dev {iif_q}"); + return match router_run(ip, &rollback, false).map(drop) { + Ok(()) => Err(anyhow::anyhow!( + "recording membership {mac} on {iif} failed, \ + membership rolled back: {record_err:#}" + )), + Err(rollback_err) => Err(anyhow::anyhow!( + "recording membership {mac} on {iif} failed: {record_err:#}; \ + rolling back the membership also failed: {rollback_err:#}" + )), + }; + } + Ok(()) +} + +/// Remove a membership from the ownership record with an atomic replacement. +/// The record disappears with its last entry, keeping `down` free of leftover +/// state on the router. +fn remove_recorded_membership( + ip: &str, + iif: &str, + mac: &str, +) -> anyhow::Result<()> { + let mac_q = shell_quote(mac); + let record_path = record_file(iif); + let record = shell_quote(&record_path); + let tmp = shell_quote(&format!("{record_path}.tmp")); + let cmd = format!( + "set -e; \ + if [ -e {record} ]; then \ + if grep -vxF {mac_q} {record} > {tmp}; then \ + mv {tmp} {record}; \ + else \ + status=$?; \ + if [ \"$status\" -eq 1 ]; then \ + rm -f {tmp} {record}; \ + else \ + rm -f {tmp}; \ + exit \"$status\"; \ + fi; \ + fi; \ + fi" + ); + router_run(ip, &cmd, false) + .map(drop) + .with_context(|| format!("removing {mac} from {record_path}")) +} + +/// The router's current ingress filters. +/// +/// This reads even under `--dry-run`, which mutates nothing, so a preview +/// reports the prefs the real run would touch. A preview tolerates an +/// unreachable router. +fn show_filters( + ip: &str, + iif: &str, + dry_run: bool, +) -> anyhow::Result> { + let cmd = format!("tc -json filter show dev {iif} ingress"); + match ssh_try_capture(ip, &cmd) { + Ok(out) => { + parse_filters(&out).with_context(|| format!("`{cmd}` on {ip}")) + } + Err(SshFailure::Unreachable) if dry_run => Ok(Vec::new()), + Err(e) => Err(router_read_err(ip, &cmd, e)), + } +} + +/// The `tc` command installing a group's single filter: match the group's +/// address, then mirror to every scrimlet NIC through chained actions (see +/// the module doc for why the actions must chain). +fn filter_cmd( + iif: &str, + group: &Ipv4Addr, + pref: u32, + ifaces: &[String], +) -> String { + // The handle has to be explicit for `replace` to be idempotent. Left at 0, + // the kernel treats the request as a fresh insert, and flower rejects a + // second filter carrying a key it already holds with EEXIST rather than + // overwriting the first. Handles are scoped to a (pref, protocol), and this + // installs one filter per pref, so HANDLE is always the one to replace. + let mirrors: String = ifaces + .iter() + .map(|dev| format!(" action {MIRRED} {EGRESS} {MIRROR} dev {dev}")) + .collect(); + format!( + "tc filter replace dev {iif} ingress handle {HANDLE} pref {pref} protocol ip flower \ + dst_ip {group}{mirrors}" + ) +} + +/// Point each group's host route at `ce`, install the mirror, and pin the +/// link-layer membership that lets the router accept each group. Safe to +/// re-run because this environment's routes are deleted before being re-added, +/// `tc filter replace` is idempotent and the membership add is guarded by a +/// lookup, with each add recorded on the router so `down` removes only +/// memberships voxel itself created. A route already owned by another Falcon +/// environment is rejected. +/// +/// # Errors +/// +/// Fails when a group is not an IPv4 multicast address, when `ce`'s or the +/// router's external address cannot be resolved, when the router is +/// unreachable, or when a host route does not take. +pub(crate) async fn up( + cfg: &VoxelConfig, + name: &str, + groups: &[String], + dry_run: bool, +) -> anyhow::Result<()> { + let addrs = group_addrs(groups)?; + let nexthop = ce_nexthop(cfg, name).await?; + let MirrorTarget { router, iif, ifaces } = MirrorTarget::new(cfg)?; + let router_ip = node_addr(cfg, name, &router).await?; + let mut state = read_multicast_state(name)? + .unwrap_or_else(|| MulticastState::new(name)); + + eprintln!( + "[voxel] multicast: {} group(s) via ce {nexthop}, mirroring {iif} -> {}", + addrs.len(), + ifaces.join(" ") + ); + + addrs.iter().try_for_each(|addr| { + let group = addr.to_string(); + let entries = route_entries(); + let mut owned_gateways = vec![nexthop.clone()]; + if let Some(route) = state.route(addr) + && !owned_gateways.contains(&route.gateway) + { + owned_gateways.push(route.gateway.clone()); + } + let foreign: Vec = route_gateways(&entries, addr) + .into_iter() + .filter(|gateway| !owned_gateways.contains(gateway)) + .collect(); + + // The record, not the gateway, is what makes a route this + // environment's. An unrecorded one may belong to another environment + // or be a leftover whose record was removed, and the two are + // indistinguishable from here, so neither is swept. + ensure!( + foreign.is_empty(), + "host route {group} is not recorded for Falcon environment \ + '{name}' (gateway {})", + foreign.join(", ") + ); + // Record ownership before the route exists. A crash between the two + // then leaves a record with no route, which `down` reads as nothing + // to delete and the next `up` overwrites. The reverse ordering leaves + // a route with no record, which nothing afterwards can prove is ours, + // so a groupless `down` would have to leave it behind. The gateways + // eligible for sweeping were captured above, so overwriting the entry + // here does not lose the prior one. + if !dry_run { + state.set_route(*addr, nexthop.clone()); + write_multicast_state(name, &state).with_context(|| { + format!("recording host route {group} for Falcon '{name}'") + })?; + } + purge_route(addr, &entries, &owned_gateways, dry_run); + host_route(&["add", "-host", &group, &nexthop], dry_run); + ensure!( + dry_run || route_ok(addr, &nexthop), + "host route {group} -> {nexthop} did not take" + ); + Ok(()) + })?; + + // Add the qdisc rather than recreating it, and take a pref per group rather + // than the whole ingress: `up --group A` then `up --group B` has to leave A + // working, and anything else attached here is not voxel's to delete. + router_try(&router_ip, &format!("tc qdisc add dev {iif} clsact"), dry_run); + let installed = show_filters(&router_ip, &iif, dry_run)?; + addrs + .iter() + .scan(Vec::new(), |taken, addr| { + Some(( + addr, + group_pref(&installed, addr) + .unwrap_or_else(|| free_pref(&installed, taken)), + )) + }) + .try_for_each(|(addr, pref)| { + router_run( + &router_ip, + &filter_cmd(&iif, addr, pref, &ifaces), + dry_run, + ) + .map(drop) + })?; + + // The membership that lets the NIC accept each group's frames (see the + // module doc). This is added only when absent, because `ip maddress add` + // stacks reference counts and one `down` has to undo any number of `up` + // runs. + // + // Each successful add lands in the router-side record, so `down` can tell + // voxel's memberships from ones the router already held. + let members = router_members(&router_ip, &iif)?; + let owned = recorded_members(&router_ip, &iif)?; + let (present, absent): (Vec<_>, Vec<_>) = + group_macs(&addrs).into_iter().partition(|mac| members.contains(mac)); + for mac in present.iter().filter(|mac| !owned.contains(mac)) { + eprintln!( + "[voxel] multicast: membership {mac} on {iif} predates \ + voxel, leaving it to its owner" + ); + } + absent.iter().try_for_each(|mac| { + add_recorded_membership(&router_ip, &iif, mac, dry_run) + })?; + if !dry_run { + eprintln!("[voxel] multicast: up"); + } + Ok(()) +} + +/// Remove each group's mirror filter, link-layer membership, and host routes. +/// Filters and memberships for other groups, memberships voxel did not +/// create, the `clsact` qdisc itself, the rack's own pool route, and the +/// external segment are all left alone. +/// +/// With no groups given, this tears down everything this Falcon environment has +/// recorded or still owns on its router, so a groupless `down` undoes any +/// sequence of `up` runs without scanning another environment's host routes. +/// +/// # Errors +/// +/// Fails when a group is not an IPv4 multicast address, the router address +/// cannot be resolved, or anything survives the teardown. Deleting something +/// already absent is not an error, and neither is an unreachable router after +/// the host routes have been removed. +pub(crate) async fn down( + cfg: &VoxelConfig, + name: &str, + groups: &[String], + dry_run: bool, +) -> anyhow::Result<()> { + let explicit = !groups.is_empty(); + let mut state = read_multicast_state(name)?; + let mut addrs = if explicit { + group_addrs(groups)? + } else { + state_groups(state.as_ref()) + }; + eprintln!( + "[voxel] multicast: removing mirror filters, memberships, and host routes" + ); + + // Host routes go first. They are the one piece that outlives + // `voxel destroy`, so they must be removed before attempting to inspect + // the router. Only routes recorded for this environment are eligible. + purge_state_routes(&addrs, state.as_ref(), dry_run); + + let MirrorTarget { router, iif, .. } = MirrorTarget::new(cfg)?; + + // Address discovery uses the serial console in LAN mode. Its failure does + // not establish that the rack was destroyed, so propagate it instead of + // reporting a successful teardown that may have left router state behind. + let router_ip = match node_addr(cfg, name, &router).await { + Ok(ip) => ip, + Err(e) if dry_run => { + eprintln!( + "[voxel] multicast: cannot resolve {router} ({e:#}); \ + skipping router preview" + ); + return Ok(()); + } + Err(e) => return Err(e), + }; + let installed = match show_filters(&router_ip, &iif, dry_run) { + Ok(filters) => filters, + Err(e) if is_unreachable(&e) => { + if !dry_run { + report_foreign_routes(&addrs, state.as_ref(), name); + if let Some(state) = state.as_ref() { + let remaining = + remaining_host_routes(&route_entries(), &addrs, state); + ensure!( + remaining.is_empty(), + "still plumbed after teardown:\n {}", + remaining.join("\n ") + ); + } + if let Some(state) = state.as_mut() { + state.remove_groups(&addrs); + write_multicast_state(name, state)?; + } + } + eprintln!( + "[voxel] multicast: {e:#}: if the rack was destroyed, the \ + mirror and memberships went with it" + ); + return Ok(()); + } + Err(e) => return Err(e), + }; + + // Without an explicit group list, the router's owned filters complete + // the set: a group can keep a filter and membership after its host route + // is gone, and the routes those groups may still hold go the same way as + // the rest. + if !explicit { + let discovered: Vec = owned_group_addrs(&installed) + .filter(|d| !addrs.contains(d)) + .collect(); + for d in discovered { + addrs.push(d); + } + } + report_foreign_routes(&addrs, state.as_ref(), name); + + // The install's full identifier tuple. A bare `pref` wildcards protocol + // and handle and would take any other classifier sharing the priority + // with it. + addrs + .iter() + .filter_map(|addr| group_pref(&installed, addr)) + .for_each(|pref| { + router_try( + &router_ip, + &format!( + "tc filter del dev {iif} ingress handle {HANDLE} pref {pref} protocol ip flower" + ), + dry_run, + ) + }); + + // Groups alias Ethernet addresses (see `group_mac`), so a membership goes + // only when no group staying behind still maps to it, and only when the + // router-side record says `up` created it. One that predates voxel (the + // kernel's all-hosts mapping, or a join inside the router) stays with its + // owner. + let staying: Vec = installed + .iter() + .filter_map(|f| f.dst.as_deref()?.parse().ok()) + .filter(|d| !addrs.contains(d)) + .collect(); + let owned = recorded_members(&router_ip, &iif)?; + let (dropped, foreign) = deletable_members(&addrs, &staying, &owned); + if !foreign.is_empty() { + let present = router_members(&router_ip, &iif)?; + for mac in foreign.iter().filter(|m| present.contains(*m)) { + eprintln!( + "[voxel] multicast: membership {mac} on {iif} was not added \ + by voxel, leaving it" + ); + } + } + for mac in &dropped { + router_try( + &router_ip, + &format!("ip maddress del {mac} dev {iif}"), + dry_run, + ); + } + + if dry_run { + return Ok(()); + } + + // `router_try` swallows exit status, so a delete that failed for a real + // reason is indistinguishable from one that had nothing to remove. + // Re-read the state rather than reporting success blind. + let left = show_filters(&router_ip, &iif, false)?; + let members = router_members(&router_ip, &iif)?; + let mut remaining: Vec = addrs + .iter() + .flat_map(|addr| { + group_pref(&left, addr) + .map(|_| format!("mirror of {addr} on {iif}")) + .into_iter() + }) + .chain( + dropped + .iter() + .filter(|mac| members.contains(*mac)) + .map(|mac| format!("membership {mac} on {iif}")), + ) + .collect(); + if let Some(state) = state.as_ref() { + remaining.extend(remaining_host_routes( + &route_entries(), + &addrs, + state, + )); + } + + ensure!( + remaining.is_empty(), + "still plumbed after teardown:\n {}", + remaining.join("\n ") + ); + if let Some(state) = state.as_mut() { + state.remove_groups(&addrs); + write_multicast_state(name, state)?; + } + + // With the deletes verified, retire their entries from the record. An + // atomic replace rather than an in-place edit preserves the old record + // when the update fails, and the error tells the caller that teardown is + // incomplete. + for mac in &dropped { + remove_recorded_membership(&router_ip, &iif, mac)?; + } + + let owned = recorded_members(&router_ip, &iif)?; + let remaining: Vec = + dropped.iter().filter(|mac| owned.contains(*mac)).cloned().collect(); + ensure!( + remaining.is_empty(), + "ownership record still contains: {}", + remaining.join(", ") + ); + Ok(()) +} + +/// Every piece of plumbing for `groups`, each paired with whether it is in +/// place: the per-group host route, the router mirror, and the link-layer +/// membership. +/// +/// One walk serves both consumers: [`check`] prints every item, and +/// [`missing_plumbing`] keeps only the absent ones. +pub(crate) async fn plumbing_status( + cfg: &VoxelConfig, + name: &str, + groups: &[String], +) -> anyhow::Result> { + let addrs = group_addrs(groups)?; + let nexthop = ce_nexthop(cfg, name).await?; + let MirrorTarget { router, iif, ifaces } = MirrorTarget::new(cfg)?; + let router_ip = node_addr(cfg, name, &router).await?; + + let mut out: Vec<(String, bool)> = addrs + .iter() + .map(|addr| { + ( + format!("host route {addr} -> {nexthop}"), + route_ok(addr, &nexthop), + ) + }) + .collect(); + let filters = match show_filters(&router_ip, &iif, false) { + Ok(filters) => filters, + // `check` reports rather than aborts, so a router that cannot be read + // is one more missing item, with the reason attached. + Err(e) => { + out.push((format!("mirror on {iif}: {e:#}"), false)); + return Ok(out); + } + }; + out.extend(addrs.iter().map(|addr| { + ( + format!("mirror of {addr} on {iif} -> {}", ifaces.join(" ")), + mirror_installed(&filters, addr, &ifaces), + ) + })); + match router_members(&router_ip, &iif) { + Ok(members) => out.extend(addrs.iter().map(|addr| { + let mac = group_mac(addr); + let present = members.contains(&mac); + (format!("membership {mac} ({addr}) on {iif}"), present) + })), + // Report rather than abort here too. + Err(e) => out.push((format!("memberships on {iif}: {e:#}"), false)), + } + Ok(out) +} + +/// The pieces of plumbing missing for `groups`, as human-readable items. +/// Empty here means the path is complete. +/// +/// This is the `commtest` preflight's view, so that a multicast run reports +/// the missing plumbing instead of a delivery failure. +pub(crate) async fn missing_plumbing( + cfg: &VoxelConfig, + name: &str, + groups: &[String], +) -> anyhow::Result> { + Ok(plumbing_status(cfg, name, groups) + .await? + .into_iter() + .filter_map(|(item, ok)| (!ok).then_some(item)) + .collect()) +} + +/// The groups this Falcon environment has plumbed, discovered from its +/// persisted host routes plus the selected router's owned mirror filters. An +/// unreachable router is the destroyed-rack case and contributes nothing. +/// Address-discovery failures and commands that ran and failed propagate, so a +/// broken read cannot hide router-only state. +/// Empty means no multicast is set up. +async fn plumbed_groups( + cfg: &VoxelConfig, + name: &str, +) -> anyhow::Result> { + let state = read_multicast_state(name)?; + let mut addrs = state_groups(state.as_ref()); + let MirrorTarget { router, iif, .. } = MirrorTarget::new(cfg)?; + let ip = node_addr(cfg, name, &router).await?; + let router_groups: Vec = match show_filters(&ip, &iif, false) { + Ok(filters) => owned_group_addrs(&filters).collect(), + Err(e) if is_unreachable(&e) => { + eprintln!("[voxel] multicast: {e:#}: counting host routes only"); + Vec::new() + } + Err(e) => return Err(e), + }; + for d in router_groups { + if !addrs.contains(&d) { + addrs.push(d); + } + } + Ok(addrs.iter().map(ToString::to_string).collect()) +} + +/// The external groups a switch has programmed, each with its NAT target, +/// parsed from `swadm multicast list` output. One row per group, +/// tab-aligned: GROUP IP, KIND, EXT GROUP ID, UL GROUP ID, TAG, DETAIL. +/// External rows carry the NAT target as `nat=` in DETAIL, `nat=-` when +/// none is set. The KIND column separates external rows from underlay ones +/// (an external group can be IPv6 too, so the address family cannot), and +/// the header never parses as an address, so no line count is assumed. +fn external_nat_targets(out: &str) -> Vec<(IpAddr, String)> { + out.lines() + .filter_map(|line| { + let mut f = line.split_whitespace(); + let group = f.next()?.parse::().ok()?; + (f.next()? == "external").then_some(())?; + let nat = f.find_map(|t| t.strip_prefix("nat="))?; + Some((group, nat.to_string())) + }) + .collect() +} + +/// Print each group's control-plane mapping onto the underlay, read from +/// `swadm multicast list` in every switch zone. An external group's entry +/// names its NAT target, the admin-scoped underlay group the switch +/// replicates onto, which is the hop after the external path `check` +/// asserts. +/// +/// This is read-only, so nothing here affects whether the check passes. +/// The entry appears once a multicast group is created against the rack +/// API (a commtest run does this), so a freshly plumbed group legitimately +/// has none, and a rack that is down or unreadable gets a note rather than +/// a failed check. +async fn print_underlay_mappings( + cfg: &VoxelConfig, + name: &str, + groups: &[String], +) { + let Ok(addrs) = group_addrs(groups) else { + return; + }; + // Isolated mode numbers every sled statically, so the falcon topology + // (whose Runner construction logs at INFO) is only built when a + // scrimlet's lease actually has to be read. + let mut topo = None; + for (idx, sled) in + cfg.sleds().into_iter().filter(|s| s.scrimlet).enumerate() + { + let label = format!("switch{idx} ({})", sled.name); + let ip = if let Some(ip) = static_ip(cfg, &sled.name) { + ip + } else { + if topo.is_none() { + let Ok(t) = build_topo(cfg, name) else { return }; + topo = Some(t); + } + let Some(t) = &topo else { return }; + let resolved = match t.node_ref(&sled.name) { + Some(n) => { + serial_bounded( + &format!("reading {}'s address", sled.name), + resolve_external_ip( + cfg, &t.runner, &sled.name, n, false, + ), + ) + .await + } + None => Err(anyhow::anyhow!("not in the topology")), + }; + match resolved { + Ok(ip) => ip, + Err(_) => { + println!( + "underlay: {label} unresolvable, mapping unknown \ + (is the rack up?)" + ); + continue; + } + } + }; + let Some(out) = + ssh_capture(&ip, &zlogin(&format!("{SWADM} multicast list"))) + else { + println!( + "underlay: switch zone on {label} unreadable, mapping unknown" + ); + continue; + }; + let programmed = external_nat_targets(&out); + if programmed.is_empty() { + println!("underlay: no external groups programmed on {label}"); + continue; + } + for addr in &addrs { + match programmed.iter().find(|(g, _)| *g == IpAddr::V4(*addr)) { + Some((_, nat)) if nat != "-" => { + println!("underlay: {addr} -> {nat} on {label}") + } + Some(_) => { + println!("underlay: {addr} has no NAT target on {label}") + } + None => println!("underlay: {addr} not programmed on {label}"), + } + } + } +} + +/// Assert the whole external host path is live, printing one line per item. +/// +/// With no groups given, this covers everything voxel has plumbed, the same +/// set a groupless `down` tears down, so any sequence of `up` runs is +/// asserted whole. +/// +/// Only the external side is asserted here, and the trailing `underlay:` +/// lines show where the rack sends each group next (see +/// [`print_underlay_mappings`]). Proving underlay delivery past the switch +/// still needs a member in the group (a commtest run, or a joined probe +/// answering `ping`). +/// +/// # Errors +/// +/// Fails when any item is missing, so the CLI exit code reflects the result. +pub(crate) async fn check( + cfg: &VoxelConfig, + name: &str, + groups: &[String], +) -> anyhow::Result<()> { + let groups = if groups.is_empty() { + let plumbed = plumbed_groups(cfg, name).await?; + if plumbed.is_empty() { + println!("check: nothing plumbed"); + return Ok(()); + } + plumbed + } else { + groups.to_vec() + }; + let status = plumbing_status(cfg, name, &groups).await?; + let mut complete = true; + for (item, ok) in &status { + if *ok { + println!("ok: {item}"); + } else { + println!("MISSING: {item}"); + complete = false; + } + } + print_underlay_mappings(cfg, name, &groups).await; + if complete { + println!("check: PASS"); + Ok(()) + } else { + bail!("check: FAIL") + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn group_addr_strips_source_and_rejects_unicast() { + assert_eq!( + group_addr("232.100.0.1@192.168.1.199").unwrap(), + Ipv4Addr::new(232, 100, 0, 1) + ); + assert_eq!( + group_addr("239.1.1.1").unwrap(), + Ipv4Addr::new(239, 1, 1, 1) + ); + assert!(group_addr("198.51.100.1").is_err()); + assert!(group_addr("ff05::1").is_err()); + } + + #[test] + fn group_addrs_dedupes_repeats_in_first_occurrence_order() { + let groups = vec![ + "239.1.1.2".to_string(), + "239.1.1.1@192.168.1.199".to_string(), + "239.1.1.2@10.0.0.1".to_string(), + "239.1.1.1".to_string(), + ]; + assert_eq!( + group_addrs(&groups).unwrap(), + vec![Ipv4Addr::new(239, 1, 1, 2), Ipv4Addr::new(239, 1, 1, 1)] + ); + } + + #[test] + fn external_nat_targets_keeps_external_rows_of_both_families() { + // Shape of `swadm multicast list`: a header, external rows (v4 and + // v6) whose DETAIL carries `nat=`, and underlay rows that share the + // column layout but must not be counted as programmed groups. + let out = "\ +GROUP IP KIND EXT GROUP ID UL GROUP ID TAG DETAIL +239.100.0.1 external 8 - nexus nat=ff04::e4:64:0:1 vlan=- src=any +ff05::100 external 9 - nexus nat=ff04::e4:0:0:100 vlan=2 src=any +239.100.0.9 external 10 - nexus nat=- vlan=- src=any +ff04::e4:64:0:1 underlay 8 11 nexus rear0/0(underlay) rear1/0(underlay) +"; + assert_eq!( + external_nat_targets(out), + vec![ + ("239.100.0.1".parse().unwrap(), "ff04::e4:64:0:1".to_string()), + ("ff05::100".parse().unwrap(), "ff04::e4:0:0:100".to_string()), + ("239.100.0.9".parse().unwrap(), "-".to_string()), + ] + ); + assert!(external_nat_targets("GROUP IP KIND\n").is_empty()); + } + + #[test] + fn filter_chains_every_scrimlet_in_one_rule() { + let ifaces = vec!["enp0s9".to_string(), "enp0s10".to_string()]; + let cmd = + filter_cmd("enp0s11", &Ipv4Addr::new(239, 1, 1, 1), 100, &ifaces); + assert_eq!( + cmd, + "tc filter replace dev enp0s11 ingress handle 1 pref 100 protocol ip flower \ + dst_ip 239.1.1.1 action mirred egress mirror dev enp0s9 \ + action mirred egress mirror dev enp0s10" + ); + } + + /// `tc -json filter show dev enp0s11 ingress` with three voxel filters + /// (239.1.1.1 fully mirrored, 239.1.1.2 mirrored to one NIC only, and + /// 239.1.1.4 redirecting rather than mirroring) plus two foreign ones: a + /// flower filter below `PREF_BASE` matching 239.1.1.1's address, and a + /// matchall filter sitting inside the pref range. + const FILTERS: &str = r#"[ + {"protocol":"ip","pref":10,"kind":"flower","chain":0, + "options":{"handle":1,"keys":{"eth_type":"ipv4","dst_ip":"239.1.1.1"}, + "actions":[ + {"order":1,"kind":"mirred","mirred_action":"mirror","direction":"egress","to_dev":"enp0s9"}, + {"order":2,"kind":"mirred","mirred_action":"mirror","direction":"egress","to_dev":"enp0s10"}]}}, + {"protocol":"all","pref":102,"kind":"matchall","chain":0, + "options":{"handle":1,"actions":[]}}, + {"protocol":"ip","pref":100,"kind":"flower","chain":0}, + {"protocol":"ip","pref":100,"kind":"flower","chain":0, + "options":{"handle":1,"keys":{"eth_type":"ipv4","dst_ip":"239.1.1.1"}, + "actions":[ + {"order":1,"kind":"mirred","mirred_action":"mirror","direction":"egress","to_dev":"enp0s9"}, + {"order":2,"kind":"mirred","mirred_action":"mirror","direction":"egress","to_dev":"enp0s10"}]}}, + {"protocol":"ip","pref":101,"kind":"flower","chain":0, + "options":{"handle":1,"keys":{"eth_type":"ipv4","dst_ip":"239.1.1.2"}, + "actions":[ + {"order":1,"kind":"mirred","mirred_action":"mirror","direction":"egress","to_dev":"enp0s9"}]}}, + {"protocol":"ip","pref":103,"kind":"flower","chain":0, + "options":{"handle":1,"keys":{"eth_type":"ipv4","dst_ip":"239.1.1.4"}, + "actions":[ + {"order":1,"kind":"mirred","mirred_action":"redirect","direction":"egress","to_dev":"enp0s9"}, + {"order":2,"kind":"mirred","mirred_action":"mirror","direction":"ingress","to_dev":"enp0s10"}]}} + ]"#; + + #[test] + fn mirror_installed_is_scoped_to_the_group_filter() { + let filters = parse_filters(FILTERS).unwrap(); + let ifaces = vec!["enp0s9".to_string(), "enp0s10".to_string()]; + assert!(mirror_installed( + &filters, + &Ipv4Addr::new(239, 1, 1, 1), + &ifaces + )); + // enp0s10 is only in the first group's filter, so the second is + // partial. + assert!(!mirror_installed( + &filters, + &Ipv4Addr::new(239, 1, 1, 2), + &ifaces + )); + assert!(!mirror_installed( + &filters, + &Ipv4Addr::new(239, 1, 1, 3), + &ifaces + )); + // Neither a redirect nor an ingress mirror counts as an egress mirror. + assert!(!mirror_installed( + &filters, + &Ipv4Addr::new(239, 1, 1, 4), + &ifaces + )); + } + + #[test] + fn prefs_are_reused_per_group_and_never_collide() { + let filters = parse_filters(FILTERS).unwrap(); + assert_eq!( + group_pref(&filters, &Ipv4Addr::new(239, 1, 1, 2)), + Some(101) + ); + assert_eq!(group_pref(&filters, &Ipv4Addr::new(239, 1, 1, 3)), None); + // The foreign filter at pref 10 matches 239.1.1.1's address but sits + // outside voxel's pref range, so the group resolves to its own filter. + assert_eq!( + group_pref(&filters, &Ipv4Addr::new(239, 1, 1, 1)), + Some(100) + ); + + // A new group takes the lowest pref no installed filter holds, voxel's + // or not, and two new groups in one run do not land on the same one. + let mut taken = Vec::new(); + assert_eq!(free_pref(&filters, &mut taken), 104); + assert_eq!(free_pref(&filters, &mut taken), 105); + } + + #[test] + fn parse_filters_keeps_foreign_entries_for_pref_accounting() { + // A u32 filter with a string handle, an entry with only a pref, and + // an entry whose pref is not a number: the first two must keep their + // prefs out of `free_pref`'s reach, the last is skipped alone. + let json = r#"[ + {"protocol":"all","pref":100,"kind":"u32", + "options":{"handle":"800::800"}}, + {"pref":101}, + {"protocol":"ip","pref":"bogus","kind":"flower"} + ]"#; + let filters = parse_filters(json).unwrap(); + assert_eq!( + filters.iter().map(|f| f.pref).collect::>(), + vec![100, 101] + ); + assert!(filters.iter().all(|f| !f.owned())); + let mut taken = Vec::new(); + assert_eq!(free_pref(&filters, &mut taken), 102); + } + + /// Capture of `tc -json filter show dev enp0s11 ingress` from a live cr1 + /// (iproute2 on the router image) after + /// `up --group 239.1.1.1 --group 239.2.2.2`. Unlike the hand-written + /// `FILTERS`, this keeps everything real output carries: each filter's + /// per-pref summary stub without `options`, `not_in_hw`, `control_action`, + /// and the index/ref/bind action fields. + const CAPTURED_FILTERS: &str = include_str!("testdata/tc-filter-show.json"); + + #[test] + fn parse_filters_handles_a_real_capture() { + let filters = parse_filters(CAPTURED_FILTERS).unwrap(); + // The summary stubs come through as unowned entries alongside the + // full ones, so each pref appears twice. + assert_eq!( + filters.iter().map(|f| f.pref).collect::>(), + [100, 100, 101, 101] + ); + assert_eq!(filters.iter().filter(|f| f.owned()).count(), 2); + + let groups = [Ipv4Addr::new(239, 1, 1, 1), Ipv4Addr::new(239, 2, 2, 2)]; + let ifaces = vec!["enp0s9".to_string(), "enp0s10".to_string()]; + assert_eq!(group_pref(&filters, &groups[0]), Some(100)); + assert_eq!(group_pref(&filters, &groups[1]), Some(101)); + assert!(mirror_installed(&filters, &groups[0], &ifaces)); + assert!(mirror_installed(&filters, &groups[1], &ifaces)); + + let mut taken = Vec::new(); + assert_eq!(free_pref(&filters, &mut taken), 102); + } + + #[test] + fn group_mac_maps_the_low_23_bits() { + assert_eq!( + group_mac(&Ipv4Addr::new(239, 1, 1, 1)), + "01:00:5e:01:01:01" + ); + assert_eq!( + group_mac(&Ipv4Addr::new(239, 255, 255, 255)), + "01:00:5e:7f:ff:ff" + ); + // Groups differing only above bit 23 alias the same Ethernet address, + // and the dedupe keeps one entry for both. + let aliased = Ipv4Addr::new(224, 129, 1, 1); + assert_eq!(group_mac(&aliased), "01:00:5e:01:01:01"); + assert_eq!( + group_macs(&[Ipv4Addr::new(239, 1, 1, 1), aliased]), + ["01:00:5e:01:01:01"] + ); + } + + #[test] + fn down_deletes_only_recorded_memberships() { + let addrs = + [Ipv4Addr::new(239, 1, 1, 1), Ipv4Addr::new(239, 128, 0, 1)]; + // Only the first group's membership is recorded as voxel's. The + // second aliases the all-hosts address 01:00:5e:00:00:01, which the + // kernel holds on every interface. + let owned = ["01:00:5e:01:01:01".to_string()]; + let (dropped, foreign) = deletable_members(&addrs, &[], &owned); + assert_eq!(dropped, ["01:00:5e:01:01:01"]); + assert_eq!(foreign, ["01:00:5e:00:00:01"]); + + // A staying group aliasing the recorded membership keeps it. + let staying = [Ipv4Addr::new(224, 129, 1, 1)]; + let (dropped, foreign) = deletable_members(&addrs, &staying, &owned); + assert!(dropped.is_empty()); + assert_eq!(foreign, ["01:00:5e:00:00:01"]); + } + + #[test] + fn route_gateways_takes_host_routes_for_the_group_only() { + let entry = |dest: &str, gw: &str, flags: &str| RouteEntry { + dest: dest.into(), + gateway: gw.into(), + flags: flags.into(), + }; + let entries = [ + // Header line, as route_entries passes it through. + entry("Destination", "Gateway", "Flags"), + // The interface route illumos holds for all of 224.0.0.0/4. + entry("224.0.0.0", "172.30.199.2", "U"), + // A unicast host route (the rack's external segment). + entry("198.51.100.20", "172.30.199.14", "UGH"), + entry("239.1.1.1", "172.30.199.14", "UGH"), + // A stale duplicate from a prior launch. + entry("239.1.1.1", "172.30.199.16", "UGH"), + entry("239.2.2.2", "172.30.199.14", "UGH"), + ]; + assert_eq!( + route_gateways(&entries, &Ipv4Addr::new(239, 1, 1, 1)), + ["172.30.199.14", "172.30.199.16"] + ); + // The interface route covering the group is not a host route. + assert!( + route_gateways(&entries, &Ipv4Addr::new(224, 0, 0, 0)).is_empty() + ); + } + + #[test] + fn state_groups_come_from_the_record_alone() { + let state = MulticastState { + environment: "voxel".to_string(), + routes: vec![ + MulticastRoute { + group: Ipv4Addr::new(239, 1, 1, 1), + gateway: "172.30.199.14".to_string(), + }, + MulticastRoute { + group: Ipv4Addr::new(239, 2, 2, 2), + gateway: "172.30.199.14".to_string(), + }, + ], + }; + assert_eq!( + state_groups(Some(&state)), + [Ipv4Addr::new(239, 1, 1, 1), Ipv4Addr::new(239, 2, 2, 2)] + ); + // No record means no groups, so a groupless `down` touches nothing. + assert!(state_groups(None).is_empty()); + } + + #[test] + fn route_scope_leaves_another_environment_alone() { + let entry = |dest: &str, gw: &str| RouteEntry { + dest: dest.into(), + gateway: gw.into(), + flags: "UGH".into(), + }; + let entries = [ + entry("239.1.1.1", "172.30.199.14"), + entry("239.1.1.1", "172.30.199.24"), + ]; + let state = MulticastState { + environment: "voxel".to_string(), + routes: vec![MulticastRoute { + group: Ipv4Addr::new(239, 1, 1, 1), + gateway: "172.30.199.14".to_string(), + }], + }; + let group = Ipv4Addr::new(239, 1, 1, 1); + + assert_eq!( + route_gateways(&entries, &group), + ["172.30.199.14", "172.30.199.24"] + ); + assert_eq!( + remaining_host_routes(&entries, &[group], &state), + ["host route 239.1.1.1 -> 172.30.199.14"] + ); + assert_eq!( + foreign_host_routes(&entries, &[group], Some(&state)), + ["host route 239.1.1.1 -> 172.30.199.24"] + ); + } + + #[test] + fn parse_filters_rejects_non_json() { + assert!(parse_filters("filter protocol ip pref 100 flower").is_err()); + } + + #[test] + fn parse_members_keeps_link_layer_entries_only() { + let json = r#"[ + {"ifindex":2,"ifname":"enp0s11","maddr":[ + {"link":"33:33:00:00:00:01"}, + {"family":"inet","address":"224.0.0.1"}, + {"link":"01:00:5e:01:01:01","users":2}]}]"#; + assert_eq!( + parse_members(json).unwrap(), + ["33:33:00:00:00:01", "01:00:5e:01:01:01"] + ); + } + + #[test] + fn parse_members_rejects_non_json() { + assert!(parse_members("1:\tlo\n\tinet 224.0.0.1\n").is_err()); + } + + #[test] + fn gateway_matches_the_field_not_the_whole_output() { + let out = " route to: 239.1.1.1\ndestination: 239.1.1.1\n gateway: 172.30.199.16\n \ + interface: voxel_ext0\n"; + assert!(gateway_matches(out, "172.30.199.16")); + // A prefix of the real gateway must not pass. + assert!(!gateway_matches(out, "172.30.199.1")); + assert!(!gateway_matches(out, "239.1.1.1")); + } + + #[test] + fn mirror_target_derives_cr1_from_topology() { + let cfg = VoxelConfig::from_toml("").unwrap(); + let MirrorTarget { router, iif, ifaces } = + MirrorTarget::new(&cfg).unwrap(); + assert_eq!(router, "cr1"); + assert_eq!(iif, "enp0s11"); + assert_eq!(ifaces, ["enp0s9", "enp0s10"]); + } +} diff --git a/voxel/src/net.rs b/voxel/src/net.rs index 54642bc..b9b37af 100644 --- a/voxel/src/net.rs +++ b/voxel/src/net.rs @@ -257,6 +257,41 @@ pub(crate) fn ssh_capture(ip: &str, remote: &str) -> Option { .then(|| String::from_utf8_lossy(&out.stdout).into_owned()) } +/// How a remote command could fail: the ssh transport never reached the node, +/// or the node ran the command and it exited non-zero. +/// +/// This split lets callers take an "unreachable node" path (a destroyed rack) +/// without also swallowing real command failures on a live one. +#[derive(Debug)] +pub(crate) enum SshFailure { + /// ssh could not connect or authenticate (exit 255), so the node itself + /// is unreachable. + Unreachable, + /// The command ran remotely and failed, or ssh could not be run locally; + /// holds the failure text. + Failed(String), +} + +/// Like [`ssh_capture`], but keeps the failure mode instead of folding every +/// failure into `None`. +pub(crate) fn ssh_try_capture( + ip: &str, + remote: &str, +) -> Result { + let Some(out) = ssh_exec(ip, remote) else { + return Err(SshFailure::Failed("ssh could not be run locally".into())); + }; + if out.status.code() == Some(255) { + return Err(SshFailure::Unreachable); + } + if !out.status.success() { + return Err(SshFailure::Failed( + String::from_utf8_lossy(&out.stderr).into_owned(), + )); + } + Ok(String::from_utf8_lossy(&out.stdout).into_owned()) +} + /// Like [`ssh_capture`], but returns the remote command's combined output even /// when it exits non-zero - for callers (e.g. `sp exec`) that want the remote /// tool's OWN error text (faux-mgs prints `Error: ...`, which the caller folds @@ -411,16 +446,18 @@ fn dig_soa(dns_ip: &str, zone: &str) -> Option { } } -/// (Re)point the host route for the rack's external network (`prefix`) at ce's -/// current external IP. ce's host-facing NIC gets a fresh random MAC - and thus -/// a fresh DHCP IP - every launch, so any static route goes stale; discovering -/// it here keeps the external services reachable without a manual hunt. The -/// route is keyed by `prefix`, so racks with distinct external prefixes don't -/// collide. With `apply == false` it just prints the command. -/// The gateways currently routing `dest` (an IPv4 network address like -/// `198.51.100.0`), read from `netstat -rn -f inet`. Used to purge every stale -/// route for a prefix - dead-ce gateways from prior launches pile up otherwise. -pub(crate) fn route_gateways(dest: &str) -> Vec { +/// A routing-table entry as `netstat -rn -f inet` prints it, reduced to the +/// destination, gateway, and flags columns. +pub(crate) struct RouteEntry { + pub dest: String, + pub gateway: String, + pub flags: String, +} + +/// The IPv4 routing table, one entry per line with at least a destination and +/// gateway column. Headers and separators come through as unparseable +/// destinations, so consumers matching on an address never see them. +pub(crate) fn route_entries() -> Vec { let out = match std::process::Command::new("netstat") .args(["-rn", "-f", "inet"]) .output() @@ -431,13 +468,32 @@ pub(crate) fn route_gateways(dest: &str) -> Vec { out.lines() .filter_map(|l| { let mut it = l.split_whitespace(); - let d = it.next()?; - let gw = it.next()?; - (d == dest).then(|| gw.to_string()) + Some(RouteEntry { + dest: it.next()?.to_string(), + gateway: it.next()?.to_string(), + flags: it.next().unwrap_or_default().to_string(), + }) }) .collect() } +/// The gateways currently routing `dest` (an IPv4 network address like +/// `198.51.100.0`). Used to purge every stale route for a prefix - dead-ce +/// gateways from prior launches pile up otherwise. +pub(crate) fn route_gateways(dest: &str) -> Vec { + route_entries() + .into_iter() + .filter_map(|e| (e.dest == dest).then_some(e.gateway)) + .collect() +} + +/// (Re)point the host route for the rack's external network (`prefix`) at ce's +/// current external IP. ce's host-facing NIC gets a fresh random MAC, and thus +/// a fresh DHCP IP, every launch, so any static route goes stale. +/// +/// Discovering it here keeps the external services reachable without manual hunting. +/// The route is keyed by `prefix`, so racks with distinct external prefixes don't +/// collide. With `apply == false` it just prints the command. pub(crate) async fn set_external_route( d: &Runner, ce: NodeRef, diff --git a/voxel/src/network.rs b/voxel/src/network.rs index e6630e9..d5e557c 100644 --- a/voxel/src/network.rs +++ b/voxel/src/network.rs @@ -21,7 +21,7 @@ use voxel_config::{SledDesc, VoxelConfig}; use crate::net::{resolve_external_ip, ssh_capture, ssh_output, zlogin}; use crate::topo::build_topo; -const SWADM: &str = "/opt/oxide/dendrite/bin/swadm"; +pub(crate) const SWADM: &str = "/opt/oxide/dendrite/bin/swadm"; const MGADM: &str = "/opt/oxide/mgd/bin/mgadm"; /// The global switch index (`switchN`) for each scrimlet, in order. diff --git a/voxel/src/patch.rs b/voxel/src/patch.rs index b30efbb..ec52113 100644 --- a/voxel/src/patch.rs +++ b/voxel/src/patch.rs @@ -31,6 +31,7 @@ use anyhow::{Context, anyhow}; use camino::{Utf8Path, Utf8PathBuf}; +use itertools::Itertools; use libfalcon::{NodeRef, Runner}; use slog::{info, warn}; use std::time::Duration; @@ -290,7 +291,7 @@ fn fetch_sha(url: &str) -> anyhow::Result { /// download. Returns the local tarball path. fn acquire(comp: &Component, reference: &str) -> anyhow::Result { let dir = cache_dir().join(comp.repo).join(reference); - std::fs::create_dir_all(&dir).with_context(|| format!("mkdir {}", dir))?; + std::fs::create_dir_all(&dir).with_context(|| format!("mkdir {dir}"))?; let ext = comp.archive.ext(); let tarball = dir.join(format!("{}.{ext}", comp.pkg)); @@ -534,7 +535,7 @@ pub(crate) async fn cmd_rack_patch( comp.name, comp.repo, comp.pkg, - nodes.iter().map(|(n, _)| n.as_str()).collect::>().join(", "), + nodes.iter().map(|(n, _)| n.as_str()).join(", "), comp.note ); if dry_run { @@ -643,7 +644,7 @@ pub(crate) fn cmd_image_patch( } // FALCON_DATASET is already exported by resolve_falcon_env; patch-image.sh + // build-image.sh read it. - let status = cmd.status().map_err(|e| anyhow!("run {}: {e}", script))?; + let status = cmd.status().map_err(|e| anyhow!("run {script}: {e}"))?; if !status.success() { return Err(anyhow!("patch-image.sh failed")); } diff --git a/voxel/src/rack.rs b/voxel/src/rack.rs index e6482ca..8ea62cb 100644 --- a/voxel/src/rack.rs +++ b/voxel/src/rack.rs @@ -13,8 +13,8 @@ use voxel_config::VoxelConfig; use crate::isolated_external::{DryRun, link_mtu, up as external_up}; use crate::net::{ - ce_static_ip, resolve_external_ip, set_external_route, ssh_capture, - ssh_output, wait_external_reachable, zlogin, + ce_static_ip, resolve_external_ip, serial_bounded, set_external_route, + ssh_capture, ssh_output, wait_external_reachable, zlogin, }; use crate::rss::watch_rss; use crate::topo::{ @@ -646,6 +646,82 @@ fn rss_watch_cap(emu_sp: bool, racks: usize) -> std::time::Duration { }) } +/// Per-sled rpool usage and pool health, parsed from one ssh round trip. +struct SledDisk { + used: u64, + avail: u64, + degraded: bool, +} + +/// Separates the `zfs list` usage line from the `zpool status -x` health +/// summary in the combined remote command output. +const DISK_REPORT_SEP: &str = "--voxel-disk--"; + +fn parse_sled_disk(out: &str) -> Option { + let (usage, health) = out.split_once(DISK_REPORT_SEP)?; + let mut fields = usage.split_whitespace(); + let used = fields.next()?.parse().ok()?; + let avail = fields.next()?.parse().ok()?; + let health = health.trim(); + let degraded = + !health.is_empty() && !health.contains("all pools are healthy"); + Some(SledDisk { used, avail, degraded }) +} + +fn gib(bytes: u64) -> String { + format!("{:.1}G", bytes as f64 / (1024.0 * 1024.0 * 1024.0)) +} + +/// Report each sled's rpool usage and pool health. The sparse U.2/M.2 vdev +/// backing files in the guest's /var/tmp overcommit the rpool and only ever +/// grow (illumos file vdevs never return freed blocks), so a long-held rack +/// drifts toward the ENOSPC cliff, which takes out svc.configd and, with it, +/// dendrite in the switch zone. Surfacing the pressure here lets an operator +/// relaunch before the cliff. +async fn print_sled_disk_pressure(cfg: &VoxelConfig, topo: &Topo) { + println!("sled disks:"); + for (s, n) in &topo.sleds { + let resolved = serial_bounded( + &format!("reading {}'s address", s.name), + resolve_external_ip(cfg, &topo.runner, &s.name, *n, false), + ) + .await; + let Ok(ip) = resolved else { + println!(" {}: unreachable (is the rack up?)", s.name); + continue; + }; + let out = ssh_capture( + &ip, + &format!( + "zfs list -Hpo used,avail rpool; \ + echo {DISK_REPORT_SEP}; zpool status -x" + ), + ); + let Some(disk) = out.as_deref().and_then(parse_sled_disk) else { + println!(" {}: rpool unreadable", s.name); + continue; + }; + let total = disk.used + disk.avail; + let pct = if total == 0 { + 0 + } else { + (disk.used as f64 / total as f64 * 100.0).round() as u64 + }; + let low = if pct >= 85 { " WARNING: low space" } else { "" }; + let degraded = if disk.degraded { + " DEGRADED pools (zpool status -x on the sled)" + } else { + "" + }; + println!( + " {}: rpool {} / {} ({pct}%){low}{degraded}", + s.name, + gib(disk.used), + gib(total) + ); + } +} + pub(crate) async fn cmd_status( cfg: &VoxelConfig, name: &str, @@ -657,6 +733,7 @@ pub(crate) async fn cmd_status( bail!("no RSS sled in topology"); } let d = &topo.runner; + print_sled_disk_pressure(cfg, &topo).await; // Multi-rack racks converge under each other's load - watch longer (matches // cmd_launch). Duration is Copy, so each watcher closure gets its own. let watch_cap = rss_watch_cap(false, racks); diff --git a/voxel/src/rss_request.rs b/voxel/src/rss_request.rs index 2b65c91..4d07909 100644 --- a/voxel/src/rss_request.rs +++ b/voxel/src/rss_request.rs @@ -13,8 +13,8 @@ use rack_init_config::{ BgpPeerConfig, BootstrapAddressDiscovery, IdOrdMap, IpRange, Ipv4Range, Ipv6Range, LinkFec, LinkSpeed, LldpAdminStatus, LldpPortConfig, MaxPathConfig, PortConfig, RackInitializeRequest, RackNetworkConfig, - RecoverySiloConfig, RouteConfig, RouterLifetimeConfig, RouterPeerType, - ServiceIpPoolConfig, SwitchSlot, UplinkAddress, UplinkAddressConfig, + RecoverySiloConfig, RouteConfig, RouterLifetimeConfig, ServiceIpPoolConfig, + SwitchSlot, UnnumberedRouter, UplinkAddress, UplinkAddressConfig, UplinkPorts, }; use voxel_config::{RouterMode, UplinkPort, VoxelConfig}; @@ -86,12 +86,13 @@ fn uplink_port(p: &UplinkPort, mode: RouterMode) -> Result { vec![BgpPeerConfig { asn: p.peer_asn, port: p.port.clone(), - addr: RouterPeerType::Unnumbered { + addr: UnnumberedRouter { router_lifetime: RouterLifetimeConfig::new( p.router_lifetime, ) .map_err(|e| anyhow::anyhow!("router_lifetime: {e}"))?, - }, + } + .into(), hold_time: None, idle_hold_time: None, delay_open: None, @@ -136,6 +137,7 @@ fn uplink_port(p: &UplinkPort, mode: RouterMode) -> Result { autoneg: false, lldp: Some(lldp(&p.switch, &p.lldp)), tx_eq: None, + allow_ddm_traffic: false, }) } @@ -156,6 +158,9 @@ fn interconnect_port(switch: &str, port: &str) -> Result { autoneg: false, lldp: Some(lldp(switch, &format!("interconnect-{port}"))), tx_eq: None, + // Interconnect ports carry cross-rack mg-ddm peering, the case the + // multirack DDM gate exists for. The flag is unenforced today. + allow_ddm_traffic: true, }) } @@ -319,6 +324,8 @@ pub fn config_rss_toml(cfg: &VoxelConfig, rack: usize) -> Result { #[cfg(test)] mod tests { + use rack_init_config::RouterPeerType; + use super::*; fn config(racks: usize, network: &str) -> VoxelConfig { diff --git a/voxel/src/sp_cmd.rs b/voxel/src/sp_cmd.rs index 6ebfed7..850c02d 100644 --- a/voxel/src/sp_cmd.rs +++ b/voxel/src/sp_cmd.rs @@ -309,7 +309,7 @@ async fn sp_reflash( image: &Utf8Path, ) -> anyhow::Result<()> { if !image.exists() { - return Err(anyhow!("image not found: {}", image)); + return Err(anyhow!("image not found: {image}")); } let local = image.as_str(); let topo = build_topo(cfg, name)?; @@ -334,8 +334,7 @@ async fn sp_reflash( // instance (build-cp.sh copies `[sp].rot_image` -> rot.flash). Replace it // and restart them all; the SPs reconnect to the bridge. eprintln!( - "[voxel] reflashing shared RoT (rot.flash) on {sw} from {}", - image + "[voxel] reflashing shared RoT (rot.flash) on {sw} from {image}" ); if !scp_to( &ip, @@ -371,8 +370,7 @@ async fn sp_reflash( let port = resolve_port(&fleet, target)?; let zip = image.file_name().ok_or_else(|| anyhow!("bad image filename"))?; eprintln!( - "[voxel] reflashing SP {target} (port {port}) on {sw} from {}", - image + "[voxel] reflashing SP {target} (port {port}) on {sw} from {image}" ); let remote_zip_gz = format!("{SWITCH_ZONE_ROOT}/var/tmp/{zip}"); if !scp_to(&ip, local, &remote_zip_gz) { @@ -464,7 +462,7 @@ async fn sp_debug( let local = crate::util::temp_dir().join(format!("voxel-sp-env-{port}.scfg")); std::fs::write(&local, &content) - .map_err(|e| anyhow!("write {}: {e}", local))?; + .map_err(|e| anyhow!("write {local}: {e}"))?; let remote_gz = format!("{SWITCH_ZONE_ROOT}/var/tmp/voxel-sp-env-{port}.scfg"); let remote = format!("/var/tmp/voxel-sp-env-{port}.scfg"); @@ -603,7 +601,7 @@ exit 1 ); let local = crate::util::temp_dir().join(format!("voxel-ipcc-{port}.sh")); std::fs::write(&local, &script) - .map_err(|e| anyhow!("write {}: {e}", local))?; + .map_err(|e| anyhow!("write {local}: {e}"))?; if !scp_to( &ip, local.as_str(), @@ -746,7 +744,7 @@ async fn sp_dump( let local = crate::util::temp_dir() .join(format!("voxel-sp-dumpenv-{port}.scfg")); std::fs::write(&local, &content) - .map_err(|e| anyhow!("write {}: {e}", local))?; + .map_err(|e| anyhow!("write {local}: {e}"))?; let remote_gz = format!("{SWITCH_ZONE_ROOT}/var/tmp/voxel-sp-dumpenv-{port}.scfg"); let remote = format!("/var/tmp/voxel-sp-dumpenv-{port}.scfg"); @@ -787,7 +785,7 @@ async fn sp_dump( // Pull the zip to the host and decode it there (humility + archive live here). let host_dir = crate::util::temp_dir().join(format!("voxel-spdump-{port}")); std::fs::create_dir_all(&host_dir) - .map_err(|e| anyhow!("mkdir {}: {e}", host_dir))?; + .map_err(|e| anyhow!("mkdir {host_dir}: {e}"))?; let zip_local = host_dir.join("dump.zip"); let zip_remote = format!("{SWITCH_ZONE_ROOT}{dump_dir}/dump.zip"); if !scp_from(&ip, &zip_remote, zip_local.as_str()) { @@ -828,8 +826,7 @@ async fn sp_dump( print!("{}", String::from_utf8_lossy(&dec.stdout)); eprint!("{}", String::from_utf8_lossy(&dec.stderr)); eprintln!( - "[voxel] dump saved: {} - inspect further with `{humility} -d {} `", - hydrated, hydrated + "[voxel] dump saved: {hydrated} - inspect further with `{humility} -d {hydrated} `" ); Ok(()) } @@ -992,9 +989,9 @@ fn flash( anyhow!("[sp].emu_bin is not set (path to the sp-emu binary)") })?; if !image.exists() { - return Err(anyhow!("image not found: {}", image)); + return Err(anyhow!("image not found: {image}")); } - eprintln!("[voxel] flashing {} -> {}", image, out); + eprintln!("[voxel] flashing {image} -> {out}"); let status = std::process::Command::new(emu_bin) .env("SP_EMU_FLASH", out) .args(["flash", "a"]) @@ -1004,21 +1001,20 @@ fn flash( if !status.success() { return Err(anyhow!("sp-emu flash failed")); } - println!("flashed {}", out); + println!("flashed {out}"); Ok(()) } fn build(commit: &str) -> anyhow::Result<()> { let script = build_sp_script()?; eprintln!( - "[voxel] building sp-emu hubris images for {commit} via {}", - script + "[voxel] building sp-emu hubris images for {commit} via {script}" ); let status = std::process::Command::new("bash") .arg(&script) .arg(commit) .status() - .map_err(|e| anyhow!("run {}: {e}", script))?; + .map_err(|e| anyhow!("run {script}: {e}"))?; if !status.success() { return Err(anyhow!("build-sp.sh failed for {commit}")); } diff --git a/voxel/src/testdata/tc-filter-show.json b/voxel/src/testdata/tc-filter-show.json new file mode 100644 index 0000000..6e8f47c --- /dev/null +++ b/voxel/src/testdata/tc-filter-show.json @@ -0,0 +1,98 @@ +[ + { + "protocol": "ip", + "pref": 100, + "kind": "flower", + "chain": 0 + }, + { + "protocol": "ip", + "pref": 100, + "kind": "flower", + "chain": 0, + "options": { + "handle": 1, + "keys": { + "eth_type": "ipv4", + "dst_ip": "239.1.1.1" + }, + "not_in_hw": true, + "actions": [ + { + "order": 1, + "kind": "mirred", + "mirred_action": "mirror", + "direction": "egress", + "to_dev": "enp0s9", + "control_action": { + "type": "pipe" + }, + "index": 1, + "ref": 1, + "bind": 1 + }, + { + "order": 2, + "kind": "mirred", + "mirred_action": "mirror", + "direction": "egress", + "to_dev": "enp0s10", + "control_action": { + "type": "pipe" + }, + "index": 2, + "ref": 1, + "bind": 1 + } + ] + } + }, + { + "protocol": "ip", + "pref": 101, + "kind": "flower", + "chain": 0 + }, + { + "protocol": "ip", + "pref": 101, + "kind": "flower", + "chain": 0, + "options": { + "handle": 1, + "keys": { + "eth_type": "ipv4", + "dst_ip": "239.2.2.2" + }, + "not_in_hw": true, + "actions": [ + { + "order": 1, + "kind": "mirred", + "mirred_action": "mirror", + "direction": "egress", + "to_dev": "enp0s9", + "control_action": { + "type": "pipe" + }, + "index": 3, + "ref": 1, + "bind": 1 + }, + { + "order": 2, + "kind": "mirred", + "mirred_action": "mirror", + "direction": "egress", + "to_dev": "enp0s10", + "control_action": { + "type": "pipe" + }, + "index": 4, + "ref": 1, + "bind": 1 + } + ] + } + } +] diff --git a/voxel/src/topo.rs b/voxel/src/topo.rs index 1a96d44..9a8ea3f 100644 --- a/voxel/src/topo.rs +++ b/voxel/src/topo.rs @@ -6,7 +6,7 @@ //! cargo-bay staging that feeds it (generated sled/RSS/FRR/switch1 config + //! sprockets keys). -use anyhow::{Context, anyhow}; +use anyhow::{Context, anyhow, bail}; use attest_mock::MockData; use camino::{Utf8Path, Utf8PathBuf}; use indoc::formatdoc; @@ -220,8 +220,7 @@ pub(crate) fn reset_node_cargo_bay(cfg: &VoxelConfig) -> anyhow::Result<()> { for node in nodes { let dir = cargo_bay(&node); if dir.exists() { - fs::remove_dir_all(&dir) - .with_context(|| format!("reset {}", dir))?; + fs::remove_dir_all(&dir).with_context(|| format!("reset {dir}"))?; } fs::create_dir_all(&dir)?; } @@ -572,9 +571,95 @@ pub(crate) fn stage_config( stage_sp_emu(cfg, &fleet, &dir, emu_rot)?; } } + + stage_ssh_pubkey(cfg)?; + Ok(()) +} + +/// Stage the operator's SSH public key into every node's cargo-bay as +/// `root_authorized_keys`, which voxel-init's `setup_ssh` (sled and router +/// roles alike) appends to root's `authorized_keys`. Plain `ssh root@` +/// then authenticates by key, with no need for the `SSH_ASKPASS` shim that +/// supplies the rack's empty root password. +/// +/// An explicit `[falcon].ssh_pubkey` must exist and validate; the default +/// probes `~/.ssh/id_ed25519.pub`, `id_ecdsa.pub`, `id_rsa.pub` in that order +/// (an explicit list, so certificates and surprising `id_*` variants are never +/// picked up) and skips staging when none exist. +fn stage_ssh_pubkey(cfg: &VoxelConfig) -> anyhow::Result<()> { + let path = if let Some(p) = &cfg.falcon.ssh_pubkey { + let p = Utf8PathBuf::from(p); + if !p.is_file() { + bail!("[falcon].ssh_pubkey '{p}' does not exist"); + } + p + } else { + let Ok(home) = std::env::var("HOME") else { return Ok(()) }; + let ssh = Utf8Path::new(&home).join(".ssh"); + let Some(p) = ["id_ed25519.pub", "id_ecdsa.pub", "id_rsa.pub"] + .iter() + .map(|f| ssh.join(f)) + .find(|p| p.is_file()) + else { + return Ok(()); + }; + p + }; + let body = read_ssh_pubkey(&path) + .with_context(|| format!("ssh public key {path}"))?; + eprintln!("[voxel] staging ssh public key {path} into the cargo-bays"); + let mut nodes: Vec = + cfg.sleds().into_iter().map(|s| s.name).collect(); + nodes.extend(cfg.topology.routers.iter().cloned()); + for node in nodes { + let dir = cargo_bay(&node); + fs::create_dir_all(&dir)?; + fs::write(dir.join("root_authorized_keys"), &body)?; + } Ok(()) } +/// Read and validate a public-key file, returning its normalized content +/// (CRs stripped, trailing newline). Validation is by content, not filename: +/// the cargo-bay is mounted into every guest, so an `ssh_pubkey` mispointed +/// at a private key would hand the secret to the whole rack. Every non-empty +/// line must carry a recognized public-key algorithm prefix. +fn read_ssh_pubkey(path: &Utf8Path) -> anyhow::Result { + validate_ssh_pubkey(&fs::read_to_string(path)?) +} + +/// The content check behind [`read_ssh_pubkey`], split out for unit tests. +fn validate_ssh_pubkey(raw: &str) -> anyhow::Result { + if raw.contains("PRIVATE KEY") { + bail!("this is a private key; refusing to stage it into the guests"); + } + let mut body = String::new(); + for line in raw.lines() { + let line = line.trim_end_matches('\r'); + if line.is_empty() { + continue; + } + let ok = ["ssh-ed25519 ", "ssh-rsa ", "ssh-dss "] + .iter() + .any(|p| line.starts_with(p)) + || line.starts_with("ecdsa-sha2-") + || line.starts_with("sk-ssh-ed25519@") + || line.starts_with("sk-ecdsa-"); + if !ok { + bail!( + "line does not look like an OpenSSH public key: '{}...'", + line.chars().take(24).collect::() + ); + } + body.push_str(line); + body.push('\n'); + } + if body.is_empty() { + bail!("no keys found"); + } + Ok(body) +} + /// Stage the `sp-emu` binary + each emulated SP's flashed hubris image into a /// scrimlet's cargo-bay (`sp-emu/`), so `voxel-init` can run the real-firmware SPs /// in that switch zone. Each emu SP's image is flashed into `.flash` @@ -594,8 +679,8 @@ fn stage_sp_emu( let out = dir.join("sp-emu"); fs::create_dir_all(&out)?; // Always write the fleet manifest (`rot <0|1>` + ` ` lines) so - // voxel-init knows the SP set, each SP's role, and whether --emu-rot is on — - // even when it boots from the image's BAKED /opt/oxide/sp-emu artifacts + // voxel-init knows the SP set, each SP's role, and whether --emu-rot is on, + // even when it boots from the image's baked /opt/oxide/sp-emu artifacts // (self-contained) rather than these staged copies. (The staged rot.flash was // previously the only signal of --emu-rot; the baked path needs it explicit.) // Fleet manifest: `rot <0|1>` then ` ` per @@ -613,7 +698,7 @@ fn stage_sp_emu( } let ports_manifest = out.join("ports"); fs::write(&ports_manifest, manifest) - .with_context(|| format!("write {}", ports_manifest))?; + .with_context(|| format!("write {ports_manifest}"))?; // Dev override: with [sp].emu_bin set, stage the binary + hubris archives from // the local build for fast iteration (no rebake). Unset -> voxel-init uses the // baked image artifacts. @@ -747,6 +832,33 @@ pub(crate) fn stage_sprockets(cfg: &VoxelConfig) -> anyhow::Result<()> { mod tests { use super::*; + #[test] + fn accepts_and_normalizes_public_keys() { + let out = validate_ssh_pubkey( + "ssh-ed25519 AAAAC3Nza me@host\r\n\nssh-rsa AAAAB3Nza me@host", + ) + .unwrap(); + assert_eq!( + out, + "ssh-ed25519 AAAAC3Nza me@host\nssh-rsa AAAAB3Nza me@host\n" + ); + } + + /// The cargo-bay is mounted into every guest, so a `[falcon].ssh_pubkey` + /// mispointed at the private half must hard-fail, not stage. + #[test] + fn rejects_private_keys_and_non_keys() { + let openssh = "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXk...\n\ + -----END OPENSSH PRIVATE KEY-----\n"; + assert!(validate_ssh_pubkey(openssh).is_err()); + let pem = "-----BEGIN RSA PRIVATE KEY-----\nMIIEow...\n\ + -----END RSA PRIVATE KEY-----\n"; + assert!(validate_ssh_pubkey(pem).is_err()); + assert!(validate_ssh_pubkey("not a key at all\n").is_err()); + assert!(validate_ssh_pubkey("\n\n").is_err()); + assert!(validate_ssh_pubkey("").is_err()); + } + /// A stand-in checkout carrying the two files the sled-schema detection /// reads: the sled-agent config field and the sled-hardware enum variants. fn fake_sled_src(name: &str, field: &str, variants: &str) -> Utf8PathBuf { diff --git a/voxel/src/wicket_setup.rs b/voxel/src/wicket_setup.rs index ce1a7fe..0eab004 100644 --- a/voxel/src/wicket_setup.rs +++ b/voxel/src/wicket_setup.rs @@ -16,7 +16,7 @@ pub(crate) fn dryrun( num_sleds: usize, ) -> Result<()> { let config_rss = std::fs::read_to_string(config_rss_path) - .with_context(|| format!("read {}", config_rss_path))?; + .with_context(|| format!("read {config_rss_path}"))?; // Offline check assumes a single rack (slots 0..n); the live multi-rack slot // set comes from the topology in `drive`. let slots: Vec = (0..num_sleds as u16).collect(); From a397e568991480d42ed1be8caed11e1b0bcdec92 Mon Sep 17 00:00:00 2001 From: Zeeshan Lakhani Date: Thu, 27 Aug 2026 00:14:52 +0000 Subject: [PATCH 2/5] [external-mode] Pin the lan-mode external link from config `lan` mode wired every node's external NIC onto the host's default-route interface unless $EXT_INTERFACE overrode it. On a host whose default route is a public link while the LAN under test sits on a second NIC, that fallback lands the guests on the wrong network, and the env var overrides `isolated` mode's etherstub too, so a persistent export silently rewires `isolated` launches. Now, `[external] link` names the lan-mode link in config instead (`voxel config set external.link igb1`), while we keep `ext_interface`'s precedence: $EXT_INTERFACE first as the escape hatch, then the config link, and then falcon's default-route fallback. The `lan` MTU preflight also checks the same link it would wire up, where it previously ignored the config and probed only the default route. Note: `isolated` mode ignores the key. --- README.md | 10 +++++++--- docs/parameters.md | 3 ++- voxel-config/src/config.rs | 15 +++++++++++++++ voxel/src/rack.rs | 28 +++++++++++++++++----------- voxel/src/topo.rs | 18 ++++++++++++------ 5 files changed, 53 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 12971be..7a7868f 100644 --- a/README.md +++ b/README.md @@ -176,9 +176,13 @@ in place, without fetching or changing its Git state. ## Isolated external network (optional) By default (`[external] mode = "lan"`), every node's external NIC lands on the -host's default-route interface (or `$EXT_INTERFACE`) and leases an address from -whatever DHCP serves the network that link attaches to. That is option 1 ("an -existing IPv4 network") of Omicron's [how-to-run external networking]. +host's default-route interface and leases an address from whatever DHCP serves +the network that link attaches to. That is option 1 ("an existing IPv4 +network") of Omicron's [how-to-run external networking]. When the LAN under +test is not the default-route network (say, a lab segment on a second NIC), +pin the link with `voxel config set external.link igb1` (`$EXT_INTERFACE` +overrides both). + On a host without such a network, voxel can instead build the whole external segment itself, option 2 ("an external network that only exists on your test machine") of the same doc, which a4x2 required the user to plumb by hand. diff --git a/docs/parameters.md b/docs/parameters.md index 7a8fb97..104fb6e 100644 --- a/docs/parameters.md +++ b/docs/parameters.md @@ -76,7 +76,8 @@ rack's RSS config. See the README's "Isolated external network" section. | Key | Type | Default | Notes | |-----|------|---------|-------| -| `mode` | enum | `"lan"` | `lan` attaches node external NICs to the host's default-route link (or `$EXT_INTERFACE`). `isolated` builds the segment on a host etherstub with NAT out `uplink`. | +| `mode` | enum | `"lan"` | `lan` attaches node external NICs to `link`, or the host's default-route link (`$EXT_INTERFACE` overrides both). `isolated` builds the segment on a host etherstub with NAT out `uplink`. | +| `link` | string | unset | Lan-mode external link (e.g. `igb1`), for hosts whose default-route interface is not the LAN under test. Ignored in isolated mode. | | `uplink` | string | unset | Physical link the isolated segment NATs out of (e.g. `igb0`). Required in isolated mode. | | `subnet` | string | `"172.30.199.0/24"` | The isolated segment's subnet, chosen to avoid common home/office LANs. `up` refuses if it overlaps a host address. | | `host_ip` | string | `"172.30.199.199"` | Host address on the segment: the nodes' default gateway and NAT inside address. Image builds also use `host_ip - 1` for the builder VM. | diff --git a/voxel-config/src/config.rs b/voxel-config/src/config.rs index 4a423f3..105964f 100644 --- a/voxel-config/src/config.rs +++ b/voxel-config/src/config.rs @@ -65,6 +65,15 @@ pub enum ExternalMode { pub struct External { /// lan (default; existing behavior) or isolated. pub mode: ExternalMode, + /// Link the nodes' external NICs attach to in lan mode (e.g. igb1), for + /// hosts whose default-route interface is not the LAN under test. + /// + /// `$EXT_INTERFACE` still overrides it. + /// + /// This is ignored in isolated mode, which wires the voxel-managed + /// etherstub. + #[serde(skip_serializing_if = "Option::is_none")] + pub link: Option, /// Physical link the isolated subnet NATs out of (e.g. igb0). Required /// in isolated mode, and validated before use. #[serde(skip_serializing_if = "Option::is_none")] @@ -87,6 +96,7 @@ impl Default for External { fn default() -> Self { Self { mode: ExternalMode::Lan, + link: None, uplink: None, subnet: "172.30.199.0/24".into(), host_ip: "172.30.199.199".into(), @@ -1397,6 +1407,11 @@ mod tests { let cfg = VoxelConfig::from_toml(&out).unwrap(); assert!(cfg.external.isolated()); assert_eq!(cfg.external.uplink.as_deref(), Some("igb0")); + // The lan-mode link pin round-trips and defaults to unset. + assert!(d.external.link.is_none()); + let out = set(&d.to_toml(), "external.link", "igb1").unwrap(); + let cfg = VoxelConfig::from_toml(&out).unwrap(); + assert_eq!(cfg.external.link.as_deref(), Some("igb1")); // deny_unknown_fields catches typos. assert!(set(&out, "external.uplnk", "igb0").is_err()); } diff --git a/voxel/src/rack.rs b/voxel/src/rack.rs index 8ea62cb..1c14fcd 100644 --- a/voxel/src/rack.rs +++ b/voxel/src/rack.rs @@ -110,16 +110,21 @@ fn default_route_iface() -> Option { /// Refuse a `lan`-mode launch whose external link is jumbo. voxel-init classifies /// a sled NIC as underlay iff it accepts mtu=9000. Guest VNICs on a jumbo link /// all pass that probe, so the sleds' external NICs get misclassified as -/// underlay and never come up. Anything below 9000 is fine. Best-effort: if -/// the link or its MTU can't be read we skip and let falcon surface the -/// problem. -fn lan_mtu_preflight() -> anyhow::Result<()> { +/// underlay and never come up. Anything below 9000 is fine. +/// +/// This is best-effort: if the link or its MTU can't be read, we skip and let +/// falcon surface the problem instead. The link checked mirrors +/// `ext_interface`'s precedence: `$EXT_INTERFACE`, then `[external] link`, +/// then the default-route interface. +fn lan_mtu_preflight(cfg: &VoxelConfig) -> anyhow::Result<()> { let link = match std::env::var("EXT_INTERFACE") { Ok(l) => l, - Err(_) => match default_route_iface() { - Some(l) => l, - None => return Ok(()), - }, + Err(_) => { + match cfg.external.link.clone().or_else(default_route_iface) { + Some(l) => l, + None => return Ok(()), + } + } }; if let Some(mtu) = link_mtu(&link) && mtu.parse::().is_ok_and(|m| m >= 9000) @@ -127,8 +132,9 @@ fn lan_mtu_preflight() -> anyhow::Result<()> { bail!( "external link {link} has mtu {mtu}: sled NICs are classified as underlay \ iff they accept mtu=9000, so external NICs on a jumbo link are \ - misclassified and never come up. Point EXT_INTERFACE at a sub-9000-mtu \ - link or use isolated mode (voxel config set external.mode isolated)." + misclassified and never come up. Point [external] link or EXT_INTERFACE \ + at a sub-9000-mtu link or use isolated mode \ + (voxel config set external.mode isolated)." ); } Ok(()) @@ -320,7 +326,7 @@ pub(crate) async fn cmd_launch( external_up(&cfg.external, DryRun::No) .context("bringing up the isolated external segment")?; } else { - lan_mtu_preflight()?; + lan_mtu_preflight(cfg)?; } reset_node_cargo_bay(cfg)?; stage_config(cfg, emu_sp, emu_rot, wicket_setup)?; diff --git a/voxel/src/topo.rs b/voxel/src/topo.rs index 9a8ea3f..1f3daf3 100644 --- a/voxel/src/topo.rs +++ b/voxel/src/topo.rs @@ -44,9 +44,11 @@ impl Topo { } } -/// Wire a node's external NIC. Precedence: `$EXT_INTERFACE` env, then the -/// config-driven link (the voxel-managed stub in isolated mode), then falcon's -/// default (the host's default-route interface). +/// Wire a node's external NIC. +/// +/// Precedence: `$EXT_INTERFACE` env, then the config-driven link (the +/// voxel-managed stub in isolated mode, `[external] link` in lan mode), then +/// falcon's default (the host's default-route interface). fn ext_interface( d: &mut Runner, n: NodeRef, @@ -123,9 +125,13 @@ pub(crate) fn build_topo( } // Isolated mode wires every external NIC onto the voxel-managed etherstub - // instead of the host LAN ($EXT_INTERFACE still wins inside ext_interface). - let ext_if = - cfg.external.isolated().then_some(crate::isolated_external::STUB); + // instead of the host LAN. Lan mode honors `[external] link` when set + // ($EXT_INTERFACE still wins inside ext_interface). + let ext_if = if cfg.external.isolated() { + Some(crate::isolated_external::STUB) + } else { + cfg.external.link.as_deref() + }; let all_scrimlets: Vec = sleds.iter().filter(|(s, _)| s.scrimlet).map(|(_, n)| *n).collect(); From a7cae076c3944da66cd5e0d9720cc66128e49f67 Mon Sep 17 00:00:00 2001 From: Zeeshan Lakhani Date: Thu, 27 Aug 2026 02:40:24 +0000 Subject: [PATCH 3/5] [external-addressing] Static node addresses on a DHCP-less LAN `isolated` mode has fused two concerns since it was introduced: voxel owning the L2 (etherstub, gateway VNIC, NAT) and voxel assigning static node addresses. A fully static lab LAN (e.g., a segment behind a switch in AP mode with DHCP off) needs only the second part: the network already exists, but nothing on it serves DHCP. This means that the `lan` mode's lease-everything approach cannot bring a rack up in that setup. Our change, `[external] addressing = "dhcp" | "static"`, splits the concerns. Static `lan` addressing reuses `isolated` mode's machinery (unchanged), numbering nodes from `ip_start` and staging address + gateway + DNS into each cargo-bay, which `voxel-init` already applies wherever the file appears. Host-side address resolution (`ce_static_ip`, RSS watch known-ip, multicast's `static_ip`) now keys on `static_addressing()`, which is true in `isolated` mode always, meaning `isolated` behavior is unchanged. The image builder follows suit: a `lan`-mode builder attaches to `[external] link` when set, and static addressing derives the same `host_ip - 1` builder address the `isolated` segment uses. This is still following option 1 of omicron's how-to-run external networking, the all-static variant of an existing network, while option 2 remains `isolated` mode. --- Cargo.lock | 62 ++++++++++++++++++++++++++------------ Cargo.toml | 6 ++-- README.md | 8 +++++ docs/multicast.md | 24 +++++++++------ docs/parameters.md | 5 +-- voxel-config/src/config.rs | 57 +++++++++++++++++++++++++++++++++-- voxel/src/imagebuild.rs | 40 ++++++++++++++---------- voxel/src/multicast.rs | 8 ++--- voxel/src/net.rs | 6 ++-- voxel/src/rack.rs | 4 +-- voxel/src/topo.rs | 11 ++++--- 11 files changed, 166 insertions(+), 65 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index d302041..8bdd192 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -122,7 +122,7 @@ checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" [[package]] name = "api_identity" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "omicron-workspace-hack", "proc-macro2", @@ -409,7 +409,7 @@ dependencies = [ [[package]] name = "bootstore" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "bytes", "camino", @@ -438,7 +438,7 @@ dependencies = [ [[package]] name = "bootstrap-agent-lockstep-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "anyhow", "chrono", @@ -1551,7 +1551,7 @@ dependencies = [ [[package]] name = "gateway-types-versions" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "daft", "dropshot", @@ -1619,7 +1619,7 @@ dependencies = [ [[package]] name = "gfss" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "digest 0.10.7", "omicron-workspace-hack", @@ -2766,7 +2766,7 @@ dependencies = [ [[package]] name = "omicron-common" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "anyhow", "api_identity", @@ -2785,10 +2785,10 @@ dependencies = [ "ipnetwork", "itertools 0.14.0", "macaddr", + "omicron-generation-kinds", "omicron-ledger", "omicron-uuid-kinds", "omicron-workspace-hack", - "oxide-generation", "oxnet", "parse-display", "progenitor-client 0.14.0", @@ -2810,10 +2810,19 @@ dependencies = [ "uuid", ] +[[package]] +name = "omicron-generation-kinds" +version = "0.1.0" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" +dependencies = [ + "oxide-generation", + "oxide-generation-macros", +] + [[package]] name = "omicron-ledger" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "async-trait", "atomicwrites", @@ -2830,7 +2839,7 @@ dependencies = [ [[package]] name = "omicron-passwords" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "argon2", "omicron-workspace-hack", @@ -2845,7 +2854,7 @@ dependencies = [ [[package]] name = "omicron-uuid-kinds" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "daft", "newtype-uuid", @@ -2914,6 +2923,20 @@ dependencies = [ "slog", ] +[[package]] +name = "oxide-generation-macros" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68b727e329bc9ff425bf3051e7f39986943bea27aefd6e30b43b468ca999a348" +dependencies = [ + "heck 0.5.0", + "proc-macro2", + "quote", + "serde", + "serde_tokenstream 0.3.0", + "syn 3.0.3", +] + [[package]] name = "oxnet" version = "0.1.6" @@ -3586,7 +3609,7 @@ checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" [[package]] name = "rack-init-config" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "bootstrap-agent-lockstep-types", "iddqd", @@ -4554,7 +4577,7 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "sled-agent-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "anyhow", "async-trait", @@ -4584,7 +4607,7 @@ dependencies = [ [[package]] name = "sled-agent-types-versions" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "anyhow", "async-trait", @@ -4598,6 +4621,7 @@ dependencies = [ "ipnetwork", "itertools 0.14.0", "omicron-common", + "omicron-generation-kinds", "omicron-ledger", "omicron-passwords", "omicron-uuid-kinds", @@ -4623,7 +4647,7 @@ dependencies = [ [[package]] name = "sled-hardware-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "daft", "omicron-workspace-hack", @@ -5521,7 +5545,7 @@ dependencies = [ [[package]] name = "trust-quorum-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "omicron-workspace-hack", "trust-quorum-types-versions", @@ -5530,7 +5554,7 @@ dependencies = [ [[package]] name = "trust-quorum-types-versions" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "byte-wrapper", "daft", @@ -6118,7 +6142,7 @@ dependencies = [ [[package]] name = "wicketd-commission-client" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "iddqd", "omicron-uuid-kinds", @@ -6136,7 +6160,7 @@ dependencies = [ [[package]] name = "wicketd-commission-types" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "omicron-workspace-hack", "wicketd-commission-types-versions", @@ -6145,7 +6169,7 @@ dependencies = [ [[package]] name = "wicketd-commission-types-versions" version = "0.1.0" -source = "git+https://github.com/oxidecomputer/omicron?rev=521e1b3903a3e4a1652a15395e0922723fc90cdc#521e1b3903a3e4a1652a15395e0922723fc90cdc" +source = "git+https://github.com/oxidecomputer/omicron?rev=a41832147bb7fe1517ad8409983b1a9a03ae6691#a41832147bb7fe1517ad8409983b1a9a03ae6691" dependencies = [ "gateway-types-versions", "iddqd", diff --git a/Cargo.toml b/Cargo.toml index 8091f13..4be9f51 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -11,9 +11,9 @@ members = [ anyhow = "1.0.102" # The omicron commit voxel is pinned to. Bumping this rev is the new-omicron # motion; build.rs surfaces it to voxel image create. Keep all revs identical. -rack-init-config = { git = "https://github.com/oxidecomputer/omicron", rev = "521e1b3903a3e4a1652a15395e0922723fc90cdc" } -wicketd-commission-types-versions = { git = "https://github.com/oxidecomputer/omicron", rev = "521e1b3903a3e4a1652a15395e0922723fc90cdc" } -wicketd-commission-client = { git = "https://github.com/oxidecomputer/omicron", rev = "521e1b3903a3e4a1652a15395e0922723fc90cdc" } +rack-init-config = { git = "https://github.com/oxidecomputer/omicron", rev = "a41832147bb7fe1517ad8409983b1a9a03ae6691" } +wicketd-commission-types-versions = { git = "https://github.com/oxidecomputer/omicron", rev = "a41832147bb7fe1517ad8409983b1a9a03ae6691" } +wicketd-commission-client = { git = "https://github.com/oxidecomputer/omicron", rev = "a41832147bb7fe1517ad8409983b1a9a03ae6691" } clap = { version = "4.6.1", features = ["derive", "env"] } expectorate = "1.2.0" # Our crates use only #[tokio::main] + tokio::time; libfalcon pulls its own diff --git a/README.md b/README.md index 7a7868f..25ff227 100644 --- a/README.md +++ b/README.md @@ -183,6 +183,14 @@ test is not the default-route network (say, a lab segment on a second NIC), pin the link with `voxel config set external.link igb1` (`$EXT_INTERFACE` overrides both). +A LAN that runs no DHCP at all (a fully static lab segment) can keep the +nodes static instead: `voxel config set external.addressing static` with +`subnet`, `host_ip` (the LAN's existing gateway), and `ip_start` set to that +LAN's values. This is still [option 1][how-to-run external networking], the +network exists and addresses are carved from it, while voxel just stages each +node's address exactly as isolated mode does below, without owning the +network. + On a host without such a network, voxel can instead build the whole external segment itself, option 2 ("an external network that only exists on your test machine") of the same doc, which a4x2 required the user to plumb by hand. diff --git a/docs/multicast.md b/docs/multicast.md index 366d738..07dde19 100644 --- a/docs/multicast.md +++ b/docs/multicast.md @@ -67,10 +67,10 @@ Omicron commit pulls the whole set. | Repository | Branch / rev | PR | Carried by | Trajectory | | --- | --- | --- | --- | --- | -| omicron | `zl/mcast-build` leaf | #11128 open, top of the PR stack below | workspace `rack-init-config` pin, via `voxel image create` | stack lands bottom-up into `main`, starting at #9912 | -| dendrite | `multicast-e2e`, `3d49b131` | #224 open | Omicron `tools/dendrite_*` pins | #224 to `main` | -| maghemite | `zl/ddm-mcast`, `96a2f153` | #696 open, stacked on `zl/mrib`; related #402 (`zl/mgd-ddm-meta`) | Omicron `tools/maghemite_*` pins | #696 to `main` after `zl/mrib` | -| opte | `master`, `0525f2f95` (0.41.506) | #1012 merged 2026-07-25 | Omicron `Cargo.toml` / `tools/opte_version` | landed; Omicron pins 0.41.506 intentionally (master's later #1040 is a comment-only xde change) | +| omicron | `zl/mcast-build` leaf, [`a4183214`](https://github.com/oxidecomputer/omicron/pull/11128/commits/a41832147bb7fe1517ad8409983b1a9a03ae6691) | #11128 open, top of the PR stack below | workspace `rack-init-config` pin, via `voxel image create` | stack lands bottom-up into `main`, starting at #9912 | +| dendrite | `multicast-e2e`, `447aa04f` | #224 open | Omicron `tools/dendrite_*` pins | #224 to `main` | +| maghemite | `zl/ddm-mcast`, `36e43651` | #696 open, stacked on `zl/mrib`; related #402 (`zl/mgd-ddm-meta`) | Omicron `tools/maghemite_*` pins | #696 to `main` after `zl/mrib` | +| opte | `master`, `0525f2f95` (0.41.506) | #1012 merged 2026-07-25; #1049 (`zl/mcast-source-validation`, `3e2615e5`) open | Omicron `Cargo.toml` / `tools/opte_version` | landed; Omicron pins 0.41.506 intentionally (master's later #1040 is a comment-only xde change). #1049 aligns source-address validation with dpd, nexus and mgd, and is not pinned yet | | propolis | `zl/multicast`, `3c07d60a` | [#1093] open | Omicron `package-manifest.toml` (guest propolis-server) | [#1093] to `master`; host side also needs the viona V7 tables below | | thundermuffin | `zl/multicast-joiner`, `486559bc` | #14 open | Omicron `package-manifest.toml` (probe zone, prebuilt) | #14 to `main` | | sidecar-lite | `zl/multicast`, `461cbe19` | #152 open | `voxel image create` (`SIDECAR_LITE_REV`) | #152 to `main` | @@ -108,12 +108,15 @@ Two pieces sit outside the image, on the host itself: group frame. It also needs the SMBIOS type 1 fix from [#1200], which is not multicast-specific. Without it falcon's a4x2 identity never reaches the sled VM and RSS fails trust quorum validation on any voxel rack. The - `mcast-smbios-test` branch (`8bb6a90b`) merges both. + `mcast-smbios-test` branch (`8e283bee`) merges those two along with the + softnpu management-uart deadlock fix from [#1206]. That branch is the local + integration point and is never itself PR'd, so each fix is proposed + upstream from its own branch off `master`. Build it with the `falcon` feature: ```sh - git checkout mcast-smbios-test # zl/multicast + propolis#1200 + git checkout mcast-smbios-test # zl/multicast + propolis#1200 + #1206 cargo build --release --bin propolis-server --features falcon ``` @@ -201,9 +204,11 @@ A pool carries a `pool_type` discriminator, `unicast` (the default) or - Every range in the pool must be entirely Any-Source Multicast (ASM) or entirely Source-Specific Multicast (SSM), never both. SSM is `232.0.0.0/8` for IPv4 and the per-scope `ff3x::/32` blocks for IPv6 ([RFC 4607]); - everything else is ASM. An ASM group set and an SSM group set therefore - need two pools. The split is about address space, not filtering: joins on - ASM addresses may still carry `source_ips` (see [Join forms](#join-forms)). + everything else is ASM. Within the v4 range, `232.0.0.0/24` is refused, + reserved by [RFC 4607] §4.3, so a v4 SSM pool starts at `232.0.1.0`. An + ASM group set and an SSM group set therefore need two pools. The split is + about address space, not filtering: joins on ASM addresses may still carry + `source_ips` (see [Join forms](#join-forms)). - A silo may hold at most one default pool per (pool type, IP version) pair, four in total. A multicast pool linked non-default is still usable; the group is then resolved by address rather than by the silo default. @@ -870,6 +875,7 @@ from this host-sourced path. [#1093]: https://github.com/oxidecomputer/propolis/pull/1093 [#1200]: https://github.com/oxidecomputer/propolis/pull/1200 +[#1206]: https://github.com/oxidecomputer/propolis/pull/1206 [omicron#11128]: https://github.com/oxidecomputer/omicron/pull/11128 [omicron#11118]: https://github.com/oxidecomputer/omicron/pull/11118 [omicron#10520]: https://github.com/oxidecomputer/omicron/pull/10520 diff --git a/docs/parameters.md b/docs/parameters.md index 104fb6e..a043c71 100644 --- a/docs/parameters.md +++ b/docs/parameters.md @@ -77,10 +77,11 @@ rack's RSS config. See the README's "Isolated external network" section. | Key | Type | Default | Notes | |-----|------|---------|-------| | `mode` | enum | `"lan"` | `lan` attaches node external NICs to `link`, or the host's default-route link (`$EXT_INTERFACE` overrides both). `isolated` builds the segment on a host etherstub with NAT out `uplink`. | +| `addressing` | enum | `"dhcp"` | `dhcp` leases node addresses from the LAN. `static` stages per-node addresses from `ip_start` for a LAN that runs no DHCP. Ignored in isolated mode, which is always static. | | `link` | string | unset | Lan-mode external link (e.g. `igb1`), for hosts whose default-route interface is not the LAN under test. Ignored in isolated mode. | | `uplink` | string | unset | Physical link the isolated segment NATs out of (e.g. `igb0`). Required in isolated mode. | -| `subnet` | string | `"172.30.199.0/24"` | The isolated segment's subnet, chosen to avoid common home/office LANs. `up` refuses if it overlaps a host address. | -| `host_ip` | string | `"172.30.199.199"` | Host address on the segment: the nodes' default gateway and NAT inside address. Image builds also use `host_ip - 1` for the builder VM. | +| `subnet` | string | `"172.30.199.0/24"` | The static addressing subnet: the isolated segment's (chosen to avoid common home/office LANs; `up` refuses if it overlaps a host address), or the LAN's under `addressing = "static"`. | +| `host_ip` | string | `"172.30.199.199"` | The nodes' default gateway. Isolated mode creates it on the etherstub (also the NAT inside address); static lan addressing expects it to already exist on the LAN. Image builds also use `host_ip - 1` for the builder VM. | | `ip_start` | string | `"172.30.199.10"` | First static node address. Nodes number contiguously, sleds then `topology.routers`. | | `dns` | list | `["1.1.1.1", "9.9.9.9"]` | Nameservers handed to the nodes. | | `mtu` | int | `1500` | Etherstub MTU. Must stay below 9000 so voxel-init's jumbo probe classifies external NICs correctly. | diff --git a/voxel-config/src/config.rs b/voxel-config/src/config.rs index 105964f..c8bad20 100644 --- a/voxel-config/src/config.rs +++ b/voxel-config/src/config.rs @@ -58,6 +58,20 @@ pub enum ExternalMode { Isolated, } +/// How nodes get their external addresses. +#[derive(Debug, Clone, Copy, PartialEq, Default, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ExternalAddressing { + /// Lease from whatever DHCP serves the external network (default). + #[default] + Dhcp, + /// Stage static per-node addresses from `ip_start`, for a LAN that runs + /// no DHCP. + /// + /// Isolated mode always addresses statically. + Static, +} + /// The rack's external segment. Host-only plumbing; never reaches the rack's /// RSS config. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] @@ -65,6 +79,9 @@ pub enum ExternalMode { pub struct External { /// lan (default; existing behavior) or isolated. pub mode: ExternalMode, + /// dhcp (default) or static. Ignored in isolated mode, which is always + /// static. + pub addressing: ExternalAddressing, /// Link the nodes' external NICs attach to in lan mode (e.g. igb1), for /// hosts whose default-route interface is not the LAN under test. /// @@ -78,9 +95,12 @@ pub struct External { /// in isolated mode, and validated before use. #[serde(skip_serializing_if = "Option::is_none")] pub uplink: Option, - /// The isolated segment's subnet. + /// The static addressing subnet: the isolated segment's, or the LAN's + /// under `addressing = "static"`. pub subnet: String, - /// Host address on the etherstub and the nodes' default gateway. + /// The nodes' default gateway. Isolated mode creates it as the host's + /// address on the etherstub; static lan addressing expects it to already + /// exist on the LAN (e.g. the host's own address on `link`). pub host_ip: String, /// First static node address; nodes number contiguously from here in /// sleds() then routers order. @@ -96,6 +116,7 @@ impl Default for External { fn default() -> Self { Self { mode: ExternalMode::Lan, + addressing: ExternalAddressing::Dhcp, link: None, uplink: None, subnet: "172.30.199.0/24".into(), @@ -118,6 +139,12 @@ impl External { self.mode == ExternalMode::Isolated } + /// Whether nodes get staged static external addresses rather than DHCP + /// leases: isolated mode always, lan mode when `addressing = "static"`. + pub fn static_addressing(&self) -> bool { + self.isolated() || self.addressing == ExternalAddressing::Static + } + /// Prefix length parsed from subnet; None if it is not CIDR. pub fn prefix_length(&self) -> Option { Some(u32::from(self.subnet_net()?.width())) @@ -1416,6 +1443,32 @@ mod tests { assert!(set(&out, "external.uplnk", "igb0").is_err()); } + #[test] + fn static_addressing_follows_mode_and_config() { + // Default lan mode leases. + let d = External::default(); + assert!(!d.static_addressing()); + // Lan mode goes static via the knob, without becoming isolated. + let lan_static = External { + addressing: ExternalAddressing::Static, + ..External::default() + }; + assert!(lan_static.static_addressing() && !lan_static.isolated()); + // Isolated mode is always static, whatever the knob says. + let isolated = + External { mode: ExternalMode::Isolated, ..External::default() }; + assert!(isolated.static_addressing()); + // The knob round-trips through voxel config set. + let out = set( + &VoxelConfig::default().to_toml(), + "external.addressing", + "static", + ) + .unwrap(); + let cfg = VoxelConfig::from_toml(&out).unwrap(); + assert!(cfg.external.static_addressing() && !cfg.external.isolated()); + } + #[test] fn cp_commit_strips_prefix_and_variant_suffix() { let mut img = Image { diff --git a/voxel/src/imagebuild.rs b/voxel/src/imagebuild.rs index 4661736..d334067 100644 --- a/voxel/src/imagebuild.rs +++ b/voxel/src/imagebuild.rs @@ -61,27 +61,35 @@ pub(crate) struct BuilderNetwork { /// The builder normally DHCPs an external NIC. In isolated mode that network is /// the voxel-managed segment, which runs no DHCP, so the segment is brought up /// here and the builder gets the stub plus a static address derived from -/// `host_ip - 1`. In lan mode falcon's default link and DHCP already work, so -/// this is empty. +/// `host_ip - 1`. A lan-mode builder uses `[external] link` when set, +/// with the same `host_ip - 1` static address under static addressing. Under +/// DHCP addressing, falcon's default link and a lease already work. pub(crate) fn builder_network( external: Option<&voxel_config::External>, ) -> Result { - let Some(x) = external.filter(|x| x.isolated()) else { + let Some(x) = external else { return Ok(BuilderNetwork::default()); }; - isolated_external::up(x, isolated_external::DryRun::No) - .context("bringing up the isolated external segment for the builder")?; - let address = x.builder_net().with_context(|| { - format!( - "cannot derive a usable isolated builder address below host_ip '{}' within \ - subnet '{}'; choose a host_ip at least two addresses above the subnet network", - x.host_ip, x.subnet - ) - })?; - Ok(BuilderNetwork { - interface: Some(isolated_external::STUB.to_string()), - static_address: Some(address), - }) + let interface = if x.isolated() { + isolated_external::up(x, isolated_external::DryRun::No).context( + "bringing up the isolated external segment for the builder", + )?; + Some(isolated_external::STUB.to_string()) + } else { + x.link.clone() + }; + let static_address = if x.static_addressing() { + Some(x.builder_net().with_context(|| { + format!( + "cannot derive a usable static builder address below host_ip '{}' within \ + subnet '{}'; choose a host_ip at least two addresses above the subnet network", + x.host_ip, x.subnet + ) + })?) + } else { + None + }; + Ok(BuilderNetwork { interface, static_address }) } /// Bring up the builder, install, quiesce, capture, tear down. diff --git a/voxel/src/multicast.rs b/voxel/src/multicast.rs index 26cabcf..c01fd7c 100644 --- a/voxel/src/multicast.rs +++ b/voxel/src/multicast.rs @@ -291,12 +291,12 @@ fn mirror_router(cfg: &VoxelConfig) -> anyhow::Result { .context("topology.routers has no fabric router to mirror from") } -/// A node's external address as voxel assigned it. `None` outside isolated -/// mode, where addresses are leased and only discoverable from the running -/// node. +/// A node's external address as voxel assigned it. This is `None` under DHCP +/// addressing, where addresses are leased and only discoverable from the +/// running node. fn static_ip(cfg: &VoxelConfig, node: &str) -> Option { cfg.external - .isolated() + .static_addressing() .then(|| { cfg.static_external_ips() .into_iter() diff --git a/voxel/src/net.rs b/voxel/src/net.rs index b9b37af..4f1c6e6 100644 --- a/voxel/src/net.rs +++ b/voxel/src/net.rs @@ -146,7 +146,7 @@ pub(crate) async fn resolve_external_ip( n: NodeRef, is_router: bool, ) -> anyhow::Result { - if cfg.external.isolated() + if cfg.external.static_addressing() && let Some((_, ip)) = cfg.static_external_ips().into_iter().find(|(name, _)| name == node) { @@ -156,13 +156,13 @@ pub(crate) async fn resolve_external_ip( } /// ce's stable nexthop, when one is known without touching the guest. An -/// explicit `[topology].ce_external_ip` wins, otherwise isolated mode's static +/// explicit `[topology].ce_external_ip` wins, otherwise static addressing's /// numbering supplies it. pub(crate) fn ce_static_ip(cfg: &voxel_config::VoxelConfig) -> Option { if let Some(ip) = &cfg.topology.ce_external_ip { return Some(ip.clone()); } - if !cfg.external.isolated() { + if !cfg.external.static_addressing() { return None; } cfg.static_external_ips() diff --git a/voxel/src/rack.rs b/voxel/src/rack.rs index 1c14fcd..9d690d8 100644 --- a/voxel/src/rack.rs +++ b/voxel/src/rack.rs @@ -440,7 +440,7 @@ pub(crate) async fn cmd_launch( ); } let watch_cap = rss_watch_cap(emu_sp, racks); - let known_ip = if cfg.external.isolated() { + let known_ip = if cfg.external.static_addressing() { cfg.static_external_ips() .into_iter() .find(|(name, _)| name == &s.name) @@ -743,7 +743,7 @@ pub(crate) async fn cmd_status( // Multi-rack racks converge under each other's load - watch longer (matches // cmd_launch). Duration is Copy, so each watcher closure gets its own. let watch_cap = rss_watch_cap(false, racks); - let ips = if cfg.external.isolated() { + let ips = if cfg.external.static_addressing() { cfg.static_external_ips() } else { Vec::new() diff --git a/voxel/src/topo.rs b/voxel/src/topo.rs index 1f3daf3..c30a3a4 100644 --- a/voxel/src/topo.rs +++ b/voxel/src/topo.rs @@ -484,11 +484,12 @@ pub(crate) fn stage_config( fs::write(dir.join("ce-external-ip"), ip)?; } - // Isolated mode: no DHCP server; instead stage each node's assigned static - // address into its cargo-bay. voxel-init picks it up on both sled and router - // roles. The router role also needs the interface name (routers can't jumbo- - // probe their way to it the way sleds do); sleds self-classify. - if cfg.external.isolated() { + // Static addressing (isolated mode, or a lan without DHCP): stage each + // node's assigned address into its cargo-bay. voxel-init picks it up on + // both sled and router roles. The router role also needs the interface + // name (routers can't jumbo-probe their way to it the way sleds do); + // sleds self-classify. + if cfg.external.static_addressing() { let prefix = cfg.external.prefix_length().ok_or_else(|| { anyhow!( "[external].subnet '{}' must be CIDR (a.b.c.d/len)", From 5c18155997bfc3132369bb2c2f5b9cd7a28f5d8e Mon Sep 17 00:00:00 2001 From: Zeeshan Lakhani Date: Mon, 31 Aug 2026 15:57:25 +0000 Subject: [PATCH 4/5] [docs] refresh the multicast deps table --- docs/multicast.md | 84 +++++++++++++++++++++++++---------------------- 1 file changed, 45 insertions(+), 39 deletions(-) diff --git a/docs/multicast.md b/docs/multicast.md index 07dde19..31f2b62 100644 --- a/docs/multicast.md +++ b/docs/multicast.md @@ -1,16 +1,14 @@ # Multicast on a voxel rack -Three API objects express multicast on an Oxide rack: a multicast IP pool, a -group, and its members. An operator creates only the pool directly. Nexus -implicitly materializes a group when the first member joins an address the pool -covers, and reaps it when the last member leaves. This document is a reference -for those objects as they behave on a voxel rack, then covers the host-side -topology management that gets externally sourced multicast traffic into the -rack at all. +Multicast uses three API objects on an Oxide rack: an IP pool, a group, and its +members. An operator creates the pool. Nexus creates a group when the first +member joins an address covered by the pool, and removes it after the last +member leaves. This document also describes the host topology that carries +externally sourced multicast into the rack. The traffic here originates on the **host**, not in a guest, so these runs -exercise the external-to-underlay ingress path, from host through to switch and -to sled for each subscribing member. Nothing described below drives the +test the external-to-underlay ingress path, from host through switch to sled for +each subscribing member. This document does not cover the guest-sourced egress path (or OPTE's sled-side next-hop selection). Exercising that path would take a sender from inside the rack, where, here, the members only answer: a probe's echo reply is unicast, so no guest originates multicast @@ -22,7 +20,7 @@ underlay addresses for materialized groups alone, so a report is denied rather than encapsulated and the sled-side next-hop selection never runs. The rack drives forwarding from the API subscription, not from the report. Consuming it instead is what [RFD 488]'s dynamic group identification (IGMP snooping and -querying) proposes, and nothing here exercises that as of yet. +querying) proposes. This document does not cover that mode. `voxel commtest --traffic multi` performs every API step below automatically. The API sections document what it creates, and how to create the same objects @@ -52,8 +50,8 @@ by hand when taking a run apart. ### Dependencies -> TODO: the branches below are in flight and will collapse into their main -> branches as they land upstream. Until then, the Omicron pin is the pushed leaf +> TODO: These branches will return to their main branches as they land upstream. +> Until then, the Omicron pin is the pushed leaf > of `zl/mcast-build`, held by the workspace `rack-init-config` dependency in > `Cargo.toml`; `build.rs` surfaces this bookmark to `voxel image create`, > so that a commitless `voxel image create` builds the pin. @@ -62,21 +60,27 @@ The multicast stack spans multiple repositories, but voxel itself pins only two of them: the Omicron commit handed to `voxel image create` (`OMICRON_REPO` selects the clone source) and the sidecar-lite artifact rev the build fetches (`SIDECAR_LITE_REV`, already defaulted to the multicast rev). Everything else -rides the chosen Omicron commit's own pins, so pointing the image at the right -Omicron commit pulls the whole set. +comes from the chosen Omicron commit's own pins. Selecting the correct Omicron +commit selects the rest of the stack. -| Repository | Branch / rev | PR | Carried by | Trajectory | +| Repository | Branch / rev | PR | Carried by | Merge path | | --- | --- | --- | --- | --- | | omicron | `zl/mcast-build` leaf, [`a4183214`](https://github.com/oxidecomputer/omicron/pull/11128/commits/a41832147bb7fe1517ad8409983b1a9a03ae6691) | #11128 open, top of the PR stack below | workspace `rack-init-config` pin, via `voxel image create` | stack lands bottom-up into `main`, starting at #9912 | | dendrite | `multicast-e2e`, `447aa04f` | #224 open | Omicron `tools/dendrite_*` pins | #224 to `main` | | maghemite | `zl/ddm-mcast`, `36e43651` | #696 open, stacked on `zl/mrib`; related #402 (`zl/mgd-ddm-meta`) | Omicron `tools/maghemite_*` pins | #696 to `main` after `zl/mrib` | | opte | `master`, `0525f2f95` (0.41.506) | #1012 merged 2026-07-25; #1049 (`zl/mcast-source-validation`, `3e2615e5`) open | Omicron `Cargo.toml` / `tools/opte_version` | landed; Omicron pins 0.41.506 intentionally (master's later #1040 is a comment-only xde change). #1049 aligns source-address validation with dpd, nexus and mgd, and is not pinned yet | -| propolis | `zl/multicast`, `3c07d60a` | [#1093] open | Omicron `package-manifest.toml` (guest propolis-server) | [#1093] to `master`; host side also needs the viona V7 tables below | +| propolis (rack image) | `zl/multicast`, `3c07d60a` | [#1093] open | Omicron `package-manifest.toml` (guest propolis-server) | [#1093] to `master` | | thundermuffin | `zl/multicast-joiner`, `486559bc` | #14 open | Omicron `package-manifest.toml` (probe zone, prebuilt) | #14 to `main` | | sidecar-lite | `zl/multicast`, `461cbe19` | #152 open | `voxel image create` (`SIDECAR_LITE_REV`) | #152 to `main` | | softnpu | `zl/multicast`, `284c6830` | #183 open | Omicron `tools/softnpu_version` (xtask `SOFTNPU_COMMIT`) | #183 to `main` | | p4 | `zl/multicast` (`p4rs`) | #240 open | transitive, via softnpu and sidecar-lite lockfiles | #240 to `main`, then #183/#152 repoint | +The propolis row above describes only the server shipped inside the rack image. +The host-side propolis falcon runs is a separate integration checkout, the local +`mcast-smbios-test` branch (`8e283bee`), selected by `[falcon].propolis_binary` +rather than by the Omicron package manifest. What it carries and how to build it +are covered with the host-side pieces below. + The omicron work is a stack of PRs, each based on the branch below it. Voxel pins the leaf, so one commit carries the whole chain: @@ -99,19 +103,20 @@ frame toward the rack. Probe-based commtest requires both. Two pieces sit outside the image, on the host itself: -- The **host propolis** needs propolis `zl/multicast`'s viona MAC-filter - wiring (PR [#1093] above), or the sled VMs receive no multicast at all. +- The **host propolis** needs the `mcast-smbios-test` integration branch listed + above. It combines propolis `zl/multicast`'s viona MAC-filter wiring (PR + [#1093]), the SMBIOS type 1 fix from [#1200], and the softnpu management-UART + deadlock fix from [#1206]. Without the MAC-filter wiring, the sled VMs + receive no multicast at all. + This is the falcon VM boundary, not the rack's guest instances: each sled is itself a propolis VM whose illumos `vioif` negotiates `VIRTIO_NET_F_CTRL_RX` but never programs a multicast table, so viona narrows the link to no-multicast at feature negotiation and drops every - group frame. It also needs the SMBIOS type 1 fix from [#1200], which is - not multicast-specific. Without it falcon's a4x2 identity never reaches - the sled VM and RSS fails trust quorum validation on any voxel rack. The - `mcast-smbios-test` branch (`8e283bee`) merges those two along with the - softnpu management-uart deadlock fix from [#1206]. That branch is the local - integration point and is never itself PR'd, so each fix is proposed - upstream from its own branch off `master`. + group frame. The SMBIOS fix is not multicast-specific: without it falcon's + a4x2 identity never reaches the sled VM and RSS fails trust quorum validation + on any voxel rack. The integration branch is local and is never itself PR'd, + so each fix is proposed upstream from its own branch off `master`. Build it with the `falcon` feature: @@ -379,8 +384,8 @@ A probe is the lightest member primitive: it needs no guest, its create request pins it to a named sled, and it auto-replies to an echo request sent to a group it has joined, so that a plain `ping` observes delivery. This is why `commtest` uses probes, and why they are the easiest member to exercise -manually. *Note* that memberships are fixed at creation. To change them, recreate -the probe. +manually. *Note* that memberships are fixed at creation. To change them, +recreate the probe. ```sh sleds=$(oxide api /v1/system/hardware/sleds | jq -r '.items[].id') @@ -496,8 +501,8 @@ routers. `voxel launch`, not before: it resolves `ce`'s address and reaches `cr1` over SSH, both of which need running nodes. **Run it again after every launch**. The mirror filter and the link-layer membership are runtime state inside the `cr1` -VM and go away when it does, so only the host routes carry across, which is what -the ownership record described below exists to track. The command is idempotent, +VM and go away when it does, so only the host routes carry across, which is what +the ownership record described below exists to track. The command is idempotent, so re-running it over the same groups costs nothing. ### Host routes @@ -528,17 +533,17 @@ for group in 239.100.0.1 239.100.0.2 232.100.0.1; do done ``` -The route table belongs to the Helios host rather than to any one, specific +The route table belongs to the Helios host rather than to any one, specific Falcon environment, so voxel records each group's gateway in `.falcon/multicast-.json` and treats that record, not the gateway address, as proof of ownership. `up` writes the record before adding the route. An interrupted run then leaves a record with no route, which the next `up` overwrites and a groupless `down` reads as nothing to remove. A reverse ordering would leave a route that nothing afterwards could prove was -voxel's to begin with. Both commands stop rather than guess when a group's -route is not in the record: `up` refuses the group instead of displacing the -route, and `down` leaves it alone and names it. Deleting the record while its -routes exist, therefore, locks voxel out of those groups until they are removed +voxel's to begin with. Both commands stop rather than guess when a group's +route is not in the record: `up` refuses the group instead of displacing the +route, and `down` leaves it alone and names it. Deleting the record while its +routes exist, therefore, locks voxel out of those groups until they are removed by hand with `pfexec route delete -host `. Two isolated-mode environments sharing an `[external]` subnet are the one case @@ -842,8 +847,9 @@ state it would normally learn from PIM or IGMP. Two static substitutes: What voxel does not model is the dynamic case: a customer network where PIM and IGMP are running end to end, with the upstream building its distribution -tree from receiver membership. That is the direction [RFD 488]'s IGMP host-proxying -(future work) targets, and nothing here exercises or validates that signaling. +tree from receiver membership. That is the direction [RFD 488]'s IGMP +host-proxying (future work) targets, and nothing here exercises or validates +that signaling. > TODO: when [RFD 488]'s host-proxying lands, exercising it here means > replacing the static scaffolding for that mode. The emulated upstream would @@ -866,9 +872,9 @@ tree from receiver membership. That is the direction [RFD 488]'s IGMP host-proxy | `up` reports a host route "is not recorded for Falcon environment" | The route predates this environment's `.falcon/` record, either from another environment or from a record that was deleted while its routes remained. Voxel will not displace it. Remove it with `pfexec route delete -host ` if it is stale. | | Every per-group artifact checks out and delivery still fails | The sled dataplane. Run opte's [`opte-mcast-delivery.d`] in a sled's global zone, where `xde` is loaded. `NOFWD` names a missing forwarding entry, `FILTERED` a source-filter drop, and the delivery matrix reports which ports took a copy. The script is not in the sled image, so copy it from an opte checkout. | -*TODO*: IPv6 multicast is not wired up in `commtest` yet, and the isolated external -segment voxel creates is v4-only. The API objects are not the gap: a `v6` -multicast pool and its groups work like their v4 counterparts. `lan` mode +*TODO*: IPv6 multicast is not wired up in `commtest` yet, and the isolated +external segment voxel creates is v4-only. The API objects are not the gap: a +`v6` multicast pool and its groups work like their v4 counterparts. `lan` mode also inherits whatever v6 the LAN carries, so what is missing is voxel's wiring, not the rack or the topology. Until then, v6 groups are out of reach from this host-sourced path. From 6683df1d2c89530511bfb89df9606d7fd7f08564 Mon Sep 17 00:00:00 2001 From: Zeeshan Lakhani Date: Mon, 31 Aug 2026 16:17:04 +0000 Subject: [PATCH 5/5] [launch] re-provision sled disks on a boot retry The BOOT_ATTEMPTS loop tears the deployment down before retrying, and that teardown's `zfs destroy -r` reaps the sled zvols. What happend: create_zvols ran once, before the loop, so then retries rebuilt a topology whose NVMe backends pointed at destroyed media and every sled died with "failed to read file backend metadata". Now, we re-create the disks after the teardown. Also, we document the `lan`-mode source addresses for the multi-group commtest run, which the existing example gave only for the isolated segment. --- docs/multicast.md | 21 +++++++++++++++++++++ voxel/src/rack.rs | 9 +++++++++ 2 files changed, 30 insertions(+) diff --git a/docs/multicast.md b/docs/multicast.md index 31f2b62..fc759ff 100644 --- a/docs/multicast.md +++ b/docs/multicast.md @@ -729,6 +729,27 @@ The deny source is any address other than the host's, here `172.30.199.198`, so the joined filter excludes the actual sender and the dataplane must drop the traffic. +Those source addresses belong to the isolated segment. In `lan` mode the +sender is the host on the LAN itself, so both take addresses from +`[external]`: the real source is `host_ip`, and the deny source is any other +address on `subnet` that the sender does not hold. With the static LAN +configured here (`subnet = 192.168.1.0/24`, `host_ip = 192.168.1.199`): + +```sh +voxel network multicast up --group 239.100.0.1 --group 239.100.0.2 \ + --group 232.100.0.1 --group 239.100.0.9 + +voxel commtest --source /oxide/workspace/omicron --traffic multi -- run \ + --test-duration 200s --warmup 10s --packet-rate 10 \ + --mcast-group 239.100.0.1 \ + --mcast-group 239.100.0.2@192.168.1.199 \ + --mcast-group 232.100.0.1@192.168.1.199 \ + --mcast-deny-group 239.100.0.9@192.168.1.198 +``` + +The group addresses do not change with the mode, since they are rack-side +pool addresses rather than LAN addresses. Only the sources move around. + With no `--mcast-group`, voxel supplies `239.1.1.1`, which then needs its own host route, mirror filter, and link-layer membership. See the commtest section of the [README] for the build and privilege details. diff --git a/voxel/src/rack.rs b/voxel/src/rack.rs index 6c85a38..f2bed77 100644 --- a/voxel/src/rack.rs +++ b/voxel/src/rack.rs @@ -355,6 +355,15 @@ pub(crate) async fn cmd_launch( ); let _ = teardown(&topo.runner, name); std::thread::sleep(std::time::Duration::from_secs(3)); + // teardown's `zfs destroy -r` reaps the sled media along with + // the rest of the deployment, so a retry that skipped this + // would hand propolis backend paths that no longer resolve. + crate::disks::create_zvols( + &crate::image::falcon_dataset(), + name, + &sleds, + ) + .context("recreating sled disks for boot retry")?; topo = build_topo(cfg, name)?; attempt += 1; }