macOS (Apple Silicon) backend: SNO on vfkit + vmnet-helper - #16
Merged
raghavendra-talur merged 22 commits intoAug 25, 2026
Merged
Conversation
Approved design for provisioning arm64 SNO clusters on Apple Silicon: vfkit (compute) + vmnet-helper shared-mode networking, network/PXE-style boot reusing the :9393 ignition fileserver, Rosetta for x86-64 workloads via virtiofs + binfmt, and runtime.GOOS-dispatched provider wiring. Scopes v1 to NAT + multi-cluster DR parity; defers bridge mode and real DNS. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Resolve the four open questions: reuse 192.168.126.0/24, pin per-cluster IPs via the existing ignition static keyfile (no dependence on vmnet DHCP reservations), require vfkit/vmnet-helper on PATH like virsh, and accept a user-performed vmnet-helper privilege setup. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Bite-sized TDD plan for the CI-verifiable slice: arch parameterization, build-tagged host inspector, providers/vfkit + providers/vmnethelper, createnetwork rename, VMManager-based status, PXE-assets stage + Rosetta fragment (artifacts), and runtime.GOOS dispatch. Real-hardware boot/DR validation scoped to a follow-up plan. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
The dev box is the target hardware (macOS 26.5.1, Apple Silicon) with vfkit 0.6.3 + vmnet-helper 0.12.0 installed. Fixes: vmnet-helper is a per-VM sidecar (requires --fd/--socket), not a standalone daemon, so VM start and network attach are coupled inside the vfkit/vmnethelper providers; it lives under brew libexec, not PATH; privilege is the shipped sudoers rule with a path-mismatch caveat. On-hardware validation is now in scope, not deferred. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Correct Task 4 to the real vmnet-helper model (per-VM privileged sidecar requiring --fd/--socket, resolved off-PATH at brew libexec; EnsureNetwork is bookkeeping, not a network-create shell-out). Add Phase B (Tasks 10-13) executed on the Apple Silicon dev box: sudoers privilege + preflight, live sidecar/detached-vfkit/Rosetta wiring, real aarch64 SNO boot, and the two-cluster DR check. Validation is no longer deferred to separate hardware. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Adds the macOS/Apple-Silicon provider backend behind the existing interfaces, selected by runtime.GOOS, with no cgo: - config.PayloadArch + openshift arch helpers; arch-aware mirror/client/stream URLs and InstallerSpec.Arch (threaded from runtime.GOARCH). - providers/host split by build tag (host_linux.go / host_darwin.go); Apple Silicon detected via sysctl hw.optional.arm64. - providers/vfkit: VMManager process-supervisor over vfkit (arg construction unit-tested; ISO/storage-pool methods are no-ops). - providers/vmnethelper: NetworkProvisioner + per-VM sidecar argv/binary resolver (vmnet-helper is an off-PATH privileged sidecar; EnsureNetwork is bookkeeping, not a shell-out). - rename stages/createlibvirtnetwork -> createnetwork (backend-neutral). - app/status: VM state via VMManager.IsRunning (works on both backends). - stages/publishpxeassets: macOS boot-media stage + per-OS selection in buildStages; pure KernelCmdline builder. - openshift.RosettaButaneFragment: guest binfmt/virtiofs config. - app/deps: NewDarwinDeps + dispatch, sharing baseDeps with the Linux wiring. Compiles and passes make check natively on macOS arm64 and cross-builds for Linux. On-hardware boot/Rosetta/DR validation is Phase B. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Add NetworkPreflight (optional interfaces.NetworkPreflighter): resolve the vmnet-helper binary and verify it runs under passwordless sudo, failing early with PrivilegeHint (the exact sudoers install command, NOPASSWD path pointed at the resolved binary). createnetwork.Preflight calls it via type assertion; libvirt does not implement it. This also gives the vmnethelper cmd field its first use. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Empirical results on real Apple Silicon hardware: vfkit EFI-boots the RHCOS aarch64 live ISO with an observable serial console; vfkit --ignition is ignored by RHCOS metal; coreos-installer has no macOS build. Therefore ignition must be delivered over the network (ignition.config.url), which needs --bootloader linux, while the post-install reboot needs --bootloader efi. Locks the two-phase supervisor design (linux install phase -> efi run phase) and revises the Task 11 scope accordingly. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
…k 11a)
Add Installer.CoreOSLivePXEURLs (parse metal.formats.pxe.{kernel,initramfs,
rootfs} from print-stream-json) + interfaces.CoreOSLivePXE + fake. On darwin,
download-rhcos fetches the 3 PXE assets (gunzipping the arm64 kernel, which
vfkit's linux bootloader requires uncompressed) instead of the ISO, and
download-binaries skips the Linux-only coreos-installer. The fake Downloader
now writes a placeholder file for every download so stages can read assets
back. StageContext gains RHCOS{Kernel,Initramfs,Rootfs}Path helpers.
Assisted-by: Claude Code/claude-opus-4-8
Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
… 11b) Add interfaces.SidecarLauncher + vmnethelper.StartSidecar (detached `sudo vmnet-helper --socket … --operation-mode shared`, waits for the socket to bind). Rework the vfkit VMManager into a two-phase supervisor: Create boots the install phase (--bootloader linux + live kernel/initramfs + our cmdline, NIC on the sidecar socket, --pidfile); when the install VM stops, Start transitions to the run phase (--bootloader efi booting the installed disk). Adds VMSpec.KernelPath/InitrdPath. NewVMManager now takes (stateDir, SidecarLauncher); deps wires the vmnet-helper provider as both Net and sidecar. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
…all (Task 11c) publish-pxe-assets.Apply now copies the RHCOS rootfs and SNO ignition into <fileserver-root>/<cluster>/ and records the install cmdline (console=hvc0 + rootfs/ignition urls + ignition.platform.id=metal) on the cluster. create-master-vms, on darwin, builds a vfkit VMSpec (kernel/initrd from the RHCOS cache + that cmdline; networking via the sidecar) and preflights vfkit instead of virt-install. Adds ClusterConfig.InstallKernelCmdline. Fakes: Downloader writes placeholder files, Installer writes the SNO ignition, and the fake FileServer roots at a writable temp dir so the stages run under tests. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Three fixes found booting a real aarch64 SNO on Apple Silicon: - download-rhcos: the RHCOS aarch64 live-kernel is an EFI zboot image (PE32+ wrapper around a gzip'd arm64 Image), which vfkit's linux bootloader rejects as "compressed". Parse the zboot header (MZ+zimg, payload offset/size, compression) and decompress to the raw Image. (Bare-gzip and uncompressed inputs still handled.) - vfkit: capture the supervised process's stdout/stderr to vfkit-launch.log so launch failures (e.g. the kernel rejection above) are diagnosable instead of silent. - publish-pxe-assets: macOS has no coreos-installer to embed a static-network keyfile into the ISO, so the node DHCP'd an arbitrary vmnet address instead of its allocated IP. Inject a NetworkManager keyfile into the served ignition to pin the master to its allocated IP (verified: node comes up on the expected .5, API serves there, bootstrap-in-place completes). Validated end to end: vfkit boots the live kernel -> vmnet-helper sidecar networking -> bootstrap-in-place installs to disk -> production control plane comes up at the pinned IP. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Real boot validated the full install path but the cluster reinstall-loops: vfkit reloads the live kernel on every guest reboot (no stop-on-reboot or boot-disk flag exists in vfkit/vz), so bootstrap-in-place never boots the installed disk. Documents the two viable fixes: (A) console-driven install->EFI transition, (B) EFI+ISO metal-style boot with a Go reimplementation of coreos-installer's ISO ignition-embed. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
vfkit reloads the live kernel on every guest reboot (Virtualization.framework has no stop-on-reboot), so bootstrap-in-place's post-install reboot looped the installer forever, and neither the IsRunning watchdog (vfkit never stops) nor serial reboot-line matching caught it (VZ hard-resets the guest without flushing kernel/systemd reboot lines). The install phase now spawns a watcher that polls the guest console and detects a reboot via the per-boot "First Boot Complete" target recurring (>= 2), then force-stops the install VM and relaunches in the EFI run phase to boot the now-installed disk. Also fixes per-VM sidecar lifecycle (stop the prior vmnet-helper before each relaunch) so the transition and watchdog restarts don't leak/conflict helpers. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
The install->EFI transition failed with VZ "storage device attachment is invalid" because forceStop SIGKILLed the install vfkit without waiting for it to exit, so the EFI relaunch raced the still-held disk-image lock. Track the vfkit process handle, reap it via a goroutine, and have forceStop wait for the process to actually terminate (lock released) before relaunching. Stop() uses the same handle. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
…h vfkit The --enable-tso / --enable-checksum-offload sidecar flags engage but break all IP payload traffic under vfkit: they require a virtio-net header on each socket frame to carry GSO/checksum metadata (QEMU's framing), but vfkit's Apple VZ file-handle attachment exchanges raw Ethernet frames with no such header. ARP survives; ICMP/TCP get 100% loss. Verified on hardware 2026-06-15 and reverted. Record the dead-end as an in-source guardrail so the flags are not re-added. Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
The per-VM vmnet-helper stop handle was an in-memory closure on VMManager (sidecarStops map), so only the process that started the sidecar could reap it. vfkit itself is reaped cross-process via its pidfile, but every CLI invocation after create exits (stop/delete) had an empty map and left the detached sudo vmnet-helper running — holding its vmnet bridge port and a privileged process indefinitely. Add a cross-process reaper that matches the VM-unique socket path: - interfaces.SidecarLauncher gains StopSidecar(ctx, name, socketPath); the vfkit supervisor calls it from stopSidecar as the fallback when it has no in-memory handle. - vmnethelper reaps by scanning ps for the vmnet-helper (sudo parent + dropped child) bound to this VM's socket and sudo-killing them; matching requires both the vmnet-helper token and the socket path, so vfkit and other clusters' sidecars are never touched. - StartSidecar reaps strays before binding, so a crash-leaked helper is cleared rather than stacked on. Verified on hardware: stop now reaps vfkit AND the sidecar; an injected stray is cleared at start leaving exactly one sidecar; cluster reconverges healthy. Note: this is a resource-leak/hygiene fix. Instrumented testing showed extra shared-mode helpers are benign and NOT the cause of the host->guest 'no route to host' (that was the pre-option-A reboot loop's VM-off windows). Assisted-by: Claude Code/claude-opus-4-8 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Silence the five errcheck findings the macOS backend introduced, using the repo's existing idiom for best-effort cleanup paths. Assisted-by: Claude Code/claude-fable-5 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
The vfkit VMs have always attached the rosetta virtiofs device, but the guest-side half was an unwired placeholder (RosettaButaneFragment had no callers), so binfmt_misc never knew about the translator and amd64 images could not run. Replace the fragment with RenderRosettaMachineConfig — a proper 99-master-rosetta MachineConfig carrying the /run/rosetta virtiofs mount unit and the x86-64 binfmt_misc registration (flags OCF so the interpreter resolves inside container roots). generate-ignition drops it into the install dir's openshift/ on darwin before create-single-node-ignition-config, the same mechanism the baked-image store uses, so the installed node translates amd64 from first boot. On-hardware validation surfaced an SELinux gap: the share is unlabeled_t, which confined containers cannot execute — amd64 entrypoints SIGSEGV'd when the kernel mapped the pre-opened translator. The mount unit therefore mounts the share with context=container_file_t, which must happen on the boot's first mount (the kernel refuses a context that differs from the live superblock). Assisted-by: Claude Code/claude-fable-5 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
MasterDiskGB was config-only; on disk-tight hosts (e.g. a Mac mini running two DR clusters) the 120 GB default fails the create-master-vms disk preflight with no recourse. Default unchanged at 120. Assisted-by: Claude Code/claude-fable-5 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Assisted-by: Claude Code/claude-fable-5 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
Full convergence, Rosetta (with the SELinux context= finding), and stop/start are validated on hardware. The two-cluster DR check is deferred to a >=48 GB host: 24 GB cannot keep two active 16 GB SNO VMs alive, and the induced thrash wedged the second cluster's MCD watches. Details appended to the Phase B spike spec; ROADMAP tracks the deferral, a memory preflight idea, and the rosetta-from-first-boot confirmation. Assisted-by: Claude Code/claude-fable-5 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
raghavendra-talur
added a commit
that referenced
this pull request
Aug 25, 2026
Tick completed items (Rosetta-from-first-boot, bake follow-ups merged in paper #18), drop stale branch references, add the two open ODF items: full-feature validation on a big host and the --odf-profile knob with the measured request math. Assisted-by: Claude Code/claude-fable-5 Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
macOS (Apple Silicon) backend for easyshift: SNO clusters as vfkit VMs with per-VM vmnet-helper network sidecars, validated end to end on real hardware (Mac mini M4 Pro, OCP 4.22.9).
--bootloader linux) install phase serving ignition over HTTP (publish-pxe-assets), console-driven transition to an EFI run phase that boots the installed disk — Virtualization.framework restarts a rebooting guest in place, so the supervisor watches the serial console for the post-install reboot. Validated: fresh cluster converges in ~29 min; MCO-driven reboots come back on the installed disk.vmnet-helpersidecar on the shared192.168.126.0/24subnet, privilege preflight with an actionable sudoers hint, static IP pinned via a NetworkManager keyfile injected into the served ignition, cross-process sidecar reaping.99-master-rosettaMachineConfig mounts the vfkit rosetta virtiofs share and registers the x86-64 binfmt handler. On-hardware finding: the share isunlabeled_t, which confined containers can't execute (amd64 entrypoints SIGSEGV) — the mount setscontext=container_file_t. Verified: confinedpodman run --arch amd64and a kubelet-scheduled amd64 pod both execute (x86_64).statusviaVMManager.IsRunning,--master-diskflag, errcheck cleanups.Design + findings:
docs/superpowers/specs/2026-06-13-macos-apple-silicon-support-design.md,.../2026-06-13-phaseB-vfkit-boot-spike.md. Pending items tracked inROADMAP.md(notably: the two-cluster DR gate is deferred — a 24 GB host cannot run two active 16 GB SNO VMs).Test plan
make check(vet, gofmt, golangci-lint, unit tests) — green.--simulate.easyshift createto a converged cluster, Rosetta verification through CRI-O,stop/start, cluster delete.🤖 Generated with Claude Code