bake-images: on-hardware fixes — manifest ordering, CRI-O drop-in, preflight

ROADMAP.md

@@ -28,24 +28,37 @@ mounted with `context=container_file_t` — see the Task 13 notes in the spec),
       on darwin). Needs a vmnet bridged-mode story before LAN-reachable
       clusters work on Mac.
 
-## Image baking (`--bake-images`) — branch `rtalur-bake-images-macos`
-
-Code-complete on both backends (see `docs/dev/image-baking.md`). Linux:
-skopeo + virt-make-fs → qcow2 into the libvirt pool. macOS: skopeo +
-`mke2fs -d` → raw image, APFS-cloned per cluster, merged into the HTTP-served
-ignition. `make check` green; darwin pipeline validated under `--simulate` and
-the mke2fs pack validated against real e2fsprogs.
-
-Branch note: `rtalur-bake-images` is the same feature rebased onto `main`
-(without the macOS backend); `rtalur-bake-images-macos` stacks it on
-`rtalur-macos-backend` and adds the macOS integration. Whichever merge order
-is chosen, keep only one of the two bake branches.
-
-- [ ] **End-to-end validation on a real cluster** (either OS): a
-      `--bake-images` install confirming CRI-O serves release images from the
-      store, not quay.io, in both bootstrap and post-pivot phases. Needs
-      ~40+ GB free disk for the multi-arch store — did not fit the dev Mac
-      alongside the existing cluster.
+## Image baking (`--bake-images`) — merged (#17) + follow-up fixes
+
+**Validated end-to-end on macOS hardware (2026-08-25)**: a clean
+`create --bake-images` converged in ~25.5 min (storeless baseline: 29 min)
+with the store mounted read-only from first boot, CRI-O serving ~173 store
+images via the crio drop-in, only 8 network image pulls and 1.0 GiB total RX
+in 20 min (the broken run pulled 7.1 GiB in five). Rosetta amd64 execution
+also verified from first boot. Three hardware-found fixes live on branch
+`rtalur-bake-followups`: APFS-clone-aware disk preflight, `create manifests`
+before extra-manifest drops (they were silently ignored), and the CRI-O
+drop-in replacing the unsupported `storage.conf.d` mechanism.
+
+- [ ] **Productize the macOS store builder.** skopeo cannot author an overlay
+      container store on macOS (Linux-only graph driver), so the baker's
+      skopeo step fails on a Mac. The validated workaround built the store
+      inside the podman machine and copied out `store.img` (bake stage's
+      `Ready()` then skips the build). Implement that as the darwin baker
+      backend (podman machine or any Linux builder), including multi-arch
+      (`--all`) copies — the spike store was aarch64-only.
+- [ ] **Linux-side validation**: the qcow2/virt-make-fs/libvirt-pool variant
+      of the attach path has not run against a real Linux host.
+- [ ] **Offline installs** (the end-goal baking enables). Residual online
+      dependencies measured on hardware (2026-08-25): the three OLM catalog
+      indexes (fix: ship an OperatorHub disableAllDefaultSources manifest when
+      baking), the two insights-runtime images whose imagePullPolicy=Always
+      bypasses the store (fix: disable Insights, disconnected-style), and —
+      the real blocker — magic DNS: sslip.io names resolve via public
+      nameservers, so api/api-int/*.apps are unresolvable offline on both host
+      and node. Needs a local answer for the cluster domain (host dnsmasq /
+      hosts injection + node-side resolution) or a non-magic local domain
+      mode. Binaries + RHCOS are already cached per version.
 
 ## Later phases (per project vision)
 

docs/dev/image-baking.md

@@ -64,8 +64,13 @@ RHCOS live-ISO arch is still selected separately (see
   dir (vfkit has no read-only virtio-blk, so the per-cluster copy is the
   isolation; `Delete` removes the clone).
 - The disk is mounted by label (`/dev/disk/by-label/baked-images` →
-  `/var/lib/baked-images`) and registered with CRI-O via
-  `additionalimagestores` in a `storage.conf` drop-in.
+  `/var/lib/baked-images`) and registered with CRI-O via a **CRI-O drop-in**
+  (`/etc/crio/crio.conf.d/10-baked-images.conf`,
+  `storage_option = ["overlay.imagestore=…"]`). NOT via
+  `/etc/containers/storage.conf.d/`: RHCOS 9.8's containers-common (5.8)
+  silently ignores that dir — validated on hardware, where the store sat
+  mounted-but-unread and every image still came from quay.io until the CRI-O
+  drop-in surfaced all of them.
 - That wiring is applied in **both** install phases:
   - **post-pivot:** a master `MachineConfig` dropped into the install dir's
     `openshift/` (`Installer.WriteImageStoreManifest`) so it is rendered into
@@ -77,10 +82,27 @@ RHCOS live-ISO arch is still selected separately (see
     (`embed-ignition-iso`), on macOS into the HTTP-served `config.ign`
     (`publish-pxe-assets`).
 
-Renderers live in `providers/openshift/baker.go` (`RenderStorageConfDropin`,
+Renderers live in `providers/openshift/baker.go` (`RenderCRIODropin`,
 `RenderMountUnit`, `RenderMachineConfig`, `MergeBakedStoreIntoIgnition`) and are
 unit-tested in `baker_test.go`.
 
+**Extra-manifest ordering (hardware-validated):** `openshift-install create
+single-node-ignition-config` only renders manifests dropped into `openshift/`
+if `create manifests` ran first — without that state they are silently
+ignored. `generate-ignition` therefore calls `Installer.CreateManifests`
+before the manifest writes whenever extras are needed (darwin or
+`--bake-images`).
+
+**macOS store-build limitation:** skopeo cannot author a
+`containers-storage:` overlay store on macOS (the overlay graph driver is
+Linux-only), so the coded skopeo path in the baker cannot run on a Mac. The
+validated workaround builds the store inside a Linux VM (e.g. the podman
+machine: skopeo into an overlay store on the machine's own fs, `mke2fs -d`
+pack there, copy out the single `store.img` to
+`~/.config/easyshift/imagestore/<version>/`); the bake stage's `Ready()`
+probe then skips the build. Productizing that builder is tracked in
+ROADMAP.md.
+
 ## Verification boundary
 
 The pipeline, the rendered artifacts, and the `--simulate` trace are covered by

interfaces/interfaces.go

@@ -187,6 +187,12 @@ type InstallerSpec struct {
 // Installer abstracts invocations of openshift-install and coreos-installer.
 type Installer interface {
 	WriteInstallConfig(ctx context.Context, spec InstallerSpec) error
+	// CreateManifests runs `openshift-install create manifests`, materializing
+	// the manifests/ + openshift/ dirs and the installer state. Required
+	// before dropping extra manifests (Rosetta, baked image store): without
+	// the state from this step, `create single-node-ignition-config` silently
+	// ignores files pre-dropped into openshift/ (validated on hardware).
+	CreateManifests(ctx context.Context, spec InstallerSpec) error
 	CreateIgnitionConfigs(ctx context.Context, spec InstallerSpec) error
 	// WriteImageStoreManifest drops a MachineConfig into the install dir's
 	// openshift/ so the next ignition render wires CRI-O to the baked image

providers/fakes/fakes.go

@@ -321,6 +321,7 @@ func (n *NetworkProvisioner) ResetNetwork(_ context.Context, network string) err
 type Installer struct {
 	mu                       sync.Mutex
 	WroteInstallConfig       bool
+	CreatedManifests         bool
 	CreatedIgnitions         bool
 	CreatedSingleNodeIgn     bool
 	WroteRosettaManifest     bool
@@ -341,23 +342,39 @@ type Installer struct {
 	// LiveISOURL overrides the URL returned by CoreOSLiveISOURL.
 	LiveISOURL string
 	Err        error
+	// Sequence records method names in call order, for ordering assertions
+	// (e.g. CreateManifests must precede the manifest writes).
+	Sequence []string
 }
 
 func (i *Installer) record(spec interfaces.InstallerSpec) {
 	i.LastSpec = spec
 }
 
+func (i *Installer) step(name string) { i.Sequence = append(i.Sequence, name) }
+
 func (i *Installer) WriteInstallConfig(_ context.Context, spec interfaces.InstallerSpec) error {
 	i.mu.Lock()
 	defer i.mu.Unlock()
+	i.step("WriteInstallConfig")
 	i.WroteInstallConfig = true
 	i.record(spec)
 	return i.Err
 }
 
+func (i *Installer) CreateManifests(_ context.Context, spec interfaces.InstallerSpec) error {
+	i.mu.Lock()
+	defer i.mu.Unlock()
+	i.step("CreateManifests")
+	i.CreatedManifests = true
+	i.record(spec)
+	return i.Err
+}
+
 func (i *Installer) CreateIgnitionConfigs(_ context.Context, spec interfaces.InstallerSpec) error {
 	i.mu.Lock()
 	defer i.mu.Unlock()
+	i.step("CreateIgnitionConfigs")
 	i.CreatedIgnitions = true
 	i.record(spec)
 	return i.Err
@@ -366,6 +383,7 @@ func (i *Installer) CreateIgnitionConfigs(_ context.Context, spec interfaces.Ins
 func (i *Installer) WriteRosettaManifest(_ context.Context, spec interfaces.InstallerSpec) error {
 	i.mu.Lock()
 	defer i.mu.Unlock()
+	i.step("WriteRosettaManifest")
 	i.WroteRosettaManifest = true
 	i.record(spec)
 	return i.Err
@@ -374,6 +392,7 @@ func (i *Installer) WriteRosettaManifest(_ context.Context, spec interfaces.Inst
 func (i *Installer) WriteImageStoreManifest(_ context.Context, spec interfaces.InstallerSpec) error {
 	i.mu.Lock()
 	defer i.mu.Unlock()
+	i.step("WriteImageStoreManifest")
 	i.WroteImageStoreManifest = true
 	i.record(spec)
 	return i.Err
@@ -382,6 +401,7 @@ func (i *Installer) WriteImageStoreManifest(_ context.Context, spec interfaces.I
 func (i *Installer) CreateSingleNodeIgnition(_ context.Context, spec interfaces.InstallerSpec) error {
 	i.mu.Lock()
 	defer i.mu.Unlock()
+	i.step("CreateSingleNodeIgnition")
 	i.CreatedSingleNodeIgn = true
 	i.record(spec)
 	if i.Err != nil {

providers/openshift/baker.go

@@ -235,17 +235,21 @@ func parseReleasePullspecs(data []byte) ([]string, error) {
 
 // --- CRI-O wiring renderers (pure) --------------------------------------
 
-// storageConfDropinPath is where the additional-image-store drop-in lands. The
-// containers/storage library merges *.conf from this dir over storage.conf.
-const storageConfDropinPath = "/etc/containers/storage.conf.d/10-baked-images.conf"
-
-// RenderStorageConfDropin returns the storage.conf drop-in that registers the
-// baked store as a read-only CRI-O additional image store.
-func RenderStorageConfDropin() string {
-	return fmt.Sprintf(`[storage]
-[storage.options]
-additionalimagestores = [
-  "%s",
+// crioDropinPath is where the additional-image-store drop-in lands. It rides
+// CRI-O's own drop-in dir, NOT /etc/containers/storage.conf.d/: RHCOS 9.8's
+// containers-common (5.8) silently ignores storage.conf.d, which left the
+// store mounted but unread — every image still came from quay.io (found on
+// hardware; a crio storage_option drop-in surfaced all store images).
+const crioDropinPath = "/etc/crio/crio.conf.d/10-baked-images.conf"
+
+// RenderCRIODropin returns the CRI-O drop-in that registers the baked store
+// as a read-only additional image store via the overlay driver's imagestore
+// option. RHCOS ships no active storage_option, so this list replaces
+// nothing.
+func RenderCRIODropin() string {
+	return fmt.Sprintf(`[crio]
+storage_option = [
+  "overlay.imagestore=%s",
 ]
 `, config.BakedImagesMountPath)
 }
@@ -283,7 +287,7 @@ const MachineConfigName = "99-master-baked-image-store"
 // operators.
 func RenderMachineConfig() string {
 	unitName, unitContents := RenderMountUnit()
-	storageB64 := base64.StdEncoding.EncodeToString([]byte(RenderStorageConfDropin()))
+	storageB64 := base64.StdEncoding.EncodeToString([]byte(RenderCRIODropin()))
 	return fmt.Sprintf(`apiVersion: machineconfiguration.openshift.io/v1
 kind: MachineConfig
 metadata:
@@ -307,10 +311,10 @@ spec:
         enabled: true
         contents: |
 %s
-`, MachineConfigName, storageConfDropinPath, storageB64, unitName, indent(unitContents, "          "))
+`, MachineConfigName, crioDropinPath, storageB64, unitName, indent(unitContents, "          "))
 }
 
-// MergeBakedStoreIntoIgnition adds the storage.conf drop-in file and the mount
+// MergeBakedStoreIntoIgnition adds the CRI-O drop-in file and the mount
 // unit to a raw Ignition config (the bootstrap-in-place-for-live-iso.ign), so
 // the baked store is also used during the live-ISO bootstrap phase. It edits
 // the JSON structurally to preserve whatever the installer emitted.
@@ -326,11 +330,11 @@ func MergeBakedStoreIntoIgnition(ignitionJSON []byte) ([]byte, error) {
 	}
 	files, _ := storage["files"].([]any)
 	files = append(files, map[string]any{
-		"path":      storageConfDropinPath,
+		"path":      crioDropinPath,
 		"mode":      420,
 		"overwrite": true,
 		"contents": map[string]any{
-			"source": "data:text/plain;base64," + base64.StdEncoding.EncodeToString([]byte(RenderStorageConfDropin())),
+			"source": "data:text/plain;base64," + base64.StdEncoding.EncodeToString([]byte(RenderCRIODropin())),
 		},
 	})
 	storage["files"] = files

providers/openshift/baker_test.go

@@ -69,10 +69,14 @@ func TestRenderMountUnit(t *testing.T) {
 	}
 }
 
-func TestRenderStorageConfDropin(t *testing.T) {
-	got := RenderStorageConfDropin()
-	if !strings.Contains(got, "additionalimagestores") || !strings.Contains(got, "/var/lib/baked-images") {
-		t.Fatalf("storage.conf drop-in missing key/path:\n%s", got)
+func TestRenderCRIODropin(t *testing.T) {
+	got := RenderCRIODropin()
+	// The additional image store must ride CRI-O's storage_option: RHCOS 9.8's
+	// containers-common (5.8) does not read /etc/containers/storage.conf.d/
+	// drop-ins, so a storage.conf drop-in is silently ignored (found on
+	// hardware: the node pulled everything from quay.io).
+	if !strings.Contains(got, "[crio]") || !strings.Contains(got, "overlay.imagestore=/var/lib/baked-images") {
+		t.Fatalf("crio drop-in missing storage_option/path:\n%s", got)
 	}
 }
 
@@ -82,7 +86,7 @@ func TestRenderMachineConfig(t *testing.T) {
 		"kind: MachineConfig",
 		"machineconfiguration.openshift.io/role: master",
 		MachineConfigName,
-		"/etc/containers/storage.conf.d/10-baked-images.conf",
+		"/etc/crio/crio.conf.d/10-baked-images.conf",
 		"data:text/plain;base64,",
 		`var-lib-baked\x2dimages.mount`,
 	} {
@@ -130,7 +134,7 @@ func TestMergeBakedStoreIntoIgnition(t *testing.T) {
 	}
 	var foundDropin, foundMount bool
 	for _, f := range cfg.Storage.Files {
-		if f.Path == storageConfDropinPath {
+		if f.Path == crioDropinPath {
 			foundDropin = true
 			if !strings.HasPrefix(f.Contents.Source, "data:text/plain;base64,") {
 				t.Errorf("dropin source not a data URL: %q", f.Contents.Source)
@@ -158,7 +162,7 @@ func TestMergeBakedStoreIntoIgnition_EmptyConfig(t *testing.T) {
 	}
 	// The unit name's backslash is JSON-escaped in raw output; assert on the
 	// dropin path and the mount unit body, which survive verbatim.
-	if !strings.Contains(string(out), storageConfDropinPath) ||
+	if !strings.Contains(string(out), crioDropinPath) ||
 		!strings.Contains(string(out), "Where=/var/lib/baked-images") {
 		t.Fatalf("merge into empty config dropped entries:\n%s", out)
 	}

providers/openshift/installer.go

@@ -117,11 +117,20 @@ func (i *OpenShiftInstaller) WriteInstallConfig(_ context.Context, spec interfac
 	return nil
 }
 
-// CreateIgnitionConfigs runs `openshift-install create ignition-configs`.
-func (i *OpenShiftInstaller) CreateIgnitionConfigs(ctx context.Context, spec interfaces.InstallerSpec) error {
+// CreateManifests runs `openshift-install create manifests`. See the
+// interface doc: this must precede writing extra manifests into openshift/.
+func (i *OpenShiftInstaller) CreateManifests(ctx context.Context, spec interfaces.InstallerSpec) error {
 	if _, err := i.cmd.Run(ctx, spec.InstallerPath, "create", "manifests", "--dir", spec.ClusterDir); err != nil {
 		return fmt.Errorf("create manifests: %w", err)
 	}
+	return nil
+}
+
+// CreateIgnitionConfigs runs `openshift-install create ignition-configs`.
+func (i *OpenShiftInstaller) CreateIgnitionConfigs(ctx context.Context, spec interfaces.InstallerSpec) error {
+	if err := i.CreateManifests(ctx, spec); err != nil {
+		return err
+	}
 	if _, err := i.cmd.Run(ctx, spec.InstallerPath, "create", "ignition-configs", "--dir", spec.ClusterDir); err != nil {
 		return fmt.Errorf("create ignition-configs: %w", err)
 	}

stages/createmastervms/stage.go

@@ -56,8 +56,10 @@ func (s *Stage) Preflight(ctx context.Context, sc *interfaces.StageContext) erro
 		return fmt.Errorf("query disk space at %s: %w", sc.Config.ConfigDir, err)
 	}
 	need := uint64(sc.Cluster.MasterDiskGB) * 1024 * 1024 * 1024
-	// Baking attaches a per-cluster copy of the store qcow2; count it.
-	if sc.Cluster.BakeImages {
+	// Baking attaches a per-cluster copy of the store disk; count it — except
+	// on macOS, where ImportDisk APFS-clones the cached image and the copy
+	// costs no space until modified (it never is: the guest mounts it ro).
+	if sc.Cluster.BakeImages && runtime.GOOS != "darwin" {
 		if fi, err := os.Stat(config.ImageStoreDiskPath(sc.Config.ConfigDir, sc.Cluster.OCPVersion)); err == nil {
 			need += uint64(fi.Size())
 		}

stages/generateignition/stage.go

@@ -67,6 +67,16 @@ func (s *Stage) Apply(ctx context.Context, sc *interfaces.StageContext) error {
 	if err := s.installer.WriteInstallConfig(ctx, spec); err != nil {
 		return err
 	}
+	// Extra manifests (Rosetta, baked image store) are only rendered into the
+	// ignition if `create manifests` ran first — without its state,
+	// single-node-ignition-config silently ignores files dropped into
+	// openshift/ (validated on hardware). Skip when no extras are written to
+	// keep the plain path identical.
+	if runtime.GOOS == "darwin" || sc.Cluster.BakeImages {
+		if err := s.installer.CreateManifests(ctx, spec); err != nil {
+			return err
+		}
+	}
 	if runtime.GOOS == "darwin" {
 		// vfkit attaches the Rosetta virtiofs device; drop the MachineConfig
 		// that mounts it and registers the x86-64 binfmt handler so the

stages/generateignition/stage_test.go

@@ -22,7 +22,7 @@ func TestApplyWritesRosettaManifestOnDarwin(t *testing.T) {
 	}
 	cfgDir := t.TempDir()
 	cfg := &config.Config{ConfigDir: cfgDir}
-	c := &config.ClusterConfig{Name: "t", NetworkMode: config.NetworkModeNAT}
+	c := &config.ClusterConfig{Name: "t", NetworkMode: config.NetworkModeNAT, BakeImages: true}
 	sc := &interfaces.StageContext{Config: cfg, Cluster: c}
 	if err := os.MkdirAll(sc.ClusterDir(), 0o755); err != nil {
 		t.Fatal(err)
@@ -42,4 +42,23 @@ func TestApplyWritesRosettaManifestOnDarwin(t *testing.T) {
 	if !inst.WroteRosettaManifest {
 		t.Error("expected WriteRosettaManifest to be called on darwin")
 	}
+	// openshift-install only renders pre-dropped openshift/ manifests into the
+	// ignition when `create manifests` ran first (validated on hardware —
+	// without it the extra MachineConfigs are silently ignored). So the order
+	// must be: CreateManifests, then the manifest writes, then the ignition.
+	assertOrder(t, inst.Sequence, "CreateManifests", "WriteRosettaManifest", "CreateSingleNodeIgnition")
+	assertOrder(t, inst.Sequence, "CreateManifests", "WriteImageStoreManifest", "CreateSingleNodeIgnition")
+}
+
+func assertOrder(t *testing.T, seq []string, want ...string) {
+	t.Helper()
+	i := 0
+	for _, s := range seq {
+		if i < len(want) && s == want[i] {
+			i++
+		}
+	}
+	if i != len(want) {
+		t.Errorf("call order %v does not contain %v in order", seq, want)
+	}
 }