[chruffins]

Summary

PR 2 of 6. Pull macOS machine images from OCI registries. Base: main (#498 has merged).

• Pulls darwin/arm64 machine images described by io.hypeman.machine-image.* config labels (format in docs/macos-experimental.md). Labels are validated before any layer is unpacked.
• Machine images are pulled only on macOS hosts. Other hosts refuse a darwin image before downloading its layers: a digest pull is refused by the API at once, and a tag pull fails in the worker right after it reads the image config.
• Sparse unpack: machine layers go through a dedicated extractor instead of umoci. It writes zero chunks as holes, accepts only regular files and directories, rejects path escapes and whiteouts, and checks each layer against its config diff ID. In native QA a 64 GB disk image allocated 30 GB, against the full 64 GB before.
• Files are staged in a private 0700 directory and made 0600 before they appear at their final path; finalization rolls back only files this attempt installed. The recorded size is the boot disk's logical size, as for a local import.
• Tagging works within a repository; cross-repository promotion is not supported yet.

Flow

Before this PR, a darwin OCI image went down the Linux path (full download, umoci unpack, rootfs export) and was refused only at finalize. NEW marks what this PR adds; unmarked steps are the existing flow.

CreateImage(req):
  platform = parse(req.platform or host)
  NEW  explicit darwin request on a non-Mac host → 400
  ref = resolve(req.name, platform)
  NEW  digest ref whose manifest is darwin, on a non-Mac host → 400
  NEW  explicit platform ≠ a cached macOS record's platform → 400
  reuse an existing record, or write a pending one and queue buildImage

buildImage(ref):
  if not cached: fetch manifest
    NEW  non-Mac host: read config; darwin → fail before any layer blob
    download layers
  NEW  darwin config → validate host, darwin/arm64, machine labels, and that the
       three file paths are local and distinct; unpack with the sparse extractor
  else → umoci unpack, then ExportRootfs                         (unchanged)
  NEW  darwin → copy disk and aux to private 0600 staging files beside their final
       paths; size = disk's logical size
  finalizeImage:
    NEW  darwin without a staged machine bundle → fail
    NEW  install aux.img and record the macOS platform
    install disk, manifest model, ready metadata                 (unchanged)
    on failure, remove only files this attempt installed

sparse extractor, per layer:
  regular files: size first, write only non-zero 1 MiB chunks (zeros stay holes)
  directories: 0700; anything else, non-local paths, whiteouts → reject
  stream hash must equal the layer's config diff ID

Linux code touched

• lib/images/oci.go: one dispatch to the machine extractor; Linux images still use umoci. Before writing layers to the cache, non-macOS hosts check the image config's OS; the registry client caches that config for the layout write, so Linux pulls make no extra request.
• lib/images/manager.go → lib/images/finalize.go: finalization moved into its own file (mostly moved code), now tracking installed files so auxiliary storage rolls back too. buildImage keeps its rootfs export; macOS staging is one stageMacOSOutputs hook in macos_machine.go.
• lib/images/platform.go, tag.go: darwin handling.

Testing

• Unit tests cover label validation, extractor safety (symlinks, hardlinks, escapes, whiteouts, digest mismatch), sparseness against a same-filesystem control, rollback of the disk and auxiliary storage, a registry round trip through the image manager, and a Linux host refusing a darwin image without fetching its layer.
• Native QA ran on Apple silicon (macOS 27.0.1 host and guests) across six runs plus a VZ restore spike. The simplification pass after run 6 has not been re-run natively yet. This PR's paths covered there: pulling a 64 GiB machine image, a cold boot from it, and sparse allocation on APFS.

Review focus

• lib/images/macos_unpack.go (the extractor).
• The finalization move in finalize.go, which Linux pulls also use.

Stack

Each PR is one commit on the one below it. macOS logic lives in its own files (macos*.go, *_darwin.go, exec_lifecycle.go, port_forward.go); shared Linux files get small hooks such as if stored.MacOS != nil.

1. macOS guests 1/6: VZ runtime, machine images and instance lifecycle #498 VZ runtime, machine images and instance lifecycle
2. macOS guests 2/6: OCI machine-image integration #499 OCI machine images
3. macOS guests 3/6: Darwin guest service, readiness and exec #500 Darwin guest agent, readiness and exec
4. macOS guests 4/6: snapshots, schedules and stopped forks #506 snapshots, standby and stopped clones
5. macOS guests 5/6: guest-port forwarding #514 guest-port forwarding
6. macOS guests 6/6: experimental machine-image builds #503 machine-image builds (not needed for v0; can land later)

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit fdc003f. Configure here.}

[chruffins]

QA update on reviewed head 66b5a59: real stopped installed macOS bundle OCI pull/materialization passed (test exit 0 plus the required roundtrip.json).

• 64-GiB raw disk; compressed OCI layer ~26.22 GiB; total test ~373 seconds.
• Normal image manager pulled through a loopback registry, reached ready on darwin/arm64, and verified disk/aux SHA256, complete platform metadata, manifest digest, canonical modes, cache reuse, same-repository aliasing, cross-repository rejection and wrong-platform rejection.
• Follow-up cached artifact list/get through authenticated generated HTTP image handlers also passes with race detector; unauthenticated requests rejected.
• Allowlisted environment with empty Docker config, nice=10, GOMAXPROCS=1, GOMEMLIMIT=256MiB. Sampled job-process-group RSS peak ~62 MiB; kernel file cache is not included. Free disk remained ~523 GiB, above the 60-GiB floor.

Important scope: the registry blobs were seeded via the disk-streaming handler, so this validates large real HTTP pull/materialization, not large-layer PATCH upload. It also does not prove cold boot or live GuestService provisioning/lifecycle. No existing instance was started, stopped or reprovisioned by these tests. Keep draft pending the remaining gates.

[chruffins]

Follow-up live QA: the artifact from the earlier real HTTP materialization run now passed normal authenticated API create/cold boot on Apple silicon with the source registry offline, actual macOS27.0.1/26A434 SSH readiness, template CPU/RAM inheritance, duplicate-identity409, live API restart recovery without guest reboot, and HTTP stop/start with persistent guest marker. Full immutable materialized base disk+aux SHA256 stayed identical after testing. Original instance/API/storage untouched; isolated QA VM/API ended stopped and QA slot released. Stop used system-agent-disabled fallback, not verified root-agent graceful shutdown. Combined reviewed PR1–3 + PR4 patches tested; reviewer branches not rewritten. See runtime issue/result detail: #498 (comment). Separately, actual26.22GiB streaming monolithic POST upload to empty production-registry-handler destination plus independent blob/manifest validation passed; default PATCH is still buffered/unproven, and that destination was not freshly pulled/booted: #503 (comment).

[chruffins]

Simplification update at 4888603: OCI uses the shared bounded bundle decoder; permanent TestFinalizationRollbackPreservesUninstalledManifest fixes the reproduced regression where rollback deleted an existing manifest that had never been installed. Focused image regressions passed 3 race runs; no new real upload/pull/boot claim.

All PRs remain draft. This is source-level/synthetic validation, not a new live boot or production build proof. Default Codex independent review was attempted but is still blocked by authentication (HTTP401). Published atomically after checking reviewer heads with explicit per-ref force-with-lease; prior heads preserved locally.

[hiroTamada]

reviewed — two issues worth addressing:

• P2 — lib/images/macos_machine.go:53: validate explicit platforms before reusing in-flight pulls. while a darwin/arm64 tag pull is running, requesting the same tag with linux/arm64 succeeds and returns the in-flight Darwin record. the new guard skips every non-ready record. can we reject incompatible explicit platforms on that reuse path too? reproduced with a local registry and a paused layer download.
• P2 — lib/images/macos_unpack.go:82-83,112-113: bound logical expansion of sparse tar entries. GNU PAX sparse entries surface as regular files, but tar.Reader synthesizes hole bytes without consuming the underlying byte budget. the size check limits each file, not aggregate expansion: a 40 KiB archive containing sixteen sparse 512 KiB files extracts 8 MiB with a 1 MiB budget. the raw tar-stream limit works, but extraction work can exceed it substantially. consider rejecting sparse tar metadata or separately capping total logical expansion, with a regression test.

all targeted validation, extraction, staging, rollback, tagging, and registry round-trip tests passed. full-suite and native macOS/APFS/VZ validation were not run.

[sjmiller609]

raw machine disks shouldn’t be named rootfs.ext4
lib/paths/paths.go:151–156

Use a machine-specific raw-disk path. Keep .ext4 for actual Linux filesystems on Darwin hosts.

complete boot disks shouldn’t be named overlay.raw
lib/instances/macos.go:155,169

Use something like boot.raw for macOS. This naming predates this PR, but remains on our requested cleanup list.

[chruffins]

@hiroTamada thanks, both fixed in 1fa0cb0:

• in-flight reuse: an explicit platform is now checked against an in-flight pull's requested platform (and a ready record's actual one) whenever either side is darwin. TestMacOSMachineInFlightPullRefusesOtherPlatform.
• sparse tar: entries with GNU.sparse.* PAX records are rejected; the old S type already was. TestUnpackMachineLayerRejectsGNUSparseEntries uses your 16 × 512 KiB / 1 MiB case.

[chruffins]

@sjmiller609 thanks, both renamed:

• pulled machine disks are now stored as rootfs.raw, matching local imports (05f0810); disk-utilization accounting counts it too.
• macOS instance boot disks are now boot.raw instead of overlay.raw, in macOS guests 3/6: Darwin guest service, readiness and exec #500, since that path lives in the instance code.

No fallback for the old names; images and instances created before need recreating (noted in docs/macos-experimental.md).