Repository navigation
Run the agent as a privileged container on Azure Container Linux - #370
Draft
Philip Lombardi (plombardi89) wants to merge 56 commits into
Draft
Philip Lombardi (plombardi89) wants to merge 56 commits into
Philip Lombardi (plombardi89) wants to merge 56 commits into
Conversation
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 19, 2026 19:24 — with 
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 19, 2026 20:16 — with
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
had a problem deploying
to
e2e-testing
October 19, 2026 20:47 — with
GitHub Actions
Failure
GitHub Actions
Failure
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 19, 2026 20:49 — with
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
force-pushed
the
phlombar/acl-container
branch
from
October 19, 2026 21:15
6d410b8 to
43a57f0
Compare
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 19, 2026 21:15 — with
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 19, 2026 21:35 — with
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 20, 2026 00:43 — with
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 20, 2026 01:57 — with
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 20, 2026 02:38 — with
GitHub Actions
Active
GitHub Actions
Active
Azure Container Linux ships an unlabeled /usr, so every host process runs as kernel_t, while a privileged containerd container runs as spc_t, which the policy doesn't let send D-Bus messages to kernel_t. machinectl, busctl, loginctl, hostnamectl, and resolvectl all fail from the container. SELINUX.md explains it for the ACL team, with the one rule that fixes it, allow spc_t kernel_t:dbus send_msg, verified against every host operation the agent and the unbounded library perform. selinux-repro.sh reproduces it on any ACL host; selinux-verify.sh builds candidate policies from a host's own and runs the reproduction against them.
hack/acl-container boots Azure Container Linux under QEMU/KVM from an Ignition config rendered from agent-container.bu, which runs the agent image with the host's containerd: privileged, in the host's network and PID namespaces, with the host's root at /host. In the container, machinectl, systemctl, systemd-run, and journalctl run the host's binaries through chroot /host, and flex-machine creates, starts, stops, and opens shells in the host's systemd-nspawn machines. The agent binary is in the image, but the entrypoint only checks its access to the host and idles. harness.py builds and pushes the image to a registry on the bridge, boots the VM, and runs the feasibility checks, including that machines keep running across a container restart and come back after a reboot. On current ACL images the container needs the SELinux workaround in the next commit; this is the design once ACL's policy allows it.
Until ACL's policy allows spc_t to send D-Bus messages to kernel_t (see SELINUX.md), the agent container can't use the host's machinectl. selinux-workaround.bu runs it from containerd's native snapshotter with its directory relabeled usr_t, which has no domain transition, so the agent stays kernel_t like every other host process. It is a one-shot relabel unit and a drop-in for the agent's unit, merged into the Ignition config by harness.py up --selinux-workaround. The image is squashed to one layer then, as the native snapshotter copies every layer in full. Without the flag, up now stops with a clear message when the container hits the denial, rather than timing out, and check expects the container in kernel_t with the workaround and spc_t without. Development only: revert this commit once ACL ships the fix.
TODO.md lists what is left to run the agent as a privileged container on Azure Container Linux: removing the SELinux workaround once ACL ships the fix, parity with main's agent, running the real agent in the container, and the container's launch and image. It is a living document; items are added as they come up and deleted once done.
In its container image, which sets AKS_FLEX_NODE_CONTAINERIZED, the agent's PATH has the image's package manager, so installing OS packages would install into the container, and the image and the container's own unit run the agent, so start has no binary or unit to install on the host. setupHost leaves out InstallPackages there, and start leaves out InstallService. Everything else start does is unchanged.
The agent writes host paths, such as /etc/systemd, /etc/sysctl.d, /var/lib/machines, and /run, by name. The container now mounts each at its own path, and /etc/os-release, so the agent, unchanged, writes the host's and detects the host's OS. It runs the host's systemd-nspawn, ip, nft, sysctl, modprobe, swapoff, and udevadm through host-exec, as it does machinectl and systemctl. The agent installs its host helpers under /usr/local, which host systemd runs, such as the nspawn lifecycle helper at every machine start. ACL's /usr is read-only, so an overlay mount gives /usr/local a writable layer in /var, and the container mounts it; the image's own programs move to /opt/aks-flex-node/bin. With these, main's aks-flex-node start runs from the container: it sets up the host, provisions and starts kube1, and installs the lifecycle helper, which host systemd runs.
At every startup the daemon converges the agent's blue/green binary layout under /usr/local and its systemd units, the agent's own and its recovery unit, so installs that predate them get them. In its container the agent has neither: the image and the container's own unit run it, and the container sees the host's /usr/local and /etc/systemd, so doing so would leave a second, dormant agent install on the host. The daemon skips it there, as start already does.
harness.py up --cluster creates a kind cluster on root Podman, attaches its node to the VM's bridge, deploys main's aks-flex-controller from hack/controller-deployment with a locally built image, registers the VM as a machine, and puts a main-format agent config with a bootstrap token in the Ignition config. The agent in the container then joins the VM to the cluster with no Azure resources involved. Given that config, the container's entrypoint does what main's bootstrap and the agent's unit do on a host: aks-flex-node start once, until it saves the daemon state, then aks-flex-node daemon. Without it, the entrypoint idles as before. check adds cluster checks when the VM joined one: the node is Ready, kube1 runs, a pod runs on the node and kubectl logs reaches it, and the node is Ready again after the reboot. down removes the cluster.
When the agent removes a machine, as in a repave, it unmounts the machine's bpffs, which the machine's unit mounted on the host under /run/bpffs. The container's mounts of host paths are slaves of the host's, so that unmount stayed in the container, the host kept the mount, and removing its directory failed: every repave left a mount behind. The image now links mount, umount, and mountpoint to host-exec, which runs them in the host's mount namespace, PID 1's, rather than under chroot /host as the other host tools are. As on a host, the agent's unmount then reaches the host. This works as spc_t too, with no SELinux denials, so it doesn't depend on the workaround. check tests it directly with a throwaway bpffs mount.
check now repaves the node, as main's e2e does: it changes the machine's settings version, its eTag, in the controller's store and deletes the Node. It then expects the daemon in the container to start the other machine with the new settings, a new Node to be Ready, the old machine to be removed with its bpffs mount, and a pod to run on the repaved node. The cluster checks follow the active machine in the daemon's state rather than assume kube1.
Reset removes /etc/aks-flex-node and /var/log/aks-flex-node. In the agent's container they're mounts of the host's directories at their own paths, so os.RemoveAll removes everything in them and then fails with EBUSY on the directory itself, which failed AgentReset. A busy directory that RemoveAll has emptied now counts as removed. On a host they're never mount points, so nothing changes there.
On a host, AgentReset and reset stop the agent by stopping, disabling, and removing its own systemd unit. In its container there is no such unit, so the daemon kept running after AgentReset. There, the host's unit that runs the container stands in for it: the agent disables it, so it stays stopped across reboots, then stops it, which stops the agent as stopping its own unit does on a host. The unit file stays, since the host's provisioning wrote it, so enabling it again with a new config brings the agent back. When the daemon stops its own container, the stop's SIGTERM cancels its context, which kills systemctl; that is the stop under way, not a failure.
The container's unit read its image reference from /etc/aks-flex-node/agent-container.env, but reset removes what's in /etc/aks-flex-node, so after a reset the unit could no longer start. It moves to /etc/sysconfig/aks-flex-node-container, where ACL keeps other units' settings.
check now requests a NodeReboot MachineOperation and expects the active machine to restart and the node to be Ready again. Last, it requests an AgentReset and expects the machines, their bpffs mounts, and the agent's config and state to be gone, and the container's unit to be stopped and disabled. Then it bootstraps the VM again, with a new config and the unit enabled, and expects the node to join.
This reverts commit 4701317. In a container, start now installs aks-flex-node-agent.service as it does on a host, so stopping and removing that unit is again how reset stops the agent, with main's UninstallService unchanged. The container variant, which disabled a unit the host's provisioning owned, goes.
bootstrap.sh always downloads the agent before it renders the config and runs preflight and start. In the agent's container image the agent is already there, and the image is how it's delivered. With --use-installed-agent, or AKS_FLEX_NODE_USE_INSTALLED_AGENT=true, bootstrap runs the agent in --install-dir instead, and fails if there is none. Everything else it does is unchanged.
In its container, start skipped InstallService: the container's unit, from the host's provisioning, ran the agent. So reset, which removes aks-flex-node-agent.service, had nothing to remove, and nothing could stop the agent the way main does. Now start installs aks-flex-node-agent.service in the container as on a host, and enables and starts it. There the unit runs the daemon in a container of the agent's image, which the host's provisioning names in /etc/sysconfig/aks-flex-node-container and bootstrap pulled, instead of a binary on the host. It mounts only the host's root; the image's entrypoint mounts the host paths the agent uses. start no longer decides this itself: InstallService does, as UninstallService did, and start is main's again. Reset then removes the unit, and nothing of the agent runs after it, as on a host.
The container's unit mounted each host path the agent uses at its own path, so the list lived in the host's provisioning, which Ignition applies once: a newer image that needed another path couldn't get it without reprovisioning. Now the units that start the agent's containers give them only the host's root, and the entrypoint mounts the host paths, so the list ships with the agent. It then checks the host's D-Bus, as before, and runs the command it's given, rather than running start once and the daemon or idling. The image also carries main's bootstrap.sh, and aks-flex-node-bootstrap runs it with --use-installed-agent, with bootstrap.sh's inputs read, not evaluated, from /etc/aks-flex-node/bootstrap.env. It adds hostname, which bootstrap.sh names the node with. Bind mounts from /host and their propagation work as spc_t too, so this doesn't depend on the SELinux workaround.
check runs nine groups of checks, which take about 19 minutes together. Now check --only runs some of them, in their usual order, for working on one area, and check ends with how long each group took, for working on what takes long. Each group starts from a joined node and leaves one, so any can run alone; without upgrade, agentreset joins again with the install's image.
Under the SELinux workaround, every start of one of the agent's
containers copies its image on containerd's native snapshotter, and
check starts about 20 of them. With QEMU's default caching, which
honors the guest's flushes, each took 21 to 30 seconds from its
unit's start to its entrypoint.
Now the VM's disk is cache=unsafe: the host caches its writes and
ignores its flushes. That risks only a disk QEMU dies holding, and the
harness creates this one for the VM and drops it with it, by up again
or down; a reboot of the guest keeps QEMU running.
Measured on the same host, with the restart-after-recovery fix:
before after
agent container 30.2 s 12.2 s
bootstrap 21.2 s 12.5 s
first bootstrap 90 s 45 s
check, 72 checks 17m14s 12m54s
The docs give the new timings, and check's count.
After an AgentUpgrade switches the agent, the daemon schedules its own restart a second later, from a transient timer, so it can return from the operation's handler first. A timer's AccuracySec= defaults to a minute, within which systemd fires it when it likes: in the harness, restarts due in a second came 26, 40, and 58 seconds late, all on the same second of the minute. That was nearly half of check's upgrade checks, and each AgentUpgrade, on a host as in the container, took up to a minute longer than it needed to. Now the timer's accuracy is 100 ms. On ACL, such a timer fires 1.3 seconds after it's scheduled. The restart that recovery schedules after the start limit, under the SELinux workaround, uses the same timer, and is punctual too.
check builds two candidate images for its upgrade checks, and pushes each, about 200 MB, which took about 20 seconds. Now it records what it pushed, and reuses a candidate built from the same agent image by the same recipe, while the registry, which up and down replace, still has it. That helps when running check again without pushing a new agent image, as with check --only upgrade. check also polls a MachineOperation every second rather than every three, as it waits on six of them.
The agent's unit may start 5 times in 5 minutes, as main's unit on a host may, and systemd counts restarts toward that, including an AgentUpgrade's. A start past the limit fails the unit, which recovery takes for a failed upgrade. Now that the restart comes promptly, check --only upgrade run twice in a row put six starts in under five minutes: the second run's first upgrade was refused, rolled back, and reported failed, though its candidate never ran, and, as recovery under the SELinux workaround may not reset the unit, the agent came back only five minutes later. Before the upgrade group's four restarts, and the machines group's one, check now waits, if it must, for earlier starts, as the VM's journal records them, to age out of the unit's start limit, which it reads from the unit. README.md says why upgrade may wait. DESIGN.md describes the limit's effect on upgrades, on a host too, and TODO.md how the daemon could avoid it.
check's probe pod ran `sh -c "echo ... && sleep 3600"`. As its container's PID 1, sh ignores SIGTERM without a trap, so the pod took its 30-second grace period to stop, and check deleted it without waiting. Once the checks before it got faster, NodeReboot stopped the machine while the pod was still stopping: the machine's shutdown waited on the pod's container, and the machine's unit hit its own stop timeout, so each NodeReboot took about 50 seconds rather than 10. Now the probe exits on SIGTERM, and check waits for it to be deleted, which takes a second or two.
This branch carries #374's commit, which keeps a MachineOperation from running twice, until it merges and the branch is rebased.
With prompt restarts after upgrades and a probe pod that stops at once, check's 72 checks take about 11 minutes, and its upgrade group about 3.
This reverts commit 5a8c37c. ACL ships the SELinux rule the agent's container needs, allow spc_t kernel_t:dbus send_msg, in ACL 3.0.20261019 (build 1229918), gallery image ACLDEVEL/acl-1es-eval-kvm version 3.20261019.3. On it, without the workaround, the agent's containers reach the host's systemd over D-Bus from spc_t, and from kernel_t where containerd's directory is unlabeled, as that build ships it: check passes either way, and selinux-repro.sh reports 0 of 59 failures and no denials.
With the workaround gone, so was check's test of the domain the agent's container runs in, and up's early report of a D-Bus denial. Both are back, without it. The agent's containers run as spc_t, as executing a file of containerd's snapshots, labeled for containers, moves them to it. Where containerd's files are unlabeled, as ACL 3.0.20261019 ships them, built into the image with an image of its own, nothing moves them, and they stay kernel_t, as every host process is on ACL. check now expects whichever the label of containerd's snapshots implies, and names it. up again stops with a hint, rather than timing out, when the host's machinectl is denied from the container. The comments that blamed the workaround for kernel_t now name that build, and check's upgrade candidates are no longer squashed, which only the workaround's native snapshotter needed.
The harness boots the gallery image ACLDEVEL/acl-1es-eval-kvm version 3.20261019.3 by default, the first ACL build with the SELinux rule the agent's container needs. That version is excluded from its gallery's latest, so the default names it; an empty ACL_IMAGE_VERSION asks the gallery for its latest, as before.
ACL 3.0.20261019 ships the rule the agent's containers need, and the harness boots it without the workaround, which is reverted. README.md drops up --selinux-workaround and the native snapshotter's start time, and gives the new default image, its export, and check's time, about 8 minutes. DESIGN.md gives that build's versions, and its SELinux section describes the fix and the two domains the agent's containers run in, depending on containerd's labels: kernel_t on that build as it ships, where /var/lib/containerd comes built into the image and unlabeled, and spc_t where it's labeled, with the machines they create the other way around. Its measurements are that build's. SELINUX.md opens with the fix: the build, selinux-repro.sh's 0 of 59 failures, check passing either way, and the unlabeled /var/lib/containerd, with a question for ACL about it. TODO.md replaces the workaround's removal with what's left: the image AKS will use, raising /var/lib/containerd's label with ACL, and reprovisioning development hosts.
ACL 3.0.20261019 builds /var/lib/containerd into the image unlabeled, so every container containerd starts, the agent's included, stays kernel_t, and check only ever saw the agent there. On a build that labels it as its file_contexts has it, the agent's containers run as spc_t, where ACL's new SELinux rule applies, and its machines as kernel_t. up --label-containerd, or ACLC_LABEL_CONTAINERD=1, adds a harness unit to the VM's Ignition config that runs restorecon -R /var/lib/containerd before containerd starts, so the agent's containers run as spc_t from the first bootstrap on. check already expects the domain the labels imply. README.md and DESIGN.md describe it.
The machines group restarts the agent once, and upgrade four times, toward its unit's start limit of 5 starts in 5 minutes. With upgrades now quick, machines after upgrade waited for those starts to age out: 83 seconds of a fresh check. machines also reboots the VM, which resets the limit, so it now runs first, and upgrade starts with the limit to itself.
Philip Lombardi (plombardi89)
force-pushed
the
phlombar/acl-container
branch
from
October 20, 2026 02:46
0a8cc53 to
dca9ed4
Compare
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 20, 2026 02:46 — with
GitHub Actions
Active
GitHub Actions
Active
The image carried tools only the harness used: flex-machine, which check's machines group ran in the agent's container to make a machine of its own, and crane, which flex-machine unpacked its rootfs with, along with less and procps-ng, which nothing runs. An image that's published for others to run shouldn't ship them. check's machines group now checks the agent's own active machine: that it runs in the host's machine.slice, that systemd-run --machine and machinectl shell reach it from the agent's container, that it keeps its leader while the container restarts, and that it's back after a reboot. Repave and the resets already check that the agent makes and removes machines. harness.py machine, which ran flex-machine for experiments, and check --keep go with it. The image keeps what the agent's containers run: the agent, bootstrap.sh and the tools it requires, and the entrypoint, aks-flex-node-bootstrap, and host-exec, copied by name. It's 180 MB, from 201.
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 20, 2026 03:44 — with
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
force-pushed
the
phlombar/acl-container
branch
from
October 20, 2026 03:51
463c930 to
529c9fc
Compare
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 20, 2026 03:51 — with
GitHub Actions
Active
GitHub Actions
Active
Early adopters need the agent's image from a registry. A workflow on this branch, acl-preview-image.yml, builds it on every push and publishes it to ghcr.io/azure/aks-flex-node-agent-acl-preview, for linux/amd64, tagged sha-<commit> and latest. It tests first, attests each image's build provenance, which ties its digest to the run and its commit, and gives the reference pinned by digest in the run's summary. A newer push cancels a build in progress, so latest ends on the newest commit. README.md's "Try a preview" says how to install and upgrade with it, and where credential files go. It's temporary: before this branch merges, it's either formalized, to build the image from main with the release's version tags, like the agent's other artifacts, or removed, by reverting this commit, which holds all of it.
A new "Before merging" section says the preview image workflow is temporary: before this branch merges, it's either formalized into main's release, with version tags, or removed by reverting its commit, and either way the preview package is retired.
containerd.service reports started a moment before containerd answers on its socket, and the units that run the agent's containers started ctr then, so at boot their first start could fail with "cannot access socket", and Restart= would try again. For the agent's unit, that failed start counted toward its start limit, and started its recovery unit, which would roll back an upgrade that a reboot interrupted, for a failure that wasn't the candidate's. Now each of the four units first waits, within its start timeout, until `ctr version` answers. They also default the settings' optional variables to empty, which silences systemd's warning that they're referenced but unset. A test checks that the agent's units wait first.
With the machines group checking the agent's own machine, check runs 66 checks, in about 7 minutes.
ctr put each of the agent's containers in a cgroup of its own, such as /aks-flex/aks-flex-node, outside any unit: the agent's unit held only ctr, about 10 MB and nine tasks, so systemd neither accounted for nor could limit the agent. Now every ctr run of the agent's, in its four units and for an AgentUpgrade's check of its candidate, passes --runc-systemd-cgroup and --cgroup system-aksflex.slice:aks-flex:NAME. runc asks systemd for a scope, such as aks-flex-agent.scope, in system-aksflex.slice under system.slice, which systemd tracks and accounts for, and where limits can go. A scope is a unit of its own, so stopping a service still ends its container through ExecStop, as before. check now expects the agent's container in aks-flex-agent.scope; its other containers' scopes are aks-flex-bootstrap, -reset, -recovery, and -upgrade-check-N. A test checks the candidate check's scope. DESIGN.md describes the flags and the cgroups, and that the units wait for containerd rather than retry, and the docs give check's count, 67.
DESIGN.md's security section said to sign and verify the agent's images. The decision is simpler, as main's: an image's digest identifies it, and containerd checks what it pulls against it; the host trusts whoever provides the digest, the install's provisioning or AKS, as main trusts the SHA-256 an AgentUpgrade comes with, and verifies no signature. Preview images carry signed build provenance, for audits.
"Before merging" now holds every gate: the preview workflow, ACL's SELinux rule in the image AKS will use, and agreeing on AgentUpgrade's image parameter with AKS. The /var/lib/containerd item waits on ACL's answer to the writeup sent, and adopters' access to the ACLDEVEL image is a new item. Bootstrap says what's known of --fetch-bootstrap-data in the containers; Arc and private registries are deferred. The cgroup item is done, and keeping ctr as the launcher and trusting an image's digest are decided. Offline first boot is a question of pre-seeding the image, for ACL.
Philip Lombardi (plombardi89)
force-pushed
the
phlombar/acl-container
branch
from
October 20, 2026 04:37
529c9fc to
df70b9c
Compare
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 20, 2026 04:37 — with
GitHub Actions
Active
GitHub Actions
Active
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds a second way to run the AKS Flex Node agent, for Azure Container Linux: in privileged containerd containers of the agent's own image, provisioned by Ignition, rather than installed on the host. ACL's
/usr,/usr/localincluded, is read-only, sobootstrap.shcan't install the agent there, and the agent can't install host packages, as both do on Ubuntu.On ACL, the agent keeps
main's config, commands, unit name, MachineOperations, and reset. The new behavior is in how the agent is installed, upgraded, and reaches the host, and it applies only on ACL. Ubuntu hosts behave as onmain, apart from the two changes in bold under "Changes that apply everywhere" below.hack/acl-container/DESIGN.mdexplains the design, andhack/acl-container/README.mdexplains how to run it locally.How ACL behavior is switched on
The agent image for ACL (
hack/acl-container/image/Containerfile) setsAKS_FLEX_NODE_CONTAINERIZED=true, anddaemon.Containerized()(pkg/daemon/container.go) reads it. Nothing else sets it: on Ubuntu,bootstrap.shinstalls the release binary, and every switch below takesmain's path.setupHost, instart(pkg/daemon/start.go)PATHis the image's, and would install into the container.InstallService, instart(pkg/daemon/lifecycle.go)daemon.Run(pkg/daemon/daemon.go)RecoverAgentUpgrade(pkg/daemon/agent_upgrade.go)PreflightChecks, inpreflight(pkg/cmd/preflight/preflight.go)host-packagesandcontainerd-active, which don't apply in a containerAgentUpgrade(pkg/daemon/machineoperation_reconciler.go)imagedownloadURLThe ACL code is in
pkg/daemon/container.goandpkg/daemon/agent_upgrade_container.go. The two units it installs are inpkg/daemon/assets.New behavior on ACL
mainon Ubuntubootstrap.shdownloads a release archive and installs the agent in/usr/local/binagent-container.bu). Nothing of the agent is on the host.bootstrap.shrenders the config, runs preflight, and runsstartbootstrap.shin a container of the image, with--use-installed-agent. Its inputs areAKS_FLEX_NODE_*lines in/etc/aks-flex-node/bootstrap.env.aks-flex-node-agent.serviceruns/usr/local/bin/aks-flex-node agentctr run/host. Its entrypoint mounts the host paths the agent uses at their own paths. The host'ssystemctl,machinectl,ctr, and other tools run throughhost-exec: underchroot /host, or, for mounts, in the host's mount namespace.AgentUpgradedownloadURLandsha256, and switches blue/green binariesimage, a fully qualified reference pinned by digest, whichctrverifies on pull, asmainrequiressha256(#369). An image named by tag alone fails asInvalidParameters. The agent pulls the image, checks that its agent runs, selects it in/etc/aks-flex-node/agent-image.env, and restarts on it.main's handoff judges the result.AgentReset, the deletion signal, oraks-flex-node resetaks-flex-node resetruns through the reset unit. Afterwards nothing of the agent runs, and the install stays.bootstrap.shagain, which downloads the agent its inputs nameagent-image.envmay name the image; without it, the install's image runs.Repave and
NodeRebootrun the same code on both.Changes that apply everywhere
These are in shared code. Only the two in bold change anything on Ubuntu:
RemoveRuntimeDirs, which reset uses, empties a directory it can't remove because it's a mount point, as the agent's directories are in its container. A host's never are.scripts/bootstrap.shgains--use-installed-agent(orAKS_FLEX_NODE_USE_INSTALLED_AGENT), off by default, with tests.utilexec.Ctr(), and one new direct dependency,github.com/distribution/reference, already in the module graph.AgentUpgradeparameters. Animageparameter fails anAgentUpgradeon Ubuntu asInvalidParameters, as a missingdownloadURLdid onmain, but now with a message naming it. A request naming bothimageanddownloadURLwas accepted onmain, which ignoredimage; it's now rejected.AccuracySec=1min, so the restart came up to a minute late; it measured 26–58 s on ACL. The timer now hasAccuracySec=100ms, so everyAgentUpgrade, on Ubuntu too, finishes up to a minute sooner.Preview image (temporary)
Every push to this branch builds the agent's image and publishes it to
ghcr.io/azure/aks-flex-node-agent-acl-previewfor early adopters, through.github/workflows/acl-preview-image.yml.sha-<commit>andlatest. Each run's summary gives the reference pinned by digest.gh attestation verify oci://…@sha256:… --owner Azure.hack/acl-container/README.md, "Try a preview".The workflow is temporary, and in a commit of its own. Before this merges, it's either formalized, to build the image from
mainas part of the release with the release's version tags, or removed (TODO.md, "Before merging").Testing
hack/acl-container/harness.pyboots ACL under QEMU/KVM, creates a local kind cluster runningmain's in-cluster controller, and bootstraps the VM into it:hack/acl-container/harness.py up # exports and boots ACL 3.0.20261019 hack/acl-container/harness.py checkcheckruns 67 checks in 7 to 8 minutes and covers:NodeRebootAgentUpgrade: success, a broken candidate that rolls back, an image named by tag alone that's rejected, back again, and removal of the broken imageAgentReset,reset, and the deletion signal, each leaving no unit or link behind, and each followed by a re-join, once with a given imageIt passes on ACL 3.0.20261019 as that build ships, with the agent's containers in
kernel_t. It also passes afterup --label-containerd, which labels/var/lib/containerdas ACL'sfile_contextshas it before containerd starts, with the agent's containers inspc_t, where the new rule applies.selinux-repro.shreports 0 of 59 failures and no denials there.Go tests,
golangci-lint,scripts/bootstrap_test.sh, ShellCheck, and the docs checks pass.Known gaps
Before merging: formalize or remove the preview image workflow.
See
TODO.md. Notably:/var/lib/containerdunlabeled, so every container containerd starts stayskernel_t. The agent passes either way, but a relabel would change the domains of running nodes. This is raised with ACL as SELINUX.md's question 5.imageparameter's name needs agreement with AKS.ctrplaces the agent's container outside its unit's cgroup.