Repository navigation
Stop the agent's unit last when uninstalling it - #371
Merged
Philip Lombardi (plombardi89) merged 1 commit intoOct 19, 2026
Merged
Conversation
UninstallService stopped aks-flex-node-agent.service first, then disabled it, removed its files, and reloaded systemd. When the daemon uninstalls its own unit, for an AgentReset MachineOperation or a node's deletion signal, stopping the unit sends the daemon SIGTERM, which cancels its context. The unit files were still removed, since that doesn't use the context, but disabling the unit and reloading systemd then failed with "context canceled". That left a dangling multi-user.target.wants/aks-flex-node-agent.service link and systemd not reloaded, and the task reported an error. main's e2e doesn't catch it, since it only checks that the unit file is gone. Now UninstallService disables the unit, removes its files, and reloads systemd before it stops the unit, so nothing is left for after the stop. systemd still stops a running unit whose file is gone. A failed stop is still only a warning, and aks-flex-node reset still stops the daemon before it resets the node.
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
October 19, 2026 19:45 — with
GitHub Actions
Active
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The focused ordering change addresses self-uninstallation failures, preserves warning-only stop handling, and includes regression coverage.
No open issues 🎉
What changed in this PR
Fixes agent self-uninstallation by completing systemd cleanup before stopping the daemon, avoiding context cancellation during cleanup.
Changes:
- Disables the unit, removes files, and reloads systemd before stopping.
- Adds regression coverage for ordering, cancellation, and failure handling.
| File | Description |
|---|---|
pkg/daemon/lifecycle.go |
Moves service stop last and adds injectable test dependencies. |
pkg/daemon/lifecycle_test.go |
Tests cleanup ordering and stop/removal failures. |
Review effort: Balanced
hbc (bcho)
approved these changes
Oct 19, 2026
Philip Lombardi (plombardi89)
added a commit
that referenced
this pull request
Oct 19, 2026
check now re-joins the node after AgentReset with the good upgrade candidate's image and expects the agent to run it, and after the other resets without one and expects the install's image. After every reset it expects no link enabling the agent's unit, which main's UninstallService left behind until the fix this branch carries, and after the upgrades it expects the broken candidate's image removed and the last good one kept. TODO.md: preflight in a container and image cleanup are done, the main fix is PR #371, direct activation is deferred to the end, and the preflight warnings for docker and kubelet are an unbounded bug.
1 task done
Philip Lombardi (plombardi89)
added a commit
that referenced
this pull request
Oct 19, 2026
check now re-joins the node after AgentReset with the good upgrade candidate's image and expects the agent to run it, and after the other resets without one and expects the install's image. After every reset it expects no link enabling the agent's unit, which main's UninstallService left behind until the fix this branch carries, and after the upgrades it expects the broken candidate's image removed and the last good one kept. TODO.md: preflight in a container and image cleanup are done, the main fix is PR #371, direct activation is deferred to the end, and the preflight warnings for docker and kubelet are an unbounded bug.
Philip Lombardi (plombardi89)
added a commit
that referenced
this pull request
Oct 19, 2026
#371 merged, and this branch no longer carries its commit. DESIGN.md describes main's UninstallService order as main's own, rather than as a change this branch makes, and TODO.md no longer tracks the PR. check's test that a reset leaves no link enabling the agent's unit stays, now checking main's behavior.
Philip Lombardi (plombardi89)
added a commit
that referenced
this pull request
Oct 20, 2026
check now re-joins the node after AgentReset with the good upgrade candidate's image and expects the agent to run it, and after the other resets without one and expects the install's image. After every reset it expects no link enabling the agent's unit, which main's UninstallService left behind until the fix this branch carries, and after the upgrades it expects the broken candidate's image removed and the last good one kept. TODO.md: preflight in a container and image cleanup are done, the main fix is PR #371, direct activation is deferred to the end, and the preflight warnings for docker and kubelet are an unbounded bug.
Philip Lombardi (plombardi89)
added a commit
that referenced
this pull request
Oct 20, 2026
#371 merged, and this branch no longer carries its commit. DESIGN.md describes main's UninstallService order as main's own, rather than as a change this branch makes, and TODO.md no longer tracks the PR. check's test that a reset leaves no link enabling the agent's unit stays, now checking main's behavior.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the daemon uninstalls its own unit, for an
AgentResetMachineOperation or a node's deletion signal,UninstallServicestopsaks-flex-node-agent.servicefirst. Stopping the unit sends the daemon SIGTERM, which cancels its context. The task then carries on:systemctl disableandsystemctl daemon-reloadfail withcontext canceled, and the task reports an error:This leaves a dangling
multi-user.target.wants/aks-flex-node-agent.servicelink, and systemd isn't reloaded.main's e2e doesn't catch it, because after an unjoin it only checks that the unit file is gone.Fix
UninstallServicenow disables the unit, removes its files, and reloads systemd before it stops the unit, so nothing is left to run after the stop. systemd can still stop a running unit whose file is gone. A failed stop is still only a warning, andaks-flex-node resetstill stops the daemon before it resets the node.Testing
systemctlcalls, that every file is gone before systemd reloads, and a stop that ends its caller. Moving the stop back to the front makes it fail.AgentResetleaves no.wantslink, the unit unloaded and inactive, anduninstall-servicecompletingok. Without it, it leaves the dangling link and the errors above. I checked this on an Azure Container Linux host running the agent from Run the agent as a privileged container on Azure Container Linux #370, which uses this same code path.golangci-lintpass.