Skip to content

Stop the agent's unit last when uninstalling it - #371

Merged
Philip Lombardi (plombardi89) merged 1 commit into
mainfrom
phlombar/uninstall-service-order
Oct 19, 2026
Merged

Philip Lombardi (plombardi89) merged 1 commit into
mainfrom
phlombar/uninstall-service-order

Conversation

@plombardi89

Copy link
Copy Markdown
Contributor

Problem

When the daemon uninstalls its own unit, for an AgentReset MachineOperation or a node's deletion signal, UninstallService stops aks-flex-node-agent.service first. Stopping the unit sends the daemon SIGTERM, which cancels its context. The task then carries on:

  • Removing the unit files still works, since it doesn't use the context.
  • systemctl disable and systemctl daemon-reload fail with context canceled, and the task reports an error:
level=WARN msg="failed to disable service (may not be enabled)" unit=aks-flex-node-agent.service error="failed to start .../systemctl: context canceled"
level=ERROR msg=failed task=uninstall-service ... error="systemctl daemon-reload: failed to start .../systemctl: context canceled"

This leaves a dangling multi-user.target.wants/aks-flex-node-agent.service link, and systemd isn't reloaded. main's e2e doesn't catch it, because after an unjoin it only checks that the unit file is gone.

Fix

UninstallService now disables the unit, removes its files, and reloads systemd before it stops the unit, so nothing is left to run after the stop. systemd can still stop a running unit whose file is gone. A failed stop is still only a warning, and aks-flex-node reset still stops the daemon before it resets the node.

Testing

  • Unit test: a new table-driven test checks the order of systemctl calls, that every file is gone before systemd reloads, and a stop that ends its caller. Moving the stop back to the front makes it fail.
  • On a host: with this change, an AgentReset leaves no .wants link, the unit unloaded and inactive, and uninstall-service completing ok. Without it, it leaves the dangling link and the errors above. I checked this on an Azure Container Linux host running the agent from Run the agent as a privileged container on Azure Container Linux #370, which uses this same code path.
  • Go tests and golangci-lint pass.

UninstallService stopped aks-flex-node-agent.service first, then
disabled it, removed its files, and reloaded systemd. When the daemon
uninstalls its own unit, for an AgentReset MachineOperation or a node's
deletion signal, stopping the unit sends the daemon SIGTERM, which
cancels its context. The unit files were still removed, since that
doesn't use the context, but disabling the unit and reloading systemd
then failed with "context canceled". That left a dangling
multi-user.target.wants/aks-flex-node-agent.service link and systemd
not reloaded, and the task reported an error. main's e2e doesn't catch
it, since it only checks that the unit file is gone.

Now UninstallService disables the unit, removes its files, and reloads
systemd before it stops the unit, so nothing is left for after the
stop. systemd still stops a running unit whose file is gone. A failed
stop is still only a warning, and aks-flex-node reset still stops the
daemon before it resets the node.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused ordering change addresses self-uninstallation failures, preserves warning-only stop handling, and includes regression coverage.

No open issues 🎉

What changed in this PR

Fixes agent self-uninstallation by completing systemd cleanup before stopping the daemon, avoiding context cancellation during cleanup.

Changes:

  • Disables the unit, removes files, and reloads systemd before stopping.
  • Adds regression coverage for ordering, cancellation, and failure handling.
File Description
pkg/​daemon/​lifecycle.go Moves service stop last and adds injectable test dependencies.
pkg/​daemon/​lifecycle_test.go Tests cleanup ordering and stop/removal failures.

Review effort: Balanced

Philip Lombardi (plombardi89) added a commit that referenced this pull request Oct 19, 2026
check now re-joins the node after AgentReset with the good upgrade
candidate's image and expects the agent to run it, and after the
other resets without one and expects the install's image. After every
reset it expects no link enabling the agent's unit, which main's
UninstallService left behind until the fix this branch carries, and
after the upgrades it expects the broken candidate's image removed and
the last good one kept.

TODO.md: preflight in a container and image cleanup are done, the main
fix is PR #371, direct activation is deferred to the end, and the
preflight warnings for docker and kubelet are an unbounded bug.
@plombardi89
Philip Lombardi (plombardi89) merged commit d658de5 into main Oct 19, 2026
13 checks passed
Philip Lombardi (plombardi89) added a commit that referenced this pull request Oct 19, 2026
check now re-joins the node after AgentReset with the good upgrade
candidate's image and expects the agent to run it, and after the
other resets without one and expects the install's image. After every
reset it expects no link enabling the agent's unit, which main's
UninstallService left behind until the fix this branch carries, and
after the upgrades it expects the broken candidate's image removed and
the last good one kept.

TODO.md: preflight in a container and image cleanup are done, the main
fix is PR #371, direct activation is deferred to the end, and the
preflight warnings for docker and kubelet are an unbounded bug.
Philip Lombardi (plombardi89) added a commit that referenced this pull request Oct 19, 2026
#371 merged, and this branch no longer carries its commit. DESIGN.md
describes main's UninstallService order as main's own, rather than as
a change this branch makes, and TODO.md no longer tracks the PR.
check's test that a reset leaves no link enabling the agent's unit
stays, now checking main's behavior.
Philip Lombardi (plombardi89) added a commit that referenced this pull request Oct 20, 2026
check now re-joins the node after AgentReset with the good upgrade
candidate's image and expects the agent to run it, and after the
other resets without one and expects the install's image. After every
reset it expects no link enabling the agent's unit, which main's
UninstallService left behind until the fix this branch carries, and
after the upgrades it expects the broken candidate's image removed and
the last good one kept.

TODO.md: preflight in a container and image cleanup are done, the main
fix is PR #371, direct activation is deferred to the end, and the
preflight warnings for docker and kubelet are an unbounded bug.
Philip Lombardi (plombardi89) added a commit that referenced this pull request Oct 20, 2026
#371 merged, and this branch no longer carries its commit. DESIGN.md
describes main's UninstallService order as main's own, rather than as
a change this branch makes, and TODO.md no longer tracks the PR.
check's test that a reset leaves no link enabling the agent's unit
stays, now checking main's behavior.

This branch was successfully deployed

1 active deployment
e2e-testing — 27f0ea60 Deployed Oct 19, 2026 by plombardi89 via E2E Tests #1200
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants