Repository navigation
Conversation
github-actions
Bot
added
doc
Contributor
tdnvidia
force-pushed
the
tdaugherty/operator-release-terminal-batch-slot
branch
from
October 31, 2026 05:47
d9732bb to
bb08a91
Compare
A node entered NodePriority when picked and left only on Complete, so a node whose stage timed out held its interruption-budget slot until someone deleted or reset it. With a budget of N and k such nodes the rollout ran at N minus k, and a batch whose every node died never formed another. NVIDIA#870 widened this: HoldsBatchSlot treats an erroring entry as active so a retrying reboot keeps its slot, and with no check for a Job that has given up, the lifecycle taints stopped releasing a dead node by accident too. The finished-Job sweep now records the nodes whose stage has timed out and the picker parks them before selection: it drops their NodePriority entry and refuses to re-admit them while the Job stands. A stage has timed out when the Job has given up and the entry sits at (stage, erroring) for the current spec, the pair the rerun predicate already keeps in place, shared through stageTimedOut. A Job gives up by going terminally Failed, or by sitting at FailureTarget past the grace window on an unreachable node, where its pod can never terminate and Failed never arrives. A node is released only once no other package Job on it is running or retrying, so the release never admits the next node beside work still in flight. The release keys on the Job, not on the erroring state, so a retrying attempt keeps its slot as NVIDIA#870 intended. Erroring stays, and a blocked node with a package entry at erroring now counts as a failed node for the batch, so a released dead node that the lifecycle taints block still reaches failureThreshold and safetyLimit. interruptPending makes a returning node re-run its interrupt. Parking refuses re-admission while the marker stands, so the slot is released once and NodeOrderOffset moves once, not every pass. A released node counts as processed for the strategy, so once every node is complete or failed the compartment forms no new batch and the node would wait for a reset forever. The picker now retries such nodes on their own at the strategy's initial batch size once their Job is gone, which keeps the documented re-run after JOB_TTL_FAILED. A long TTL keeps the node parked until someone resets or reruns it. Closes NVIDIA#869 Refs NVIDIA#890 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Taylor Daugherty <tdaugherty@nvidia.com>
tdnvidia
force-pushed
the
tdaugherty/operator-release-terminal-batch-slot
branch
from
October 31, 2026 14:26
bb08a91 to
2e81993
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Release a node's batch slot after its package stage times out, so waiting nodes can continue. Previously, a timed-out stage could hold its slot until manual intervention; if every node in a batch failed, the rollout stalled.
Failedcondition, orFailureTargetlasting more than five minutes, and anerroringentry for that stage. Jobs still running or retrying for the same NodeWright keep the node's slot.failureThresholdandsafetyLimit.Closes #869. Also addresses the starvation described in #890; that issue remains open for confirmation.
How this was verified
make fmt vet lint unit-testsfromoperator/: all 15 suites passed; lint reported no issues. Regression coverage includes slot release, active retries, failure accounting, recovery, and repeated reconciles.2e819933passed all four Chainsaw e2e pools on Kubernetes 1.34.11, 1.35.8, 1.36.4, and 1.37.0, plus deployment-policy, CLI, Helm, operator-agent, and migration tests on 1.37.0. Chainsaw suites were not run locally.AI assistance: Claude Code drafted the implementation, tests, and docs. I reviewed them and can defend every line. Codex edited this description.
Checklist
git commit -s -S.