Repository navigation
sysfs: don't hang on cpufreq files that return EAGAIN - #879
Merged
Merged
Conversation
|
started a thread on CNCF slack about incorporating this change https://cloud-native.slack.com/archives/C01AUBA4PFE/p1792097716085379 -- or we can continue here but i would like to +1 this as we're hitting it in our org accross nvidia GBx00 systems on 6.x kernels in node_exporter |
SuperQ
requested changes
Oct 16, 2026
On arm64, reading cpuinfo_avg_freq returns EAGAIN when the CPU has not ticked recently and every other CPU in the policy is idle (arch_freq_get_on_cpu: "No alternative to pull info from"). os.ReadFile registers sysfs fds with the netpoller, and on EAGAIN waits for a readiness edge that kernfs never sends, so the read never returns. SystemCpufreq waits on every per-CPU read, so the whole call hangs, and node_exporter's cpufreq collector leaks a goroutine and an fd on every scrape. Read the uint attributes with SysReadUintFromFile, which already exists for this failure mode and is already used for the string attributes, and skip the value on EAGAIN like a missing or unreadable file. Fixes prometheus/node_exporter#3791 Signed-off-by: Ben Lovett <64740+blovett@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blovett
force-pushed
the
sysfs-cpufreq-eagain
branch
from
October 16, 2026 15:14
d93f108 to
571d678
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes prometheus/node_exporter#3791.
On arm64, a read of
cpuinfo_avg_freqcan returnEAGAIN. This occurs when all CPUs in the policy are idle. #829 added this file toSystemCpufreq.os.ReadFileadds the file to the Go netpoller. AfterEAGAIN, it waits for a readiness event. kernfs does not send this event. Because of this, the read does not complete andSystemCpufreqdoes not return. For each scrape, node_exporter 1.12 does not release one goroutine and one file descriptor.SysReadFileprevents this problem.parseCpufreqCpuinfouses it for the string attributes. This change also usesSysReadUintFromFilefor the uint attributes. If the read returnsEAGAIN, the function does not use the value. This is the same as for a missing file.Testing
make testandgolangci-lintpass. The test fixtures cannot returnEAGAIN. For this reason, this change does not add a unit test. If you want a unit test, I can add one.I built node_exporter 1.12.1 with procfs master, and with procfs master and this change. I scraped each binary 20 times on a Radxa Orion O6 (arm64,
cppc_cpufreq):The end-to-end test output is the same for the two binaries.
#861 is a different solution. It sets a 5 s deadline for each read. This also stops the leak. But on an idle system, most calls took 5 s.
@SuperQ
Generated with assistance by Claude Code.