Skip to content

sysfs: parse NVMe diag counters (controller, namespace, multipath head) - #887

Merged
SuperQ merged 2 commits into
prometheus:masterfrom
lpneyon:nvme-telemetry
Oct 20, 2026
Merged

SuperQ merged 2 commits into
prometheus:masterfrom
lpneyon:nvme-telemetry

Conversation

@lpneyon

@lpneyon lpneyon commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #885, as mentioned there.

Linux 7.2 added diag/ attribute groups with error and recovery counters for NVMe controllers, namespace block devices and native-multipath namespace heads. This adds methods to read them on demand:

Method sysfs source
NVMeDevice.Diag() /sys/class/nvme/<ctrl>/diag/{command_error_count,reset_count,reconnect_count} (reconnect_count is fabrics-only)
NVMeDevice.NamespaceDiag(ns) /sys/class/nvme/<ctrl>/<ns>/diag/{command_retries_count,command_error_count,multipath_failover_count}
NVMeSubsystem.NamespaceDiag(ns) /sys/class/nvme-subsystem/<subsys>/<ns>/diag/{io_requeue_no_usable_path_count,io_fail_no_available_path_count}

The head counters show I/O that was requeued or failed because every path was down, which the per-path counters cannot show.

NVMeClass() and NVMeSubsystemClass() read exactly the same files as before; the diag files are only read when a method is called. NVMeDevice and NVMeSubsystem gain a Path field with their sysfs directory for the methods to read from. Every counter is a pointer and stays nil when the kernel does not provide the attribute, so older kernels and PCIe controllers (no reconnect_count) work unchanged. Namespace names that do not match the NVMe naming pattern are rejected.

Tests: fixtures extended with diag/ attributes for controllers, namespaces and a multipath head; new tests cover each method, a kernel without diag/, and invalid names. Test pointers use new(uint64(...)) following #865.

The node_exporter nvme and nvmesubsystem collector changes that expose these are ready and will follow once this is merged.

@SuperQ SuperQ left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we should split this out into a separate function from NVMeClass().

Adding a whole new dataset is going to increase the amount of load caused by all the file reading/parsing when it's less used.

Maybe we could implement this as a functions on NVMeDevice

Something like this:

func (d *NVMeDevice) NamespaceDiag() NVMeNamespaceDiag {
}

Linux 7.2 added a diag/ attribute group to NVMe controllers and
namespace block devices:

  /sys/class/nvme/<ctrl>/diag/command_error_count
  /sys/class/nvme/<ctrl>/diag/reset_count
  /sys/class/nvme/<ctrl>/diag/reconnect_count     (fabrics only)
  /sys/class/nvme/<ctrl>/<ns>/diag/command_retries_count
  /sys/class/nvme/<ctrl>/<ns>/diag/command_error_count
  /sys/class/nvme/<ctrl>/<ns>/diag/multipath_failover_count

Add NVMeDevice.Diag and NVMeDevice.NamespaceDiag to read them on
demand, so NVMeClass does not read the extra files for callers that do
not need them. NVMeDevice gains a Path field with its sysfs directory
for the methods to read from.

Every counter is a pointer and stays nil when the running kernel does
not provide the attribute, so older kernels and PCIe controllers (which
have no reconnect_count) keep working.

Signed-off-by: Lakshmi Pradeep <lakshmi.pradeep@neyon.ai>
Linux 7.2 exposes two counters on each native-multipath namespace head:

  /sys/class/nvme-subsystem/<subsys>/<ns>/diag/io_requeue_no_usable_path_count
  /sys/class/nvme-subsystem/<subsys>/<ns>/diag/io_fail_no_available_path_count

They show I/O that was queued or failed because every path to the
namespace was down, which per-path counters cannot show. Add
NVMeSubsystem.NamespaceDiag to read them on demand for one of the
subsystem's namespaces, so NVMeSubsystemClass does not read the extra
files. NVMeSubsystem gains a Path field with its sysfs directory for the
method to read from. Counters the kernel does not provide stay nil.

Signed-off-by: Lakshmi Pradeep <lakshmi.pradeep@neyon.ai>
@lpneyon

lpneyon commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the suggestion. I've reworked it along those lines: NVMeClass() and NVMeSubsystemClass() no longer read any diag/ files, and the counters are read on demand through methods:

func (d NVMeDevice) Diag() (NVMeControllerDiag, error)
func (d NVMeDevice) NamespaceDiag(namespace string) (NVMeNamespaceDiag, error)
func (s NVMeSubsystem) NamespaceDiag(namespace string) (NVMeSubsystemNamespaceDiag, error)

NVMeDevice and NVMeSubsystem now carry a Path field so the methods know where to read. I used value receivers so they can be called directly on NVMeClass map entries, and added an error return since the reads can fail. PR description updated to match.

@SuperQ SuperQ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, I like it.

@SuperQ
SuperQ merged commit 8eaeae9 into prometheus:master Oct 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants