Feature/enhanced health

[oyiz-michael]

#1675
Is this a bug fix or adding new feature?
Adding new feature

What is this PR about? / Why do we need it?
This PR implements enhanced EFS mount health monitoring to address critical production issues where pods crash-loop when EFS mounts degrade.

Problem: Current health probes only check if the CSI driver process is running, not whether EFS mounts actually work for I/O operations. This causes false positives where pods appear healthy but can't access storage.

Solution:

Real-time monitoring of EFS mount points with actual I/O testing
Multiple health endpoints (/healthz, /healthz/ready, /healthz/live, /healthz/mounts)
Prometheus-style metrics for observability
Enhanced CSI probe integration
Configurable health check intervals and timeouts
Benefits:

Prevents pod crash-loops by detecting mount issues before applications fail
Better observability with detailed mount health metrics
Kubernetes-native integration with readiness/liveness probes
Addresses issues #336, #1411, #1156.

What testing is done?
✅ Comprehensive unit tests for all health monitoring components
✅ Full driver test suite passes (31/31 tests)
✅ HTTP endpoint testing with proper status codes and JSON responses
✅ I/O testing validates actual file operations on mount points
✅ Code formatting (gofmt) and quality (go vet) verified
✅ Binary compilation successful
✅ Zero breaking changes - backward compatible

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: oyiz-michael
Once this PR has been reviewed and has the lgtm label, please assign dankova22 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @oyiz-michael. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[oyiz-michael]

cc @justinsb @mskanth972

[oyiz-michael]

/ok-to-test

[k8s-ci-robot]

@oyiz-michael: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-ci-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[abeowlu]

@DavidXU12345 this PR is important for High Availability and EFS server-side resiliency

• wanted to clarify if there is anything blocking pick up the PR exactly?
• left a comment on the fail-fast implementation on the driver here and wondering if that is it? - I can tweak the implementation to leave the driver running for dead mounts and issue events for irrecoverable volumes for kublet instead

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten