Skip to content

Support Redis Sentinel for session store (HA-aware client) #325

@ntwkninja

Description

@ntwkninja

Summary

authservice's Redis session store constructs its client via go-redis's standalone NewClient path, which is not Sentinel-aware. In deployments backed by a Sentinel-managed HA Redis (or Valkey) cluster, this means authservice continues talking to the former primary after a Sentinel-driven failover, surfacing a sustained stream of READONLY You can't write against a read only replica errors to users until the authservice pods are manually restarted.

Raising this to confirm the problem framing and see whether maintainers would welcome a contribution in this direction.

Current behavior

NewRedisClient at internal/oidc/redis.go:71,116 parses the server_uri and hands it to redis.NewClient:

opts, err := redis.ParseURL(config.GetServerUri())
// ...
return redis.NewClient(opts), nil

redis.ParseURL only accepts redis://, rediss://, and unix:// (go-redis v9.7.3 options.go:298-305). The standalone client has no mechanism to discover a new primary on its own.

Consequence on Sentinel-driven failover:

  1. Former primary is demoted and starts rejecting writes with READONLY You can't write against a read only replica.
  2. The standalone client surfaces the error to authservice with no Sentinel-aware retry or re-resolution.
  3. Pool-level teardown eventually forces a reconnect, but DNS re-resolution on the same Service name lands back on the same endpoints.
  4. Writes remain broken until authservice pods are restarted.

Related issue #240 (still open) describes the same class of breakage attributed to replica sync lag; PR #241 attempted a fix and was closed. I suspect the root cause of both is this standalone-client construction.

Proposed direction

Use go-redis's built-in Sentinel client. NewFailoverClient returns the same *redis.Client type as NewClient, so the rest of the code (via redis.Cmdable at internal/oidc/redis.go:62) is unchanged. The failover client subscribes to Sentinel's +switch-master channel and updates its internal master address on each event — no additional client-side logic needed.

The API is already available in the go-redis/v9 v9.7.3 authservice is pinned on, so this wouldn't require a dependency bump. oauth2-proxy uses this exact pattern (its buildSentinelClient) and has shipped it in production for years.

Config-wise, an optional Sentinel sub-message on RedisConfig would be additive and preserve the existing server_uri path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions