Skip to content

[client] Invalidate stale KV scanner routing on failures - #4910

Open
fxbing wants to merge 1 commit into
apache:mainfrom
fxbing:fix/20261027-kv-scanner-leader-recovery
Open

fxbing wants to merge 1 commit into
apache:mainfrom
fxbing:fix/20261027-kv-scanner-leader-recovery

Conversation

@fxbing

@fxbing fxbing commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Purpose

Closes #4909

Fix KV batch scans repeatedly targeting an offline cached leader when callers recreate the scanner using the same Connection and Table.

Brief change log

  • Invalidate table or partition bucket metadata on InvalidMetadataException, including NetworkException, and on expired or unknown scanner sessions.
  • Cover synchronous dispatch failures, asynchronous failures, and error responses for open and continuation requests.
  • Keep the failing scan terminal and preserve existing TOO_MANY_SCANNERS retries and response-buffer cleanup. Recovery happens through a caller-created scanner.

Tests

  • Added a three-node regression: cache leader A, stop A, wait for the new leader, verify the first scan fails, then rebuild the scanner once on the original Connection and Table and read all eight rows. The rebuilt scan fails before the fix and succeeds afterward.
  • Added coverage for error classification, open and continuation failures, partition isolation, and preservation of the original failure when invalidation fails.
  • Passed 84 tests across KvBatchScannerTest, TableKvScanITCase, BatchScannerITCase, KvSnapshotAndLogBatchScannerTest, CompositeBatchScannerTest, and MetadataUpdaterTest. Checkstyle, Spotless, and RAT passed.

Full repository verification and the JDK 8 CI matrix were not run locally.

API and Format

No public API, configuration, RPC protocol, or storage-format changes.

Documentation

No documentation changes.

Generative AI disclosure

Generated-by: OpenAI Codex

Implementation assisted by OpenAI Codex; independently reviewed with DeepSeek.

Invalidate table or partition bucket metadata on routing and session failures so a caller-created scanner can resolve the new leader.

Preserve existing retry and cleanup behavior; cover rebuilt-scanner recovery with unit tests and a three-node regression.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[client] KV batch scans keep using a stale leader after failover

1 participant