[kou]

Rationale for this change

arrow-flight-sql-odbc-test sometimes aborts with GRPC_CALL_ERROR_TOO_MANY_OPERATIONS in ClientReaderWriter::Finish() of the handshake stream.

This happens when a server finishes a handshake before the client writes or reads. For example, NoOpAuthHandler on the server side and Dremio do this. In this case:

1. GrpcClientAuthSender::Write() (or GrpcClientAuthReader::Read()) fails and calls Finish() on the handshake stream.
2. The returned status is OK, so the client auth handler and AuthenticateInternal() continue.
3. AuthenticateInternal() calls WritesDone(), drains the stream and calls Finish() again.
4. gRPC rejects the second RECV_STATUS_ON_CLIENT with GRPC_CALL_ERROR_TOO_MANY_OPERATIONS and aborts the process.

GRPC_TRACE=api shows this sequence:

SEND_MESSAGE                                   -> ERROR: Attempt to send message after stream was closed.
RECV_INITIAL_METADATA + RECV_STATUS_ON_CLIENT  -> OK  (Finish() by GrpcClientAuthSender::Write())
SEND_CLOSE_FROM_CLIENT                                (WritesDone())
RECV_MESSAGE                                          (drain)
RECV_STATUS_ON_CLIENT                          -> GRPC_CALL_ERROR_TOO_MANY_OPERATIONS (Finish() by AuthenticateInternal())

Whether the server finishes before the client writes is a race. So it's flaky.

What changes are included in this PR?

• Add GrpcClientHandshakeStream that wraps the handshake stream:
  • Finish() is called at most once. The second and later calls return the status of the first call.
  • Read(), Write() and WritesDone() return false without calling gRPC after the stream is finished.
• Return the status in AuthenticateInternal() without WritesDone() and draining when the stream is already finished.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes. Flight clients don't abort the process when a server finishes a handshake early.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

• GitHub Issue: [C++][FlightRPC][ODBC] Flakey GRPC_CALL_ERROR_TOO_MANY_OPERATIONS error in ODBC Linux #49818

[github-actions]

⚠️ GitHub issue #49818 has been automatically assigned in GitHub to PR creator.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lidavidm]

Thanks!