Repository navigation
Conversation
KafkaCommandDecoder never registered its channels with the RequestChannel that serves them, so backpressure on the request queue could not pause reads on Kafka listener connections; under load the unbounded queue kept growing while queued Kafka requests held on to their borrowed buffers. Register the channel in channelActive and unregister it in channelInactive, mirroring NettyServerHandler for the Fluss protocol. Unregistering also removes the closed channel from the backpressure bookkeeping so connection churn cannot leak channels into the registered-channel set. Add a @VisibleForTesting isChannelRegistered probe to RequestChannel and KafkaChannelBackpressureTest covering pause, resume, unregister-on-close, and re-delivery of frames withheld by FlowControlHandler while paused.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
KafkaCommandDecodernever registered its channels with theRequestChannelthat serves them, so backpressure on the request queue could not pause reads on Kafka listener connections: under Kafka-side overload the unbounded per-processor request queue kept growing while queued Kafka produce requests retained their borrowed (potentially large) buffers, and connections feeding the queue were invisible to the pause/resume mechanism that already protects Fluss-protocol connections (NettyServerHandlerregisters/unregisters its channels). This wires Kafka listener channels into the same mechanism.Linked issue: part of umbrella #4185 (Kafka Protocols Compatibility – Basic Produce), whose Future Work lists bounded Produce admission and flow control. Per the umbrella's review workflow, this PR references it without closing it.
Brief change log
KafkaCommandDecoder: register the channel inchannelActiveand unregister it inchannelInactive, mirroringNettyServerHandlerfor the Fluss protocol. Unregistering also removes closed channels from the backpressure bookkeeping so connection churn cannot leak channels (and their pipelines) into the registered-channel set.channelInactivefollowsNettyServerHandler'stry/finallyshape so pipeline propagation still happens if cleanup ever throws.RequestChannel: add a@VisibleForTesting isChannelRegistered(Channel)read-only probe so tests can pin the unregister half of the lifecycle (without it, a droppedunregisterChannelcall is unobservable becausepauseAllChannels/resumeAllChannelsskip inactive channels anyway).Tests
New
KafkaChannelBackpressureTest(plainEmbeddedChannel, no cluster fixture) covers:putRequestpath pauses the channel's reads when the backpressure threshold is crossed, and draining below the resume threshold resumes them;isChannelRegisteredno longer reports the closed channel;FlowControlHandlerand re-delivered (not lost) once reads resume.Red/green verified: removing the
registerChannelcall fails all three tests; removing theunregisterChannelcall fails the unregister test.mvn test -pl fluss-rpc,fluss-kafkais green on top of current main.API and Format
No public API or storage format change; the only main-code addition besides the two lifecycle calls is a
@VisibleForTestingaccessor on the internalRequestChannel.Documentation
No new feature or configuration option; no documentation change needed.
Generative AI disclosure: