Skip to content

[kafka] Miscellaneous Kafka compatibility improvements - #4865

Merged
wuchong merged 7 commits into
apache:mainfrom
gyang94:codex/kafka-misc
Oct 26, 2026
Merged

wuchong merged 7 commits into
apache:mainfrom
gyang94:codex/kafka-misc

Conversation

@gyang94

@gyang94 gyang94 commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Closes #4864.

A collection of miscellaneous Kafka compatibility improvements.

Brief change log

  • Report corrupt or truncated compressed records with the Kafka error code while preserving codec diagnostics and exception causes.
  • Bundle Kafka clients without duplicating server dependencies and include the Kafka JAR in the binary distribution.
  • Add compression regression tests and wait for table metadata readiness in concurrency tests.

Tests

  • Clean reactor build and checks: mvn -o -B -pl fluss-kafka -am -DskipTests clean verify.
  • 507 passing tests: 473 Kafka tests and 34 RPC/configuration tests.
  • Packaged-JAR plugin loading, GZIP/Snappy/LZ4/Zstd round-trips, and zero duplicate classes with the server JAR.

API and Format

No configuration or storage format changes.

Documentation

No documentation changes required.

  • Generative AI used: Codex (GPT-6).

@wuchong

wuchong commented Oct 25, 2026

Copy link
Copy Markdown
Member

License build is failed, please check it.

Comment thread website/docs/kafka-compatibility.md Outdated
Comment thread website/docs/kafka-compatibility.md Outdated
gyang94 and others added 5 commits October 26, 2026 19:12
Add dynamic service admission, ordered draining and non-retryable disabled
responses. Preserve compressed-record diagnostics and package Kafka clients
with server-provided codecs. Consolidate tests and document the lifecycle.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 772/773
AI-Contributed/UT: 2001/2001
Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 19/19
AI-Contributed/UT: 0/0
Preserve the Kafka NOTICE under META-INF and exclude duplicate root legal files.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 34/34
AI-Contributed/UT: 0/0
Move service switching instructions under cluster configuration and explain
client behavior in user-facing terms. Remove packaging implementation details.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 30/30
AI-Contributed/UT: 0/0
Keep admission checks at dispatch, backend entry, and locked native submission.
Remove repeated checks from Produce parsing and per-partition conversion.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 4/4
AI-Contributed/UT: 0/0
gyang94 and others added 2 commits October 26, 2026 20:46
Treat Kafka enablement as a best-effort admission switch. Let admitted
requests complete without rechecking service state or locking submission
against disable. Remove the guard plumbing and strict ordering tests,
and cover disable during Produce for all acknowledgement modes. Wait for
table metadata before the concurrency test sends its first Produce request.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 224/224
AI-Contributed/UT: 201/201
Restore Kafka service startup behavior and remove dynamic admission,
connection draining, protocol shutdown hooks, and their tests and docs.
Retain compression diagnostics, packaging, and distribution integration.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 519/519
AI-Contributed/UT: 1562/1562
@wuchong
wuchong merged commit 1da0ead into apache:main Oct 26, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[kafka] Miscellaneous Kafka compatibility improvements

2 participants