Skip to content

[kafka] Stream JSON Produce value conversion - #4906

Open
zephyr45 wants to merge 1 commit into
apache:mainfrom
zephyr45:kafka-json-streaming-4863
Open

zephyr45 wants to merge 1 commit into
apache:mainfrom
zephyr45:kafka-json-streaming-4863

Conversation

@zephyr45

Copy link
Copy Markdown

Purpose

Linked issue: close #4863

Replace the JSON Produce path's complete JsonNode tree with schema-compiled Jackson streaming converters. The new path preserves the existing coercion, strict parsing, projection, and rescue contracts while avoiding a global intermediate tree for ordinary values.

Brief change log

  • Compile token converters for scalar, ROW, ARRAY, and MAP types and decode projected root fields in one streaming pass.
  • Materialize only rescued values and container-to-string values, preserving raw numeric spelling for signed zero and extreme exponents.
  • Preserve nested rescue overlays, valid siblings, parser advancement, strict UTF-8, duplicate detection, object-root/trailing-content checks, and resource limits.
  • Reuse the tree conversion contract against the streaming converter and add regression coverage for numeric fidelity and array rescue positions.
  • Add a JMH tree-versus-streaming benchmark for flat, nested, and rescue payloads.

Tests

  • mvnw.cmd -pl fluss-kafka -Dtest=JsonKafkaFieldDecoderTest,JsonKafkaRescueTest,JsonToFlussComplexConvertersTest,JsonToFlussConvertersTest,JsonToFlussFlinkCompatibilityTest,JsonToFlussArrowTest -Dsurefire.failIfNoSpecifiedTests=false test — 172 tests passed.
  • mvnw.cmd -o -B -pl fluss-kafka,fluss-jmh -DskipTests -Pjava8 clean test-compile — passed on JDK 8.
  • Checkstyle, Spotless, and RAT passed for the affected build.
  • A full fluss-kafka verify ran 413 tests: 386 passed and 27 cluster-backed tests errored during setup on Windows because the existing test harness constructs file://C:\... with a URI authority. There were no assertion failures.

JMH on JDK 17 (1 fork, 3 × 1 s warmup, 5 × 1 s measurement, GC profiler):

Shape Tree ops/s Streaming ops/s Tree B/op Streaming B/op
flat 1,170,344 1,567,340 4,112 3,256
nested 548,194 663,391 8,865 7,633
rescue 274,222 312,354 10,289 8,857

API and Format

No public API, Kafka protocol, or storage-format changes.

Documentation

No user-facing feature or configuration change.

  • Generative AI tools used: Codex (GPT-6)

Compile schema-specific Jackson token converters for JSON Produce values, preserving recursive coercion, strict parsing, and field-level rescue semantics without a global JsonNode tree. Preserve raw numeric rescue tokens and signed zero, reuse conversion contract coverage, and add representative JMH allocation and throughput comparisons.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[kafka] Convert JSON Produce values with Jackson streaming API

1 participant