Skip to content

fix: Fill in uuid and auth columns before building mutation message keys - #2530

Merged
ferenc-csaky merged 1 commit into
mainfrom
fix/mutation-key-computed-columns
Oct 28, 2026
Merged

ferenc-csaky merged 1 commit into
mainfrom
fix/mutation-key-computed-columns

Conversation

@ferenc-csaky

Copy link
Copy Markdown
Collaborator

What does this change?

The server built the Kafka message key of a mutation record before it filled in the uuid and auth metadata columns. A PARTITIONED BY on such a column therefore keyed every record {} and sent all of them to one partition. Upsert mutations whose PRIMARY KEY is a uuid or auth column had the same problem, on a compacted topic.

The server now builds the key after it fills in those columns. The key also always contains every key column, so a nullable column the client omits gives {"col":null}, the same as an explicit null and the same key Flink writes since #2527.

The planner now rejects PARTITIONED BY columns that are not physical, such as the timestamp metadata column, which the server only knows after the record is written. Since #2527, Flink already rejects these at compile time, but its error names the generated key.fields option and has no useful location. The new error names the PARTITIONED BY column, points at the CREATE TABLE, and lists the columns that can be used.

Closes #2511

Area

  • Planner / compiler (sqrl-planner)
  • CLI (sqrl-cli)
  • Server (sqrl-server)
  • Deployment model (sqrl-deployment-model)
  • Schema discovery (sqrl-discovery)
  • Agent image and skills (agent)
  • Tests (sqrl-testing)
  • Build / CI
  • Docs (documentation)

How did you test it?

  • MutationConfigurationImplTest checks the serialized key for a computed key column, for an omitted and an explicit null key column, and for a table without key columns. The first two fail on the previous code.
  • DAGPlannerTest with partitionedKafkaMutationTimestampKey-fail.sqrl: the auth key column is accepted and the timestamp key column is rejected.
  • The kafka-message-keys use case has a new Alert table keyed by (region, alert_id), where alert_id is a uuid column. The snapshot shows the same key for records the server writes with an omitted and an explicit null region, and for a record Flink forwards (the random alert_id is masked).

Ran locally: MutationConfigurationImplTest, KafkaLogEngineTest, DAGPlannerTest, UseCaseCompileTest, and FullUseCaseIT for kafka-message-keys (7/7). A full FullUseCaseIT run before rebasing on main failed 5 use cases that do not use message keys (filtered-distinct, flink, sensors-teaser, simple-with-bigint-support, avro-schema). All 5 had timestamps shifted by whole hours, which points to the local timezone.

Checklist

  • The PR title starts with a type such as feat:, fix: or docs:
  • mvn clean install passes locally and the code is formatted with mvn -P dev initialize
  • New or changed behavior is covered by tests (given_when_then naming, AssertJ assertions)
  • Snapshots are updated and the diffs are reviewed, if compiler output changed
  • New dependency and plugin versions are defined as properties in the root pom.xml
  • Commits are signed off and signed (git commit -s -S)
  • Docs and agent skills are updated if user-facing behavior changed

🤖 Generated with Claude Code

#2511
Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
@ferenc-csaky ferenc-csaky added the bug Something isn't working label Oct 28, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good — no material findings.

Acceptance: key construction now follows server-filled metadata; nullable omitted keys serialize consistently; timestamp metadata partition keys are rejected; documentation and agent guidance match.

[RISK ASSESSMENT]
MEDIUM — this changes mutation-key serialization, but the server, planner, and Kafka/Flink physical-column paths agree. Verdict: approve. Architectural insight: the mutation entry is the authoritative record, so deriving the key only after computed metadata is applied keeps server and Flink keys aligned.

Full focused Maven tests could not run locally because the environment cannot authenticate to required flink-sql-runner snapshot packages (HTTP 401); current-head CI build is still in progress. This review was generated by an AI agent (OpenHands) on behalf of the user.


Was this automated review useful? React with 👍 or 👎 to this review to help us measure review quality.
Workflow run: https://github.com/DataSQRL/sqrl/actions/runs/40711330265

@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 57.89474% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.19%. Comparing base (e9cc34d) to head (fcf07ae).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
.../com/datasqrl/planner/Sqrl2FlinkSQLTranslator.java 33.33% 7 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2530      +/-   ##
============================================
- Coverage     71.20%   71.19%   -0.01%     
- Complexity     2049     2061      +12     
============================================
  Files           678      678              
  Lines         20657    20666       +9     
  Branches       2593     2590       -3     
============================================
+ Hits          14708    14713       +5     
- Misses         4625     4632       +7     
+ Partials       1324     1321       -3     
Flag Coverage Δ
container-e2e 46.56% <26.31%> (-0.01%) ⬇️
integration 61.58% <57.89%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ferenc-csaky
ferenc-csaky merged commit 9ff1527 into main Oct 28, 2026
20 checks passed
@ferenc-csaky
ferenc-csaky deleted the fix/mutation-key-computed-columns branch October 28, 2026 15:56
ferenc-csaky added a commit that referenced this pull request Oct 30, 2026
…eys (#2530)

Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mutation Kafka key is empty when PARTITIONED BY uses a uuid or auth METADATA column

2 participants