Repository navigation
fix: Fill in uuid and auth columns before building mutation message keys - #2530
Conversation
#2511 Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
There was a problem hiding this comment.
good — no material findings.
Acceptance: key construction now follows server-filled metadata; nullable omitted keys serialize consistently; timestamp metadata partition keys are rejected; documentation and agent guidance match.
[RISK ASSESSMENT]
MEDIUM — this changes mutation-key serialization, but the server, planner, and Kafka/Flink physical-column paths agree. Verdict: approve. Architectural insight: the mutation entry is the authoritative record, so deriving the key only after computed metadata is applied keeps server and Flink keys aligned.
Full focused Maven tests could not run locally because the environment cannot authenticate to required flink-sql-runner snapshot packages (HTTP 401); current-head CI build is still in progress. This review was generated by an AI agent (OpenHands) on behalf of the user.
Was this automated review useful? React with 👍 or 👎 to this review to help us measure review quality.
Workflow run: https://github.com/DataSQRL/sqrl/actions/runs/40711330265
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2530 +/- ##
============================================
- Coverage 71.20% 71.19% -0.01%
- Complexity 2049 2061 +12
============================================
Files 678 678
Lines 20657 20666 +9
Branches 2593 2590 -3
============================================
+ Hits 14708 14713 +5
- Misses 4625 4632 +7
+ Partials 1324 1321 -3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
…eys (#2530) Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
What does this change?
The server built the Kafka message key of a mutation record before it filled in the
uuidandauthmetadata columns. APARTITIONED BYon such a column therefore keyed every record{}and sent all of them to one partition. Upsert mutations whosePRIMARY KEYis auuidorauthcolumn had the same problem, on a compacted topic.The server now builds the key after it fills in those columns. The key also always contains every key column, so a nullable column the client omits gives
{"col":null}, the same as an explicitnulland the same key Flink writes since #2527.The planner now rejects
PARTITIONED BYcolumns that are not physical, such as thetimestampmetadata column, which the server only knows after the record is written. Since #2527, Flink already rejects these at compile time, but its error names the generatedkey.fieldsoption and has no useful location. The new error names thePARTITIONED BYcolumn, points at theCREATE TABLE, and lists the columns that can be used.Closes #2511
Area
sqrl-planner)sqrl-cli)sqrl-server)sqrl-deployment-model)sqrl-discovery)agent)sqrl-testing)documentation)How did you test it?
MutationConfigurationImplTestchecks the serialized key for a computed key column, for an omitted and an explicitnullkey column, and for a table without key columns. The first two fail on the previous code.DAGPlannerTestwithpartitionedKafkaMutationTimestampKey-fail.sqrl: theauthkey column is accepted and thetimestampkey column is rejected.kafka-message-keysuse case has a newAlerttable keyed by(region, alert_id), wherealert_idis auuidcolumn. The snapshot shows the same key for records the server writes with an omitted and an explicitnullregion, and for a record Flink forwards (the randomalert_idis masked).Ran locally:
MutationConfigurationImplTest,KafkaLogEngineTest,DAGPlannerTest,UseCaseCompileTest, andFullUseCaseITforkafka-message-keys(7/7). A fullFullUseCaseITrun before rebasing onmainfailed 5 use cases that do not use message keys (filtered-distinct,flink,sensors-teaser,simple-with-bigint-support,avro-schema). All 5 had timestamps shifted by whole hours, which points to the local timezone.Checklist
feat:,fix:ordocs:mvn clean installpasses locally and the code is formatted withmvn -P dev initializegiven_when_thennaming, AssertJ assertions)pom.xmlgit commit -s -S)🤖 Generated with Claude Code