Skip to content

fix: Accept equal timestamps in the Postgres timestamp conflict check - #425

Merged
ferenc-csaky merged 1 commit into
mainfrom
fix/postgres-timestamp-guard-ties
Oct 21, 2026
Merged

ferenc-csaky merged 1 commit into
mainfrom
fix/postgres-timestamp-guard-ties

Conversation

@ferenc-csaky

Copy link
Copy Markdown
Collaborator

What does this change?

The Postgres timestamp conflict action now overwrites a row when the incoming timestamp is equal to the stored one (>= instead of >). Flink keeps the later row on a timestamp tie, but the sink kept the earlier one, so these updates were lost:

  • DISTINCT (keep-last-row deduplication) with two versions at the same timestamp.
  • Aggregates whose MIN or MAX timestamp does not change, for example when a late event updates COUNT and SUM but not MAX(ts).
  • Regular joins with a GREATEST(...) timestamp, when only the side with the older timestamp changes (fix(skill): Timestamp reasoning in implement-sqrl sqrl#2490).

A version with an older timestamp is still skipped.

Area

  • Runner CLI and Docker image (flink-sql-runner)
  • Connectors (connectors)
  • Formats (formats)
  • Function libraries (stdlib)
  • Environment utilities (env-utils)
  • Tests and samples (testing)
  • Build / CI / dependencies
  • Docs (README.md, stdlib-docs)

How did you test it?

  • Added SqrlPostgresTimestampConflictTest. It runs the generated upsert against Postgres (Testcontainers) with a newer, an equal and an older timestamp. The equal case fails without this change.
  • Updated SqrlPostgresDialectTest for the new predicate. mvn test -pl connectors/postgresql-connector passes (17 tests), and the Spotless checks pass.
  • Ran DataSQRL main end to end with only this change in the jar. The DISTINCT tie and the MIN/MAX aggregate cases now store the same rows as Flink computes. Before, they kept the first version.

Trade-off: during a replay after a restart, a replayed version with the same timestamp as the stored row overwrites it until the replay catches up. Versions with an older timestamp are still rejected.

Checklist

  • The PR title starts with a type such as feat:, fix: or docs:, optionally with a scope (e.g. fix(flexible-json):)
  • mvn clean install passes locally and the code is formatted with mvn spotless:apply (ran the module's tests and Spotless only)
  • New or changed behavior is covered by tests (given_when_then naming, AssertJ assertions)
  • New dependency and plugin versions are defined as properties in the root pom.xml (not applicable)
  • Commits are signed off and signed (git commit -s -S)
  • README.md and stdlib-docs are updated if user-facing behavior or functions changed

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
@ferenc-csaky ferenc-csaky added the bug Something isn't working label Oct 21, 2026
@ferenc-csaky ferenc-csaky added this to the 0.12.1 milestone Oct 21, 2026
@ferenc-csaky
ferenc-csaky marked this pull request as ready for review October 21, 2026 14:48
@ferenc-csaky
ferenc-csaky enabled auto-merge (squash) October 21, 2026 14:55
@ferenc-csaky
ferenc-csaky merged commit eb37da7 into main Oct 21, 2026
18 checks passed
@ferenc-csaky
ferenc-csaky deleted the fix/postgres-timestamp-guard-ties branch October 21, 2026 14:58
ferenc-csaky added a commit that referenced this pull request Oct 21, 2026
…#425)

Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants