[FLINK-41390][table-planner] Cancel refresh jobs the catalog could not record by Timm0 · Pull Request #29843 · apache/flink

Timm0 · 2026-11-10T10:48:02Z

What is the purpose of the change

When a materialized table statement starts a continuous refresh job but the catalog then fails to record it, the job keeps running unrecorded. On CREATE the table is dropped and the job is orphaned. On RESUME, convert-to-MT and the ALTER restart the table survives, so a retry, or the ALTER rollback, starts a second job writing to the same sink. This change cancels the unrecorded job on all four paths, keeps the original error when compensation fails, and names the savepoint when SUSPEND cannot record it. This is shared new behavior for the SQL Gateway and Table API.

Brief change log

executeContinuousRefreshJob cancels the job when recording it fails. CREATE, RESUME, convert, the ALTER restart and the ALTER rollback's restart all go through it. A failed submit still only throws.
ALTER: when the new job cannot be cancelled, the table is rolled back and left SUSPENDED, and the old job is not restarted, which avoids two writers.
When the compensating drop, rollback or suspend fails, the original failure is kept as a suppressed exception. CREATE reports a table left in INITIALIZING.
When the catalog write after stop-with-savepoint fails, SUSPEND (and any ALTER that stops the job) names the savepoint and the execution.state-recovery.path hint.
New FailingCatalog-based tests in DefaultMaterializedTableExecutorTest cover each path, including the cases where the cancel fails.
statements.md (en, zh) documents the cancel behaviour.

Verifying this change

This change added tests and can be verified as follows:

Added DefaultMaterializedTableExecutorTest cases that use a FailingCatalog to fail a selected alterTable call:
- CREATE, RESUME and convert cancel a refresh job they could not record
- CREATE reports a job it could neither record nor cancel
- ALTER cancels the new job before restarting the old one from its savepoint
- ALTER rolls back to SUSPENDED and does not restart the old job when the new one cannot be cancelled
- SUSPEND names the savepoint and the execution.state-recovery.path hint when the catalog write after the stop fails
The existing MaterializedTableTableApiITCase and MaterializedTableStatementITCase still pass.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? docs

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Code Opus 5.5

flinkbot · 2026-11-10T10:55:22Z

CI report:

6bff7f1 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

raminqaf · 2026-11-11T09:26:44Z

 }
 ```

+If the refresh job starts but the catalog cannot record it, CREATE cancels the job and drops the table it created.


Is this for the case of full mode? I am asking because the next sentence starts withIn continuous mode...

It was for continuous mode, I rephrased the sentence to make it clear.

raminqaf · 2026-11-11T09:27:38Z

+    private static final String LEFT_IN_INITIALIZING_STATUS =
+            " The table was left in INITIALIZING status. No materialized table statement can drop a"
+                    + " table in this status or move it out of it. Remove it from the catalog directly.";


Do users know what INITIALIZING status is?

I thought it is described in the docs, but you are right that they don't know about it. I adjusted the error message accordingly.

AHeise

The submit → record → cancel-on-failure split is a clear improvement, and leaving the table SUSPENDED when the new ALTER job can't be cancelled is consistent and recoverable. The addSuppressed fix and the drop-failure test cover what would have been my blocker.

On commit structure: the second commit mixes that bug fix with re-quoting ~30 identifier messages, most of them outside the paths this PR touches (refresh, drop, suspend workflow, ...). I'd fold the addSuppressed fix and the new test into the first commit and keep the '%s'/asSummaryString change as a separate [hotfix] commit, so the actual change stays reviewable and bisectable.

The rest is minor, and some of it predates this PR — fine as follow-ups if you prefer.

…tor messages - Print the table as `'cat.db.t'` (`asSummaryString()`) instead of `` `cat`.`db`.`t` `` in the exception and log messages of `DefaultMaterializedTableExecutor`, as `CatalogManager` does - Update the message assertions in `MaterializedTableStatementITCase`, `MaterializedTableTableApiITCase` and `DefaultMaterializedTableExecutorTest`

AHeise

Thanks, verified all addressed threads; the deferrals to FLINK-41350 and FLINK-41437 are fine with me.

Remaining ask: the two new rollback-failure paths in alterActivatedContinuousMaterializedTable (uncancelled job + rollback fails, restart fails + rollback/restart fails) and the RESUME uncancelled hint have no tests. FailingCatalog only fails the next write matching one predicate, so the rollback cases need a second failure, e.g. a queue of matchers. With those and green CI, LGTM from my side.

AHeise · 2026-11-11T14:14:00Z

@flinkbot run azure

AHeise · 2026-11-11T16:00:16Z

@Timm0 could you squash 49640ba into the FLINK-41390 commit? It only reworks code that commit introduces, so the history reads better as hotfix + feature.

…t record - Cancel a continuous refresh job when recording it in the catalog fails, on CREATE, RESUME, convert-to-MT and the ALTER restart - Name a job that could not be cancelled in the CREATE, RESUME and convert errors, so a retry does not start a second job - ALTER: if the new job cannot be cancelled, roll back to SUSPENDED and skip restarting the old job - Keep the original error as suppressed when the compensating drop, ALTER rollback, old-job restart or suspend fails - Name the savepoint in the cancelled-job error, for a catalog write that committed but still threw - SUSPEND: name the savepoint and the `execution.state-recovery.path` hint when the catalog write after the stop fails - Add `FailingCatalog`-based tests to `DefaultMaterializedTableExecutorTest` - Document the cancel behaviour for CREATE and RESUME in `statements.md`

Timm0 marked this pull request as ready for review November 10, 2026 10:48

raminqaf self-assigned this Nov 11, 2026

raminqaf reviewed Nov 11, 2026

View reviewed changes

AHeise self-assigned this Nov 11, 2026

AHeise reviewed Nov 11, 2026

View reviewed changes

AHeise mentioned this pull request Nov 11, 2026

[FLINK-41214][table] Add Table#materializeInto for materialized tables #29838

Open

1 task done

Timm0 force-pushed the FLINK-41390 branch from 39c690b to 8f79e09 Compare November 11, 2026 11:37

AHeise reviewed Nov 11, 2026

View reviewed changes

Comment thread ...pache/flink/table/planner/delegation/materializedtable/DefaultMaterializedTableExecutor.java Resolved

Comment thread ...pache/flink/table/planner/delegation/materializedtable/DefaultMaterializedTableExecutor.java Resolved

Timm0 force-pushed the FLINK-41390 branch from 49640ba to 6bff7f1 Compare November 11, 2026 16:08

Conversation

Timm0 commented Nov 10, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

raminqaf Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

Timm0 Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

Timm0 Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AHeise left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AHeise left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AHeise commented Nov 11, 2026

Uh oh!

AHeise commented Nov 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Nov 10, 2026 •

edited

Loading