Skip to content

[FLINK-41370][table] Reject a literal index below 1 in SQL VARIANT array access - #29806

Merged
raminqaf merged 1 commit into
apache:masterfrom
raminqaf:FLINK-41370
Nov 9, 2026
Merged

raminqaf merged 1 commit into
apache:masterfrom
raminqaf:FLINK-41370

Conversation

@raminqaf

@raminqaf raminqaf commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

SQL array access on a VARIANT returned NULL for a literal index below 1. The same literal already fails during planning for a plain ARRAY, and for VARIANT.at(INT) in the Table API. With a VARIANT column v holding [1, "hello", 3.14]:

SELECT v[0] FROM t;
Expression Before After
v[0] NULL Array element access needs an index starting at 1 but was 0.
v[-1] NULL Array element access needs an index starting at 1 but was -1.
v[-4294967295] NULL Array element access needs an index starting at 1 but was -4294967295.
$("v").at(0L) in Table API NULL Array element access needs an index starting at 1 but was 0.
v[i] with i = 0 NULL NULL

The index is 1-based, so such a literal can never address an element. A computed index below 1 still returns NULL, as it does for ARRAY. A computed BIGINT index outside the INT range also still returns NULL, as FLINK-41330 made it. DuckDB and Trino, the other engines with a 1-based VARIANT index, also reject index 0. The Table API check already rejected an INT literal in at(). A TINYINT, SMALLINT or BIGINT literal now fails too.

An expression with such a literal returned NULL for every row and now fails during planning. This includes a literal in a branch that never runs, such as CASE WHEN i > 0 THEN v[i] ELSE v[0] END, as it does for ARRAY. The check runs during code generation, so COMPILE PLAN still succeeds and EXECUTE PLAN fails, again as for ARRAY. No released version holds such a plan, because Flink 2.3.0 can't round-trip a compiled plan with a VARIANT expression through JSON (Unsupported RelDataType: VARIANT).

I plan to backport this to release-2.4 so it ships with 2.4.0, the first release that can restore a compiled plan with v[i].

Brief change log

  • ScalarOperatorGens.generateVariantGet runs the ARRAY literal-index check. The check moves into a shared helper and matches a literal of any integer type, since a VARIANT index can be TINYINT to BIGINT. ARRAY still only reaches it with INT, because requireInteger runs first.
  • The variant '[' INT ']' entry in sql_functions.yml says that a literal index below 1 fails during planning.

Verifying this change

This change added tests and can be verified as follows:

  • Added the three literals above to VARIANT_ARRAY_ERROR_ACCESS in VariantSemanticTest. Without the fix they fail with Expecting code to raise a throwable.
  • v[-4294967295] is a BIGINT literal, so it pins that the check accepts any integer literal and compares it as a long. Narrowed to int, the literal would be 1 and pass.
  • The existing VARIANT_ARRAY_ACCESS_WITH_NON_LITERAL_INDEX covers a computed index of 0, which still returns NULL.
  • ArrayTypeTest#testArrayIndexStaticCheckForSql still pins the ARRAY message.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no, the check runs during planning
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code 2.1.319 (Claude Opus 5.5)

@flinkbot

flinkbot commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@raminqaf
raminqaf force-pushed the FLINK-41370 branch 5 times, most recently from 88632e5 to 04960d1 Compare November 9, 2026 08:27

@twalthr twalthr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @raminqaf

…ray access

SQL `v[0]` and `v[-1]` on an array VARIANT returned NULL for every row, while the same literal fails during planning for a plain ARRAY and for `VARIANT.at(INT)` in the Table API. The index is 1-based, so such a literal can never address an element.

The ARRAY literal-index check now also runs for a VARIANT index of any integer type. A computed index below 1 still returns NULL.

Generated-by: Claude Code 2.1.319 (Claude Opus 5.5)
@raminqaf
raminqaf merged commit 485da2c into apache:master Nov 9, 2026
1 check passed
@raminqaf
raminqaf deleted the FLINK-41370 branch November 9, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants