[Feature] Support adding nested fields to ROW and ARRAY<ROW> columns #4889

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Fluss supports adding top-level columns, including columns with complex types. However, adding fields inside an existing ROW is currently not supported.

Applications storing structured records often need to extend nested objects as their data models evolve. For example, a user profile may gain an address attribute, or each element
of an order's item array may gain a price field.

Before:

profile ROW<name STRING, address ROW<city STRING>>
items ARRAY<ROW<sku STRING, quantity INT>>

After:

profile ROW<name STRING, address ROW<city STRING, country STRING>>
items ARRAY<ROW<sku STRING, quantity INT, price DOUBLE>>

Existing records should remain readable, with NULL values for newly added fields, without rewriting historical data.

This issue tracks nested field additions as an extension of the schema evolution work in #1154.

Solution

ntroduce append-only nested field additions in two stages:

1. Support adding nullable fields to existing ROW types, including multiple levels of nested ROWs.
2. Extend support to ROW types inside ARRAY elements.

Proposed semantics:

• Append each new field to the end of its parent ROW.
• Preserve existing field IDs and allocate new IDs for added fields.
• Adapt records across schema versions recursively using field IDs.
• Return NULL for newly added fields in historical records.
• Preserve NULL parent rows and NULL array elements.
• Support both log tables and primary-key tables, including reads, KV writes, lookups, scans, and recovery.
• Provide Java Admin API and Flink SQL integration.

The initial scope is limited to non-key columns in tables with lake tiering disabled. Changes beneath primary key, bucket key, or partition key columns are excluded. Field
additions inside MAP types, field removal, renaming, reordering, and type conversion are outside this initial scope.

The implementation will be contributed incrementally: recursive ROW adaptation, read-path integration, KV write and recovery support, the schema change API, Flink SQL integration,
and ARRAY support.

Validation will cover historical data, supported combinations of writer and reader schema versions, local and remote log reads, KV lookups, snapshot reads, recovery, and
unsupported-change validation. Existing top-level ADD COLUMN behavior should remain unchanged.

A follow-up FIP will define the public API, nested path representation, SQL syntax, and compatibility and upgrade requirements.

Anything else?

Related work:

• [Umbrella][Feature] Schema Evolution Support #1154 — Schema Evolution Support.
• Support Schema Evolution (Add Column at Last) #2067 — Support Schema Evolution (Add Column at Last).
• Support Add Column at Last for Complex Data Types #2071 — Support Add Column at Last for Complex Data Types.
• Add Nested Row 's field_id  #2310 / [common] Add field_id for Nested Row. #2322 — Field IDs for nested ROW fields.

I already have a local prototype covering nested ROW and ARRAY element evolution, including tests for read/write paths and recovery.

I am currently preparing the first draft PR for recursive ROW schema evolution plans and will link it here once submitted. I will also prepare a FIP for discussion on the dev
mailing list. The implementation and public interfaces will be refined based on community feedback.

I would like to drive this contribution. Could a maintainer assign this issue to me? Please coordinate in this issue before starting overlapping implementation work; I am happy to
collaborate and divide the remaining tasks.

Willingness to contribute

• I'm willing to submit a PR!