October 30, 2026: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days saw well over 60 pull requests and roughly 25 issues touched, with a healthy mix of merged fixes and new open work. Activity clustered into five themes: a sustained wave of Flink SQL function offload work, Delta/Lakehouse correctness and CI hardening, Spark 4.2 bring-up, ANSI mode completion on the Velox backend, and steady infrastructure/maintenance churn (Velox daily bumps, CI image slimming, Build/Format fixes). Roughly a third of the listed PRs are still drafts, many of them deliberately large and stacked, which suggests several long-running efforts are converging at once. A notable share of closed PRs this week were small, high-value fixes: test cleanups, documentation link repairs, and targeted correctness bugs.

Key Ongoing Projects

• Spark 4.2 support — @manoj-ragupathy has a three-part stack in flight: [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer #13021 adds the spark-4.2 build profile and shim layer, and [GLUTEN-12569][CORE] Add Spark 4.2 unit test module (gluten-ut/spark42) #13022 adds the gluten-ut/spark42 test module. Both remain drafts, and the UT lanes are additionally blocked on a manual workflow_dispatch of the Docker image workflow.
• Delta native write + materialization — @felipepessoto's [GLUTEN-13500][VL] Support Delta partition column materialization #13502 enables Delta's partition-column materialization in the native writer (tracked by [VL][Delta] Support materializePartitionColumns in native writes #13500), accompanied by [GLUTEN-13496][VL] Validate Gluten activation in Delta Spark CI #13499 on CI activation validation and [MINOR][VL] Configure Gluten for all Delta Spark test contexts #13131 on shared test-context configuration. The umbrella [Umbrella] Delta-on-Gluten: follow-ups after the Delta Spark UT pipeline #12743 now tracks 485 known-failure baseline entries and documents a 173-case IncrementMetric experiment.
• JNI exception fidelity — [CORE] Preserve original Java exceptions across JNI #13133 by @felipepessoto preserves original Java throwables across the JNI boundary; [VL] Translate native cast failures into Spark exceptions #13135 builds directly on it to translate native cast failures into proper Spark exceptions (CAST_OVERFLOW, CAST_INVALID_INPUT, and friends). Both carry very large cross-version test matrices and are still drafts.
• Flink SQL function coverage — the largest multi-contributor push of the week. Temporal functions ([FLINK][VL] Support FROM_UNIXTIME / UNIX_TIMESTAMP / TIMESTAMPADD to the velox flinksql functions #13491 by @AnissaLiu, [GLUTEN-13469][FLINK][VL] Support CURRENT_TIMESTAMP, NOW, LOCALTIMESTAMP and LOCALTIME in the velox flinksql functions #13470 by @sunshone, [GLUTEN-13454][FLINK] Support CURRENT_DATE and CURRENT_TIME offload to velox flinksql functions #13455 by @wud-if, [GLUTEN-13471][FLINK] Support TO_DATE / TO_TIMESTAMP / TO_TIMESTAMP_LTZ offload to velox flinksql functions #13473 by @TianJialong), string/pad functions ([GLUTEN-13450][FLINK][VL] Support TRIM/LTRIM/RTRIM/LPAD/RPAD to the velox flinksql functions. #13460 by @AnissaLiu, [GLUTEN-13458][FLINK] Support REPLACE, RIGHT, FROM_BASE64 and LIKE #13459 by @zhang-jianq), hashing ([GLUTEN-13480][FLINK] Support SHA2 / SHA224 / SHA256 / SHA384 / SHA512 offload to velox flinksql functions #13484 by @TianJialong, [GLUTEN-13416][FLINK] Support MD5 and SHA1 #13420 by @zhang-jianq), logical predicates ([GLUTEN-13457][FLINK][VL] Support NOT, IS FALSE, IS NOT TRUE to the velox flinksql functions #13461 by @sunshone), JSON_VALUE ([GLUTEN-13467][FLINK] Support JSON_VALUE #13472 by @zhang-jianq), arithmetic ([GLUTEN-13411][FLINK] Support the arithmetic expressions offload to velox flinksql functions #13412 by @wud-if), and keyed exchange routing ([GLUTEN-12805][FLINK] route keyed exchange by key group #13409 by @ggjh-159).
• ANSI mode completion — [GLUTEN-10134][VL] Support Spark ANSI behavior for elt, conv and element_at #12873 by @zhouyuan covers elt/conv/element_at; [GLUTEN-10134][VL] Support decimal pmod with ANSI semantics #13351 by @ReemaAlzaid adds decimal pmod; [VL] Enable guarded decimal ROUND with cross-version coverage #13103 and [VL] Use upstream integral ROUND and preserve safe SQL fallback #13282 by @luis4a0 handle decimal and integral ROUND with guarded fallback; [GLUTEN-12892][VL] Add Spark 3.5 jobs to the ANSI workflow #13263 by @rdtr extends the ANSI CI workflow to Spark 3.5.
• IncrementMetric native counting — a clean three-layer split by @malinjawi: [GLUTEN-9003][VL] Add native counter functions for Delta's IncrementMetric #13357 (native counter functions), [GLUTEN-9003][CORE] Carry native counter counts into SQL metrics #13358 (JVM metrics plumbing), and [GLUTEN-9003][VL] Count Delta's IncrementMetric natively #13165 (Delta transformer wiring) to fix inflated DML metrics.
• Connector expansion — Lance read offload landed via [VL] Add Velox offload for Lance scans via Arrow C stream #12958 by @sezruby, with [GLUTEN-13487][VL] Offload Lance scans with vector columns #13488 by @sezruby fixing FixedSizeList vector columns; Iceberg V3 deletion-vector reads are progressing through [GLUTEN-12628][VL] Read Iceberg V3 deletion vectors #12629 by @wirybeaver and [VL][Iceberg] Add support for iceberg v3 deletion vector reads #13217 by @infvg.
• Bolt backend restoration — [GLUTEN-12998][BOLT] Restore backend build and execution compatibility #13492 and [GLUTEN-12998][BOLT] Enable build-only x86 CI for Bolt #13486 by @Weixin-Xu restore native build, packaging, and end-to-end execution, and add a build-only x86 CI job. The self-hosted runner proposal in [BOLT] Enable Bolt CI tests with ECS runners #13451 is still seeking maintainer coordination.
• CI infrastructure — [VL][CI] Bake CI runtime tools into the apache/gluten images #13422 by @FelixYBW bakes runtime tooling into the apache/gluten images, eliminating a large class of package-mirror flakes; @rdtr also re-enabled several Spark 4.2 tests in [GLUTEN-13178][VL] Re-enable the Spark 4.2 acosh, broadcast size and sort-aggregate tests #13259 and [GLUTEN-13179][VL] Re-enable the Spark 4.2 null-named attribute test #13260.

Priority Items

• [CI] Detect regressions in native execution and fallback coverage #13504 by @felipepessoto — a new proposal for detecting regressions in native execution vs. fallback coverage. The concrete example (Delta 4.4 partitioned writes silently using the JVM writer) is compelling and needs maintainer design buy-in before implementation spread.
• [VL] bloom_filter_agg / might_contain are rewritten to the Velox variants even when the producer and the consumer cannot be verified to be a pair #13489 by @rdtr — a genuine correctness bug: bloom_filter_agg/might_contain are rewritten to Velox variants even when the producer/consumer pair cannot be verified, producing wrong results or outright failures. Proposed fix moves the decision to the logical plan.
• [VL] Lance scan fails at runtime on vector columns instead of falling back #13487 by @malinjawi — Lance scans crash at runtime on vector and several other column types instead of falling back. [GLUTEN-13487][VL] Offload Lance scans with vector columns #13488 by @sezruby addresses the vector portion; the remainder needs plan-time fallback.
• [VL] Native memory OOMs the test/executor JVM when reading large (billions-of-rows) Delta tables with deletion vectors #12387 by @felipepessoto — the 2-billion-row deletion-vector OOM remains unresolved and the fail-fast CI workaround is still active; the re-enable attempt [GLUTEN-12387][VL] Re-enable large deletion vector tests #12866 was closed after a multi-hour hang.
• [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer #13021 / [GLUTEN-12569][CORE] Add Spark 4.2 unit test module (gluten-ut/spark42) #13022 by @manoj-ragupathy — blocked on a committer action (image rebuild); without it the Spark 4.2 UT lanes cannot validate anything.
• [VL] Rewrite selfjoin inequality to aggregate #12756 by @hhr293 — opt-in self-join-to-aggregate rewrite with a claimed ~3.1x q95 speedup; needs careful review of its many fail-closed guards.
• [GLUTEN-12998][BOLT] Restore backend build and execution compatibility #13492 by @Weixin-Xu — large (76 files) Bolt restoration that would benefit from early reviewer attention.

Notable Discussions

• #13321 by @KushShriv: a detailed Show & Tell on the two-layer RE2 regex translation strategy (Scala-side rewriting plus a strict C++ validator), with 15/15 end-to-end parity checks. Directly accompanies [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs #13278.
• [VL] Add ANSI mode support #10134 by @philo-he: the long-running ANSI umbrella, still accumulating checklist items (BRound, RoundCeil, RoundFloor, Elt, ElementAt, VAR_POP/SAMP, STDDEV variants).
• [Umbrella] Delta-on-Gluten: follow-ups after the Delta Spark UT pipeline #12743 by @felipepessoto: the Delta-on-Gluten follow-up umbrella, now with a full per-error-kind breakdown of the remaining baseline.
• [GLUTEN-9003][VL] Count Delta's IncrementMetric natively #13165 by @malinjawi: the IncrementMetric discussion, with substantial back-and-forth on placement allowlists and MERGE CASE WHEN semantics.
• [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs #13278 by @KushShriv: 20 comments of regex-translation review, including the important revision that the C++ validator must reject raw \u rather than validate a translated copy.
• [VL] Support Iceberg V3 deletion vectors in Velox #12628 by @wirybeaver: Iceberg V3 deletion-vector roadmap, scoped as a three-PR stack.

Emerging Trends

• AI-assisted authorship is now the default. The overwhelming majority of this week's PRs carry a Generated-by: trailer, and several are explicitly co-authored. Review bandwidth, not authoring capacity, appears to be the binding constraint.
• "Correct results" is no longer sufficient. Multiple efforts ([CI] Detect regressions in native execution and fallback coverage #13504, [GLUTEN-13496][VL] Validate Gluten activation in Delta Spark CI #13499, [GLUTEN-13500][VL] Support Delta partition column materialization #13502) converge on the same insight: a green test suite can hide a total loss of native execution. Coverage-of-execution is becoming a first-class CI concern alongside correctness.
• Conservative, fail-closed design is the house style. Nearly every new optimizer rule or transformer ships with explicit allowlists, documented bail-outs, and paired positive/negative tests ([VL] Rewrite selfjoin inequality to aggregate #12756, [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs #13278, [GLUTEN-13500][VL] Support Delta partition column materialization #13502).
• Backend pluralism. Velox remains dominant, but Flink, Bolt, ClickHouse, and cuDF each had dedicated work this week — and the Flink efforts repeatedly hit the same set of shared blockers (typed NULL literals, interval type mapping).
• Test-hygiene debt is being paid down deliberately — literal unhex assertions restored ([GLUTEN-12859][TEST] Restore unhex null assertions across Spark modules #13319, [GLUTEN-12861] [TEST] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 #13287, [GLUTEN-12856][TEST] Use withTable for table cleanup in Hive SQL query suites #13405), constant version gates flagged for removal ([CORE] Drop the now-constant ">= 3.4" test gates #12992), and shared config bugs fixed at the source.

Good First Issues

• [CORE] Drop the now-constant ">= 3.4" test gates #12992 by @jackylee-ch — remove the now-constant testWithMinSparkVersion(name, "3.4") gates across ~57 call sites and 14 files. Skills: Scala test code, mechanical refactoring. Excellent entry point: no behavioral change, easy to verify, and it makes every remaining version gate meaningful.
• [VL] Support TIMESTAMP_NTZ Type #11622 by @rui-mo — TIMESTAMP_NTZ support is largely complete; the remaining open item is enabling the extract.sql, timestamp-ltz.sql, and timestamp-ntz.sql suites. Skills: Scala, Spark SQL test plumbing. Well-scoped and the groundwork is already merged.
• [CH] support MonotonicallyIncreasingID expression #6811 by @taiyang-li — implement MonotonicallyIncreasingID for the ClickHouse backend. Skills: C++ and CH query-context state management. Good for someone wanting backend-internals exposure, with a clear spec in the issue.
• [VL] Add ANSI mode support #10134 by @philo-he — the ANSI umbrella still has many unclaimed checkboxes (BRound, RoundCeil, RoundFloor, Elt, ElementAt, conv, VAR_POP/VAR_SAMP/STDDEV_POP/STDDEV_SAMP). Skills: Spark expression semantics plus a Velox function PR. Pick one box rather than the whole issue — each is a self-contained "implement native function, add parity test, document support" loop.

If you are new here, start with #12992 or a single unchecked item in #10134, open a draft PR early, and drop a note in Discussions so maintainers can point you at the right reviewer.