Repository navigation

[SPARK-60101][SQL] Support native data sources implemented in Rust or C++ - #59313

Closed

HyukjinKwon wants to merge 15 commits into

apache:masterfrom

HyukjinKwon:SPARK-60101

Closed

[SPARK-60101][SQL] Support native data sources implemented in Rust or C++#59313
HyukjinKwon wants to merge 15 commits into
apache:masterfrom
HyukjinKwon:SPARK-60101

Commits on Oct 9, 2026
- [SPARK-60101][SQL] Add a columnar data source API for JVM and native (Rust, C++) data sources
  
  HyukjinKwon
  and
  Isaac
  committed
  56acc8b
- Make NativeBridge the only public API of native data sources
  
  HyukjinKwon
  and
  Isaac
  committed
  b032a0a
- Pass the options of catalog tables to native data sources
  
  HyukjinKwon
  and
  Isaac
  committed
  1fc3fc7
- Find the native data source packages installed in SPARK_HOME automatically
  
  HyukjinKwon
  and
  Isaac
  committed
  aae6a4f
- Fix the Connect client compatibility check and the documentation build
  
  HyukjinKwon
  and
  Isaac
  committed
  a5d6708
- Find installed native data source libraries automatically
  
  HyukjinKwon
  and
  Isaac
  committed
  4a2026e
- Format NativeDataSourceConnectSuite with scalafmt
  
  HyukjinKwon
  and
  Isaac
  committed
  822d33c
Commits on Oct 10, 2026
- Address review comments on native data source lookup, distribution and lifecycle
  
  HyukjinKwon
  and
  Isaac
  committed
  9dd4f88
- Address more review comments on native data source packages and stream writers
  
  HyukjinKwon
  and
  Isaac
  committed
  3814147
- Address review comments on sessions, distributed copies, schemas and batching
  
  HyukjinKwon
  and
  Isaac
  committed
  9ac5a49
- Check the Arrow data of native data sources, create writers late, push limits once
  
  HyukjinKwon
  and
  Isaac
  committed
  acdc7eb
Commits on Oct 11, 2026
- Check the lengths of the arrays that native data sources export
  
  HyukjinKwon
  and
  Isaac
  committed
  a8f178b
Commits on Oct 13, 2026
- Address review comments on the compatibility, predicates and checks of native data sources
  
  HyukjinKwon
  and
  Isaac
  committed
  ff7b1ae
- Address review comments on predicates, strings, writes, offsets and builds of native data sources
  
  HyukjinKwon
  and
  Isaac
  committed
  aa0f3ba
- Test that a native scan reads the columns of the pushed predicates
  
  HyukjinKwon
  and
  Isaac
  committed
  e3ff80c

Commits

Commits on Oct 9, 2026

Commits on Oct 10, 2026

Commits on Oct 11, 2026

Commits on Oct 13, 2026