[WIP][SQL] Prototype a Rust SDK for native batch data sources

[gengliangwang]

Important

Updated Rust SDK: #59520 is the companion SDK for the FFM/C ABI in #59424. It adds an explicit serializable reader factory, cooperative task cancellation, and batch-only capabilities.

This PR contains the earlier JNI prototype targeting #59313. Please direct further SDK development and review to #59520.

What changes were proposed in this pull request?

Add a standalone Rust SDK prototype for batch reads under sql/native/rust,
with a separate Range connector under examples/src/main/rust/native-data-source.
The Cargo workspace and lockfile live with the SDK. The example is a workspace
member that depends on the SDK through a Cargo path dependency.

Connector authors implement BatchSource and ScanBuilder, return ordinary Arrow
RecordBatchReader instances, and register the provider with
export_data_source!(RangeSource). The SDK handles JNI exports, native handles,
typed plan serialization, errors, and Arrow C Stream ownership. Rust traits stay
inside the plugin; they are not a cross-library Rust ABI.

The prototype includes:

• Filter pushdown with Exact, PruningOnly, and Unsupported results; global
  versus per-partition limit support; and top-level column projection. The bridge
  declines limit pushdown when filters remain residual.
• A Range connector with lazy bounded batches, disjoint partitions, integer
  filter pushdown, global limits, and empty/reordered projections.
• A .sparkpkg packaging helper, a Spark smoke script, and build/API documentation.

Runtime dependency: the adapter targets the batch JNI ABI from #59313 at
ff7b1ae0150fbcb3b6a7c67bea8accf6aa4ad9bd. This PR does not include that framework.
Running the connector in Spark requires a distribution containing it. The
standalone Rust and JNI tests do not require building Spark.

This draft is intended to evaluate a connector-facing SDK on top of the proposed
native protocol. It does not establish a stable SDK or modify Spark's Maven/SBT
build. Writes, Structured Streaming, C++ bindings, cancellation, and Spark memory
accounting are outside this demo's scope. The README documents the pinned
framework's Arrow layout restrictions and the remaining integration work.

Why are the changes needed?

Implementing native connectors directly against JNI requires each connector to
repeat handle management, plan serialization, error conversion, and Arrow
lifecycle code. A Rust SDK lets connector authors focus on planning and reading
while making pushdown semantics explicit. This working prototype gives reviewers
a concrete API and connector to evaluate alongside #59313.

Does this PR introduce any user-facing change?

It adds an opt-in example and experimental Rust API. There is no change to Spark's
built-in data sources or default build, and no published crate or compatibility
commitment. With the required framework, the packaged example can be loaded via
spark.addArtifact(...) and spark.read.format("rust_sdk_range").load().

How was this patch tested?

From sql/native/rust, using Rust 1.89.0 and JDK 17:

cargo build --workspace --locked
cargo test --workspace --locked
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo fmt --all -- --check
python3 ../../../examples/src/main/rust/native-data-source/package.py \
  --library target/debug/libspark_datasource_rust_sdk_range.so \
  --output target/rust_sdk_range.sparkpkg

All 19 tests passed: 18 Rust unit tests and one integration test that compiles a
minimal Java fixture, starts a real JVM with -Xcheck:jni, loads the connector
shared library, and consumes its Arrow C Stream. Coverage includes serialized
plans, exact/residual filters, limits, empty projections, integer boundaries,
retained batch ownership after stream release, and provider errors/panics.

Also checked the package manifest and embedded library bytes, Python script
syntax, license headers, formatting, and whitespace.

The Spark smoke script is provided but was not run: the current worktree is
based on master and does not contain #59313. The JNI fixture tests are not full
Spark integration tests. Full Spark builds and Apache RAT were not run.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6)