Repository navigation
[MINOR][DOC] Add developer doc on Arrow columnar conversion - #13374
Merged
Merged
Conversation
Describe how Gluten converts between rows, Spark batches, Java Arrow, native Arrow and Velox batches today (conventions, transition graph, what each transition does, copies, batch lifecycle), and how to add an Arrow-based columnar operator, using the Lance scan offload as example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Technical descriptions and test expectations need correction, and the diagram needs MkDocs rendering support.
5 open issues
What changed in this PR
Adds a developer guide to Gluten’s Arrow conversion architecture and operator integration.
Changes:
- Documents conventions, transitions, conversion costs, and batch ownership.
- Provides Lance-based integration steps, testing guidance, and ongoing-work references.
- Adds the guide to MkDocs navigation.
| File | Description |
|---|---|
mkdocs.yml |
Adds the developer-guide navigation entry. |
docs/developers/ArrowColumnarConversion.md |
Explains Arrow conversions and operator integration. |
Review effort: Balanced
Comment on lines
+77
to
+78
| ```mermaid | ||
| flowchart LR |
Comment on lines
+131
to
+133
| | Velox → Java Arrow (`load`) | No copy | Copied: Velox's `StringView` layout is exported as an Arrow string array, since Arrow string views aren't enabled | Flattened (copied) | | ||
| | Java Arrow → native Arrow (`offload`) | No copy | No copy | Not used | | ||
| | Native Arrow → Velox | No copy | A new `StringView` buffer is built (strings of 12 bytes or less are inlined) | Not used | |
| | Operator | Batch type in → out | How it's planned | | ||
| |---|---|---| | ||
| | `ColumnarArrowEvalPythonExec` (Arrow / pandas UDFs) | Java Arrow → Java Arrow | Gluten's own operator and Python runner, from `OffloadSingleNodeRules` | | ||
| | `ColumnarPartialProjectExec`, `ColumnarPartialGenerateExec` | Velox → Velox | Load to Java Arrow inside the operator, evaluate the Spark expressions row by row, and write the results with `ArrowProjection` | |
| | `ColumnarArrowEvalPythonExec` (Arrow / pandas UDFs) | Java Arrow → Java Arrow | Gluten's own operator and Python runner, from `OffloadSingleNodeRules` | | ||
| | `ColumnarPartialProjectExec`, `ColumnarPartialGenerateExec` | Velox → Velox | Load to Java Arrow inside the operator, evaluate the Spark expressions row by row, and write the results with `ArrowProjection` | | ||
| | `ColumnarRangeExec` | – → Java Arrow | Builds `ArrowWritableColumnVector`s in the JVM | | ||
| | `VeloxRDDScanTransformer`, `VeloxLocalTableScanTransformer` | – → Velox | Write rows into `ArrowWritableColumnVector`s, then convert them to Velox inside the operator | |
Comment on lines
+283
to
+284
| - The node is offloaded, and the plan has the expected transitions: none to a vanilla row consumer, | ||
| and `OffloadArrowDataExec` → `ArrowColumnarToVeloxColumnarExec` to a Velox consumer. |
- Add section 4: a Velox BIGINT column through LoadArrowData, an Arrow UDF and back to Velox, with the reference counts at each step and the release callbacks back to Velox's memory (PlantUML sequence diagram). - Explain how an ArrowBuf is released depending on whether it was allocated in Java or imported through C Data, and the memory accounting of imported buffers (second PlantUML diagram). - Replace the mermaid transition graph with rendered images. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
8 open issues
||, which Markdown interprets as an empty first column (rendering an… · Newlayout,nav_order,parent). In MkDocs,… · NewArrow UDFf. .which reads as a typo. Remove the trailing extra.… · New
Review effort: Lite
Comment on lines
+25
to
+28
| | Concept | Code | Meaning | | ||
| |---|---|---| | ||
| | `Convention.RowType` | `gluten-core/.../transition/Convention.scala` | A row format. `VanillaRowType` is Spark's `InternalRow`; `None` means the node outputs no rows | | ||
| | `Convention.BatchType` | same | A columnar batch format. `VanillaBatchType` is Spark's `ColumnarBatch`; `None` means no batches | |
Comment on lines
+1
to
+6
| --- | ||
| layout: page | ||
| title: Arrow Columnar Conversion | ||
| nav_order: 21 | ||
| parent: /developer-overview/ | ||
| --- |
Comment on lines
+129
to
+132
| This follows one fixed-width column, a `BIGINT` column `x`, through | ||
| `Velox op → LoadArrowDataExec → ColumnarArrowEvalPythonExec → OffloadArrowDataExec → | ||
| ArrowColumnarToVeloxColumnarExec → Velox op`, for a query such as `SELECT x, f(x)` with a pandas or | ||
| Arrow UDF `f`. . |
Copilot stopped reviewing on behalf of
FelixYBW due to an error
October 20, 2026 07:33
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
11 open issues
||at the start of each row (and||---|...separators), which renders as an… · New||, which Markdown interprets as an empty first column (rendering an…. .). Remove the duplicate so the sentence… · NewArrow C_data interfaceappears to be a typo/inconsistent naming; elsewhere the doc uses `Arrow C… · NewArrow UDFf. .which reads as a typo. Remove the trailing extra.…layout,nav_order,parent). In MkDocs,…
Review effort: Lite
Comment on lines
+25
to
+31
| | Concept | Code | Meaning | | ||
| |---|---|---| | ||
| | `Convention.RowType` | `gluten-core/.../transition/Convention.scala` | A row format. `VanillaRowType` is Spark's `InternalRow`; `None` means the node outputs no rows | | ||
| | `Convention.BatchType` | same | A columnar batch format. `VanillaBatchType` is Spark's `ColumnarBatch`; `None` means no batches | | ||
| | `ConventionReq` | `ConventionReq.scala` | What a parent requires from each child: `Any` or `Is(type)`, for rows and for batches | | ||
| | `Transition` | `Transition.scala`, `TransitionGraph.scala` | An edge of the graph: a function `SparkPlan => SparkPlan` that wraps the child with a converting operator, or `Transition.empty` when the data is already compatible | | ||
| | `ConventionFunc` | `ConventionFunc.scala` | Derives the convention of any plan, including vanilla Spark plans and plans through `QueryStageExec` / `ReusedExchangeExec` | |
Comment on lines
+129
to
+132
| This follows one fixed-width column, a `BIGINT` column `x`, through | ||
| `Velox op → LoadArrowDataExec → ColumnarArrowEvalPythonExec → OffloadArrowDataExec → | ||
| ArrowColumnarToVeloxColumnarExec → Velox op`, for a query such as `SELECT x, f(x)` with a pandas or | ||
| Arrow UDF `f`. . |
| - **Imported through C Data** (`Data.importIntoVectorSchemaRoot`): the buffer has a | ||
| `ForeignAllocationManager`, whose `release0()` calls `ReferenceCountedArrowArray.release()`. When | ||
| the last buffer of the imported array is released, it calls the array's C `release` callback, | ||
| here Velox's. It's the `release` member function of Arrow C_data interface: |
Copilot stopped reviewing on behalf of
FelixYBW due to an error
October 20, 2026 08:02
zhztheplayer
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Adds
docs/developers/ArrowColumnarConversion.md(and a mkdocs nav entry), which documents:RowType/BatchType,ConventionReq,ConventionFunc, the transition graph, and howInsertTransitionsinserts transitions.VanillaRowType,VanillaBatchType,ArrowJavaBatchType(heavy batch),ArrowNativeBatchType(light batch),VeloxBatchType,VeloxCarrierRowType, with the transition graph and the composite paths the planner builds.LoadArrowDataExec,OffloadArrowDataExec,ArrowColumnarToVeloxColumnarExecand the row transitions, down to the JNI / Velox calls, where data is copied, and the in-place, reference-counted batch lifecycle.BIGINTcolumn from Velox throughLoadArrowDataExec,ColumnarArrowEvalPythonExecand back to Velox, with the Velox buffer references and Gluten'srefCntat each step, and the release callbacks that return the memory to Velox (PlantUML sequence diagram). Also how anArrowBufis released depending on whether it was allocated in Java or imported through C Data, and the memory accounting of imported buffers (second PlantUML diagram).Diagrams are PNGs under
docs/image/; the PlantUML sources are included in the doc.Based on #12289, #13140 (comment) and #13140 (comment).
How was this patch tested?
Docs only. Checked the described behavior against the code on
mainand the Arrow Java 15.0.0 sources, and that the relative links and images resolve.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5-5
🤖 Generated with Claude Code