[CORE] Normalize the case of spark.sql.mapKeyDedupPolicy and spark.io… by pedrumj2 · Pull Request #13613 · apache/gluten

pedrumj2 · 2026-11-10T21:53:53Z

What changes are proposed in this pull request?

Spark reads spark.sql.mapKeyDedupPolicy and spark.io.compression.codec in any case, but Gluten compared the policy and the codec name case-sensitively. This PR makes Gluten read both settings in any case.

Setting	Spark	Gluten before	Gluten after
`spark.sql.mapKeyDedupPolicy=exception`	recognized	not recognized	recognized
`spark.sql.mapKeyDedupPolicy=EXCEPTION`	recognized	recognized	recognized
`spark.io.compression.codec=zstd`	recognized	recognized	recognized
`spark.io.compression.codec=ZSTD`	recognized	not recognized	recognized

How was this patch tested?

Reproduction

Clone the repo and check out the base commit:

git clone https://github.com/apache/gluten.git
cd gluten
git checkout 31fb4dad58e7339ac4f4b820931d648160dc4746

Build the native libraries and the bundle jar:

docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c '
  source /opt/rh/gcc-toolset-12/enable
  ./dev/builddeps-veloxbe.sh --run_setup_script=OFF
  export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
  ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests
'

Fetch Spark 3.5.5:

curl -sSL https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | tar -xz

Write the input table and the queries:

cat > write_input.sql <<'EOF'
CREATE TABLE t USING parquet AS SELECT 'a:1,b:2,a:3' AS s;
EOF
cat > query.sql <<'EOF'
SET spark.sql.mapKeyDedupPolicy=exception;
EXPLAIN SELECT str_to_map(s, ',', ':') FROM t;
SELECT str_to_map(s, ',', ':') FROM t;
EOF
cat > agg.sql <<'EOF'
SET spark.sql.adaptive.enabled=false;
EXPLAIN SELECT s, count(*) FROM t GROUP BY s;
SELECT s, count(*) FROM t GROUP BY s;
EOF

Write the table with plain Spark, then run query.sql with Gluten:

docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c '
  export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
  rm -rf metastore_db spark-warehouse
  spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f write_input.sql 2>/dev/null
  BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
  spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
    --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
    --conf spark.plugins=org.apache.gluten.GlutenPlugin \
    --conf spark.memory.offHeap.enabled=true \
    --conf spark.memory.offHeap.size=4g \
    --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
    --conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
    --conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
    -f query.sql 2>err.log
  grep -m 1 "Duplicate" err.log
'

Run agg.sql with Gluten and spark.io.compression.codec=ZSTD:

docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c '
  export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
  BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
  spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
    --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
    --conf spark.plugins=org.apache.gluten.GlutenPlugin \
    --conf spark.memory.offHeap.enabled=true \
    --conf spark.memory.offHeap.size=4g \
    --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
    --conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
    --conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
    --conf spark.io.compression.codec=ZSTD \
    -f agg.sql 2>err.log
  grep -m 1 "compression kind" err.log
'

On the base commit, step 5 runs natively and keeps the last value of the duplicate key, and the grep finds nothing:

spark.sql.mapKeyDedupPolicy	exception
== Physical Plan ==
VeloxColumnarToRow
+- ^(1) ProjectExecTransformer [str_to_map(s#21, ,, :) AS str_to_map(s, ,, :)#22]
   +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: []


{"a":"3","b":"2"}

Step 6 runs natively and fails on the codec name, and the grep finds the reason:

spark.sql.adaptive.enabled	false
== Physical Plan ==
VeloxColumnarToRow
+- ^(2) HashAggregateTransformer(keys=[s#22], functions=[count(1)], isStreamingAgg=false)
   +- ^(2) InputIteratorTransformer[s#22, count#26L]
      +- ColumnarExchange hashpartitioning(s#22, 200), ENSURE_REQUIREMENTS, [s#22, count#26L], [plan_id=740], [shuffle_writer_type=hash]
         +- VeloxResizeBatches
            +- ^(1) ProjectExecTransformer [hash(s#22, 42) AS hash_partition_key#28, s#22, count#26L]
               +- ^(1) FlushableHashAggregateTransformer(keys=[s#22], functions=[partial_count(1)], isStreamingAgg=false)
                  +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#22] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: []


Reason: Not support compression kind ZSTD

On this pull request's branch, with the bundle jar rebuilt, step 5 fails natively on the duplicate key, and the grep finds the reason:

spark.sql.mapKeyDedupPolicy	exception
== Physical Plan ==
VeloxColumnarToRow
+- ^(1) ProjectExecTransformer [str_to_map(s#21, ,, :) AS str_to_map(s, ,, :)#22]
   +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: []


Reason: Duplicate keys are not allowed: 'a'.

Step 6 runs natively and returns the count, and the grep finds nothing:

spark.sql.adaptive.enabled	false
== Physical Plan ==
VeloxColumnarToRow
+- ^(2) HashAggregateTransformer(keys=[s#22], functions=[count(1)], isStreamingAgg=false)
   +- ^(2) InputIteratorTransformer[s#22, count#26L]
      +- ColumnarExchange hashpartitioning(s#22, 200), ENSURE_REQUIREMENTS, [s#22, count#26L], [plan_id=740], [shuffle_writer_type=hash]
         +- VeloxResizeBatches
            +- ^(1) ProjectExecTransformer [hash(s#22, 42) AS hash_partition_key#28, s#22, count#26L]
               +- ^(1) FlushableHashAggregateTransformer(keys=[s#22], functions=[partial_count(1)], isStreamingAgg=false)
                  +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#22] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: []


a:1,b:2,a:3	1

Spark without Gluten, which is the Gluten spark-sql command in each step without the --jars and --driver-class-path options and without every --conf option except spark.io.compression.codec, fails step 5 and returns the same count in step 6:

spark.sql.mapKeyDedupPolicy	exception
== Physical Plan ==
*(1) Project [str_to_map(s#15, ,, :) AS str_to_map(s, ,, :)#16]
+- *(1) ColumnarToRow
   +- FileScan parquet spark_catalog.default.t[s#15] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string>


org.apache.spark.SparkRuntimeException: [DUPLICATED_MAP_KEY] Duplicate map key a was found, please check the input data. If you want to remove the duplicated keys, you can set "spark.sql.mapKeyDedupPolicy" to "LAST_WIN" so that the key inserted at last takes precedence.

spark.sql.adaptive.enabled	false
== Physical Plan ==
*(2) HashAggregate(keys=[s#16], functions=[count(1)])
+- Exchange hashpartitioning(s#16, 200), ENSURE_REQUIREMENTS, [plan_id=29]
   +- *(1) HashAggregate(keys=[s#16], functions=[partial_count(1)])
      +- *(1) ColumnarToRow
         +- FileScan parquet spark_catalog.default.t[s#16] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string>


a:1,b:2,a:3	1

Automated tests

A new test in ScalarFunctionsValidateSuite, str_to_map with MAP_KEY_DEDUP_POLICY set in lower case, sets exception and checks that native str_to_map fails on a duplicate key. A new test in MiscOperatorSuite, native aggregation with spark.io.compression.codec set in upper case, sets ZSTD and checks that a native aggregation returns the same rows as Spark.

With this pull request's tests on the base commit's code, both tests fail:

./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false "-Dsuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite @str_to_map with MAP_KEY_DEDUP_POLICY set in lower case"

- str_to_map with MAP_KEY_DEDUP_POLICY set in lower case *** FAILED ***
  Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown (ScalarFunctionsValidateSuite.scala:2013)
…
Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***

./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false "-Dsuites=org.apache.gluten.execution.MiscOperatorSuite @native aggregation with spark.io.compression.codec set in upper case"

- native aggregation with spark.io.compression.codec set in upper case *** FAILED ***
…
Error Source: USER
Error Code: UNSUPPORTED
Reason: Not support compression kind ZSTD
Retriable: False
Function: stringToCompressionKind
…
Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***

With this change:

./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite,org.apache.gluten.execution.MiscOperatorSuite

ScalarFunctionsValidateSuite:
…
- str_to_map with MAP_KEY_DEDUP_POLICY set in lower case
…
MiscOperatorSuite:
…
- native aggregation with spark.io.compression.codec set in upper case
…
Tests: succeeded 191, failed 0, canceled 6, ignored 1, pending 0
All tests passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Co-authored with Claude Code (claude-opus-5-5)

….compression.codec for native code ## What changes are proposed in this pull request? Spark reads [spark.sql.mapKeyDedupPolicy](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L3785-L3794) and [spark.io.compression.codec](https://github.com/apache/spark/blob/v3.5.5/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L84) in any case, but Gluten [compared the policy](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/cpp/velox/compute/WholeStageResultIterator.cc#L692-L696) and [the codec name](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/common/compression/Compression.cpp#L90-L106) case-sensitively. This PR makes Gluten read both settings in any case. | Setting | Spark | Gluten before | Gluten after | |---|---|---|---| | `spark.sql.mapKeyDedupPolicy=exception` | recognized | not recognized | recognized | | `spark.sql.mapKeyDedupPolicy=EXCEPTION` | recognized | recognized | recognized | | `spark.io.compression.codec=zstd` | recognized | recognized | recognized | | `spark.io.compression.codec=ZSTD` | recognized | not recognized | recognized | ## How was this patch tested? ### Reproduction 1. Clone the repo and check out the base commit: ```bash git clone https://github.com/apache/gluten.git cd gluten git checkout 31fb4da ``` 2. Build the native libraries and the bundle jar: ```bash docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' source /opt/rh/gcc-toolset-12/enable ./dev/builddeps-veloxbe.sh --run_setup_script=OFF export JAVA_HOME=/usr/lib/jvm/java-17-openjdk ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests ' ``` 3. Fetch Spark 3.5.5: ```bash curl -sSL https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | tar -xz ``` 4. Write the input table and the queries: ```bash cat > write_input.sql <<'EOF' CREATE TABLE t USING parquet AS SELECT 'a:1,b:2,a:3' AS s; EOF cat > query.sql <<'EOF' SET spark.sql.mapKeyDedupPolicy=exception; EXPLAIN SELECT str_to_map(s, ',', ':') FROM t; SELECT str_to_map(s, ',', ':') FROM t; EOF cat > agg.sql <<'EOF' SET spark.sql.adaptive.enabled=false; EXPLAIN SELECT s, count(*) FROM t GROUP BY s; SELECT s, count(*) FROM t GROUP BY s; EOF ``` 5. Write the table with plain Spark, then run `query.sql` with Gluten: ```bash docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' export JAVA_HOME=/usr/lib/jvm/java-17-openjdk rm -rf metastore_db spark-warehouse spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f write_input.sql 2>/dev/null BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar) spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \ --jars "$BUNDLE" --driver-class-path "$BUNDLE" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \ --conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \ -f query.sql 2>err.log grep -m 1 "Duplicate" err.log ' ``` 6. Run `agg.sql` with Gluten and `spark.io.compression.codec=ZSTD`: ```bash docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' export JAVA_HOME=/usr/lib/jvm/java-17-openjdk BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar) spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \ --jars "$BUNDLE" --driver-class-path "$BUNDLE" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \ --conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \ --conf spark.io.compression.codec=ZSTD \ -f agg.sql 2>err.log grep -m 1 "compression kind" err.log ' ``` On the base commit, step 5 runs natively and keeps the last value of the duplicate key, and the grep finds nothing: ``` spark.sql.mapKeyDedupPolicy exception == Physical Plan == VeloxColumnarToRow +- ^(1) ProjectExecTransformer [str_to_map(s#21, ,, :) AS str_to_map(s, ,, :)#22] +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: [] {"a":"3","b":"2"} ``` Step 6 runs natively and fails on the codec name, and the grep finds the reason: ``` spark.sql.adaptive.enabled false == Physical Plan == VeloxColumnarToRow +- ^(2) HashAggregateTransformer(keys=[s#22], functions=[count(1)], isStreamingAgg=false) +- ^(2) InputIteratorTransformer[s#22, count#26L] +- ColumnarExchange hashpartitioning(s#22, 200), ENSURE_REQUIREMENTS, [s#22, count#26L], [plan_id=740], [shuffle_writer_type=hash] +- VeloxResizeBatches +- ^(1) ProjectExecTransformer [hash(s#22, 42) AS hash_partition_key#28, s#22, count#26L] +- ^(1) FlushableHashAggregateTransformer(keys=[s#22], functions=[partial_count(1)], isStreamingAgg=false) +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#22] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: [] Reason: Not support compression kind ZSTD ``` On this pull request's branch, with the bundle jar rebuilt, step 5 fails natively on the duplicate key, and the grep finds the reason: ``` spark.sql.mapKeyDedupPolicy exception == Physical Plan == VeloxColumnarToRow +- ^(1) ProjectExecTransformer [str_to_map(s#21, ,, :) AS str_to_map(s, ,, :)#22] +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: [] Reason: Duplicate keys are not allowed: 'a'. ``` Step 6 runs natively and returns the count, and the grep finds nothing: ``` spark.sql.adaptive.enabled false == Physical Plan == VeloxColumnarToRow +- ^(2) HashAggregateTransformer(keys=[s#22], functions=[count(1)], isStreamingAgg=false) +- ^(2) InputIteratorTransformer[s#22, count#26L] +- ColumnarExchange hashpartitioning(s#22, 200), ENSURE_REQUIREMENTS, [s#22, count#26L], [plan_id=740], [shuffle_writer_type=hash] +- VeloxResizeBatches +- ^(1) ProjectExecTransformer [hash(s#22, 42) AS hash_partition_key#28, s#22, count#26L] +- ^(1) FlushableHashAggregateTransformer(keys=[s#22], functions=[partial_count(1)], isStreamingAgg=false) +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#22] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: [] a:1,b:2,a:3 1 ``` Spark without Gluten, which is the Gluten `spark-sql` command in each step without the `--jars` and `--driver-class-path` options and without every `--conf` option except `spark.io.compression.codec`, fails step 5 and returns the same count in step 6: ``` spark.sql.mapKeyDedupPolicy exception == Physical Plan == *(1) Project [str_to_map(s#15, ,, :) AS str_to_map(s, ,, :)#16] +- *(1) ColumnarToRow +- FileScan parquet spark_catalog.default.t[s#15] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> org.apache.spark.SparkRuntimeException: [DUPLICATED_MAP_KEY] Duplicate map key a was found, please check the input data. If you want to remove the duplicated keys, you can set "spark.sql.mapKeyDedupPolicy" to "LAST_WIN" so that the key inserted at last takes precedence. ``` ``` spark.sql.adaptive.enabled false == Physical Plan == *(2) HashAggregate(keys=[s#16], functions=[count(1)]) +- Exchange hashpartitioning(s#16, 200), ENSURE_REQUIREMENTS, [plan_id=29] +- *(1) HashAggregate(keys=[s#16], functions=[partial_count(1)]) +- *(1) ColumnarToRow +- FileScan parquet spark_catalog.default.t[s#16] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> a:1,b:2,a:3 1 ``` ### Automated tests A new test in [ScalarFunctionsValidateSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala#L1943-L1998), `str_to_map with MAP_KEY_DEDUP_POLICY set in lower case`, sets `exception` and checks that native [str_to_map](https://spark.apache.org/docs/3.5.6/api/sql/index.html#str_to_map) fails on a duplicate key. A new test in [MiscOperatorSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala#L2667-L2692), `native aggregation with spark.io.compression.codec set in upper case`, sets `ZSTD` and checks that a native aggregation returns the same rows as Spark. With this pull request's tests on the base commit's code, both tests fail: ```bash ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false "-Dsuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite @str_to_map with MAP_KEY_DEDUP_POLICY set in lower case" ``` ``` - str_to_map with MAP_KEY_DEDUP_POLICY set in lower case *** FAILED *** Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown (ScalarFunctionsValidateSuite.scala:2013) … Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0 *** 1 TEST FAILED *** ``` ```bash ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false "-Dsuites=org.apache.gluten.execution.MiscOperatorSuite @Native aggregation with spark.io.compression.codec set in upper case" ``` ``` - native aggregation with spark.io.compression.codec set in upper case *** FAILED *** … Error Source: USER Error Code: UNSUPPORTED Reason: Not support compression kind ZSTD Retriable: False Function: stringToCompressionKind … Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0 *** 1 TEST FAILED *** ``` With this change: ```bash ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite,org.apache.gluten.execution.MiscOperatorSuite ``` ``` ScalarFunctionsValidateSuite: … - str_to_map with MAP_KEY_DEDUP_POLICY set in lower case … MiscOperatorSuite: … - native aggregation with spark.io.compression.codec set in upper case … Tests: succeeded 191, failed 0, canceled 6, ignored 1, pending 0 All tests passed. ``` ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Co-authored with Claude Code (claude-opus-5-5)

github-actions · 2026-11-10T22:01:55Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-11-11T20:40:55Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-11-11T21:45:56Z

Run Gluten Clickhouse CI on x86

github-actions Bot added CORE works for Gluten Core VELOX labels Nov 10, 2026

pedrumj2 marked this pull request as ready for review November 11, 2026 19:36

Merge branch 'main' into map-key-dedup-policy-upper-case

0695085

Merge branch 'main' into map-key-dedup-policy-upper-case

a6cf2d5

Navigation Menu

Repository navigation

[CORE] Normalize the case of spark.sql.mapKeyDedupPolicy and spark.io… - #13613

[CORE] Normalize the case of spark.sql.mapKeyDedupPolicy and spark.io…#13613
pedrumj2 wants to merge 3 commits into
apache:mainfrom
pedrumj2:map-key-dedup-policy-upper-case

pedrumj2 commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 11, 2026

Uh oh!

github-actions Bot commented Nov 11, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pedrumj2 commented Nov 10, 2026

What changes are proposed in this pull request?

How was this patch tested?

Reproduction

Automated tests

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 11, 2026

Uh oh!

github-actions Bot commented Nov 11, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant