Repository navigation
gluten-ut: the flatten function case caches three DataFrames and never unpersists them #12858
Description
What happens
The Gluten-authored flatten function case caches three DataFrames and never releases them. In gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDataFrameFunctionsSuite.scala the calls are at :72, :99 and :116:
// Test with cached relation, the Project will be evaluated with codegen
intDF.cache()There is no matching unpersist(), and no try/finally around the cached branch. The same case exists in all five version modules.
Why it matters
The cached blocks stay in the block manager for the rest of the suite. That is memory pressure the suite does not need, and it makes later cases in the same suite sensitive to execution order: whether a relation is already cached changes the plan they get. Order-dependent test failures are the hard kind to reproduce.
History
Not new. The case came from de26ed2da in gluten-ut/spark33 and was copied into 3.4/3.5/4.0/4.1 verbatim by #12840, where Copilot flagged it. Whitespace-stripped, the migrated copy is byte-identical to the 3.3 original.
Suggested fix
Call unpersist() after each cached branch's assertions, ideally in a finally so a failing assertion still releases the cache. Worth grepping the other Gluten*Suite files for the same shape while you are there.
Activity
shivadarshan-devadiga commentedon Nov 11, 2026
@LuciferYang I looked into this, and I don't think the cached relations outlive the test.
GlutenDataFrameFunctionsSuiteextends Spark'sDataFrameFunctionsSuite, which usesSharedSparkSession.SharedSparkSessionBase.afterEach()callsspark.sharedState.cacheManager.clearCache()after every test (Spark 3.4, 3.5, 4.0 and master). None of the Gluten traits mixed into this suite overrideafterEach, in any of the five modules. So the cache is cleared whenflatten functionfinishes, and later cases can't see it.I checked it at runtime on main (Velox, Spark 3.5) by temporarily overriding
afterEachinGlutenDataFrameFunctionsSuiteto print the cache state aroundsuper.afterEach(), then running onlyGluten - flatten function:The relations are cached while the test runs, which is what it tests (codegen over a cached relation), and gone after it. Spark's own
flatten functionandtransform keystests inDataFrameFunctionsSuitecache the same way withoutunpersist(), relying on the same cleanup. Addingunpersist()here would not change behaviour, so I think this issue can be closed.