Skip to content

[SPARK-60359][SS] Remove state only when eviction iterator advances - #59608

Open
minyyy wants to merge 1 commit into
apache:masterfrom
minyyy:stack/eviction-iterator-remove-on-next
Open

minyyy wants to merge 1 commit into
apache:masterfrom
minyyy:stack/eviction-iterator-remove-on-next

Conversation

@minyyy

@minyyy minyyy commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Make EvictionIterator remove an evictable row (and increment numRowsRemovedSoFar) in next() instead of hasNext. hasNext still finds and buffers the next evictable row, but no longer mutates the state store. Docs on EvictionIterator and StreamingAggregationStateManager.evictionIterator are updated, and the now-obsolete note in StatefulStreamlineAggregateExec is removed.

Why are the changes needed?

hasNext should not have side effects. With removal in hasNext, a caller that probes without consuming deletes a row it never received: in append mode that row is dropped from state without being emitted yet is still counted in numOutputRows, and update-mode incremental cleanup has to order its checks carefully to avoid removing an extra row per input.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New test in StreamingAggregationStateManagerSuite checking that repeated hasNext does not remove the row or update the metric, and next() does. Existing StreamingAggregationStateManagerSuite and StreamingAggregationSuite pass.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant