[rorylshanks]

What changed

• Adaptive fan-in instead of block bytes. When a destination merge fails with MEMORY_LIMIT_EXCEEDED, the compact observer now halves max_parts_to_merge_at_once (32 → 16 → 8 → 4 → 2) via ALTER TABLE … MODIFY SETTING, with merges stopped and restarted around it. merge_max_block_size_bytes is no longer changed.

• One step per poll. Several memory failures that land in the same 5 s poll ran at the same fan-in, so they now cause a single reduction instead of several.

• Keep the result instead of failing. If merges still OOM at fan-in 2, the observer runs SYSTEM STOP MERGES and returns a sentinel. Compact then cancels only the merge wait and continues through the existing measure-and-upload path used for deadline, shutdown and manual finalize:

  • If some merges completed, the partially merged output is uploaded as a reduced result.
  • If nothing merged, the batch is released unchanged.

  Both cases log a compact_memory_exhausted warning.

• Cancellation race fixed. If the merge wait finished while the observer was running the stop or alter queries, the cancelled request was returned as a fatal error. The observer now treats that as a normal exit, matching how it handles cancellation elsewhere. This also affected the old block-bytes path.

• The --compact-max-parts-to-merge help text now says it is the starting value.

Why

Compaction batches were failing with destination merge exceeded memory limit with merge_max_block_size_bytes already at minimum 1048576. The traces show that memory goes to JSON (Object/Dynamic/Variant) substream deserialization across source parts, and to opening per-substream writer buffers. Rows per read were already small (1600–8192), so lowering block bytes had little effect. The batch hit the 1 MiB floor after about five OOMs and failed, discarding merges that had already succeeded.

Peak memory scales with source parts × substreams, so fan-in is the setting that controls it. Failing the batch is the wrong outcome when the inputs are intact and partial merges are usable.

Reviewer notes

• Write-side OOMs. Fan-in 2 doesn't bound memory on the write side: the output writer's buffers scale with distinct JSON paths. A partition with very heavy parts can stay fragmented, and later runs will retry it and hit the same OOM. The compact_memory_exhausted log identifies these cases.
• Unverified: SYSTEM STOP MERGES and the failure count. I believe SYSTEM STOP MERGES cancels any running merge and records it in system.part_log with error 236 (ABORTED). If so, each fan-in step could add one non-memory failure, and three of those still fail the batch. I haven't checked this against ClickHouse. If "destination merges failed 3 non-memory times" shows up after this change, the fix is to exclude error 236 in destinationFailedMergeSummary.

Testing

• go mod tidy (no changes)
• go test ./...
• go test -race ./internal/rewrite/
• The affected tests passed 20 times in a row.
• ./e2e/run.sh passed.

New and updated tests cover:

• The fan-in step-down, including a single step per poll.
• Stopping merges at the floor.
• Two Compact cases with OOM at the floor: one where output is reduced (it uploads) and one where nothing merged (inputs come back unchanged).

🤖 Generated with Claude Code