Skip to content

fix: Disable jemalloc tcache on arm64 to stop TaskManager OOMKills - #426

Open
velo wants to merge 2 commits into
mainfrom
fix/jemalloc-arm64-tcache
Open

velo wants to merge 2 commits into
mainfrom
fix/jemalloc-arm64-tcache

Conversation

@velo

@velo velo commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

On arm64 the entrypoint now adds tcache:false to MALLOC_CONF before preloading jemalloc. jemalloc stays the allocator.

Why

Ubuntu's jemalloc 5.3.0 is built with 64 KiB pages on arm64, so each thread's cache holds far more memory than on amd64. A TaskManager running a 70-operator RocksDB job has about 2,700 threads (about 1,100 of them RocksDB checkpoint-transfer threads), and the caches add up to an OOMKill. The reproduction is in #423, and this branch is stacked on it, so JemallocArm64OomIT and the jemalloc-arm64-test job run here too.

Behaviour

  • arm64 only. amd64 doesn't show the problem, and tcache:false was never benchmarked there.
  • A user's MALLOC_CONF is kept. The option is appended, and nothing is added if they already set tcache: themselves.
  • Opt-out: set MALLOC_CONF=tcache:true, or DISABLE_JEMALLOC=true as before.

Benchmark evidence

Run on 2026-10-21 on an arm64 r6gd.xlarge (on-demand), with the TaskManager pinned to 4 CPUs and running a 70-operator RocksDB job.

Setup 3-min IT (3 runs) 2h at 3800 MiB Anon memory (MiB) Rows in ~1h57m Checkpoint avg
Baseline (6000 MiB in the 2h run) 3 of 3 OOMKilled survived (at 6000) 3800-4099 770,904 87s
rocksdb.checkpoint.transfer.thread.num: 1 3 of 3 pass OOMKilled at 53m 3370-3591 n/a n/a
MALLOC_CONF=tcache_max:1024 3 of 3 pass OOMKilled at 1h57m 3268-3496 670,362 92s
MALLOC_CONF=tcache:false 3 of 3 pass survived 3040-3262 723,887 121s

The 3-minute IT can't tell these options apart: the two that pass it still OOM in the 2-hour soak. Only tcache:false survives both.

Costs (one run each, so treat these as rough):

  • About 6% fewer rows than a baseline that had 2.2 GB more memory.
  • Checkpoints are 35–40% slower.
  • CPU usage is unchanged.

Anon memory still grew by about 220 MiB over the 2 hours.

jemalloc in the image accepts the option: its stats output shows opt.tcache: false, with no "Invalid conf" warning.

velo added 2 commits October 21, 2026 09:54
Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant