Repository navigation
Conversation
Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On arm64 the entrypoint now adds
tcache:falsetoMALLOC_CONFbefore preloading jemalloc. jemalloc stays the allocator.Why
Ubuntu's jemalloc 5.3.0 is built with 64 KiB pages on arm64, so each thread's cache holds far more memory than on amd64. A TaskManager running a 70-operator RocksDB job has about 2,700 threads (about 1,100 of them RocksDB checkpoint-transfer threads), and the caches add up to an OOMKill. The reproduction is in #423, and this branch is stacked on it, so
JemallocArm64OomITand thejemalloc-arm64-testjob run here too.Behaviour
tcache:falsewas never benchmarked there.MALLOC_CONFis kept. The option is appended, and nothing is added if they already settcache:themselves.MALLOC_CONF=tcache:true, orDISABLE_JEMALLOC=trueas before.Benchmark evidence
Run on 2026-10-21 on an arm64 r6gd.xlarge (on-demand), with the TaskManager pinned to 4 CPUs and running a 70-operator RocksDB job.
rocksdb.checkpoint.transfer.thread.num: 1MALLOC_CONF=tcache_max:1024MALLOC_CONF=tcache:falseThe 3-minute IT can't tell these options apart: the two that pass it still OOM in the 2-hour soak. Only
tcache:falsesurvives both.Costs (one run each, so treat these as rough):
Anon memory still grew by about 220 MiB over the 2 hours.
jemalloc in the image accepts the option: its stats output shows
opt.tcache: false, with no "Invalid conf" warning.