Skip to content

test: Reproduce TaskManager OOMKill caused by jemalloc on arm64 - #423

Open
velo wants to merge 1 commit into
mainfrom
test/jemalloc-arm64-oom
Open

velo wants to merge 1 commit into
mainfrom
test/jemalloc-arm64-oom

Conversation

@velo

@velo velo commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

What

Adds an integration test, JemallocArm64OomIT, that reproduces arm64 TaskManagers being OOMKilled. It also adds a jemalloc-arm64-test CI job on ubuntu-24.04-arm that runs only this test.

This test is expected to fail on arm64 until the allocator issue is fixed. The job is not in the needs of maven-deploy, build-and-push-docker-image or build, so it doesn't block deploys.

Root cause

The flink:2.3.0-java17 base image ships Ubuntu's libjemalloc2 5.3.0. On arm64 that build uses 64 KiB pages (Page size: 65536), but Graviton and Bottlerocket kernels use 4 KiB pages. The page size is fixed when jemalloc is compiled, so MALLOC_CONF can't change it.

After the startup allocation burst, amd64 gives about 500 MiB back to the OS and arm64 doesn't.

How the test works

  • Cluster: one JobManager and one TaskManager, both from the image Maven builds.
    • The TaskManager is pinned to cpuset 0-3, so jemalloc creates 16 arenas, the same as a 4-vCPU runner.
    • The job is 70 RocksDB-backed GROUP BY and dedup operators, with 4 slots and process.size: 4000m.
  • Warm-up: the TaskManager runs for 3 minutes under a loose 6000 MiB container limit.
  • Shrink: the test then lowers the limit in place to 3800 MiB with docker update.
  • Pass condition: the TaskManager must stay alive for another 3 minutes.
  • Skipped unless the Docker daemon is aarch64.

Why warm up and then shrink, instead of using a fixed limit? Measured anon memory:

Startup peak Steady state
amd64 ~3975 MiB 3350–3550 MiB
arm64 — ~3900 MiB

amd64's startup peak is above arm64's steady state, so no fixed limit separates them. At a fixed 3000 MiB both architectures were OOMKilled.

Results

Environment Result
arm64 (r6gd.xlarge, 4 vCPU), 3 runs Failed every time: OOMKilled=true exitCode=137 about 3m07s after the job started, ~228 s per test
amd64, same test with the arch gate lifted Passed, 401 s

A long-running FlinkDeployment on staging showed the same pattern:

  • arm64 TaskManagers were OOMKilled repeatedly at ~3730–3850 MiB RSS.
  • amd64 stayed steady at ~3300 MiB for more than an hour.

The ubuntu-24.04-arm CI job hasn't run yet. This PR is its first run.

Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant