[robobun]

Problem

• On Windows the scavenger is woken after its thread is gone. oven-sh/bun crashes that way at exit under Intel SDE in 4 of 7 runs, in ntdll below mi_scav_wake_one (WakeByAddressSingle, src/scavenger.c:127).
• ExitProcess terminates every other thread, then runs the FLS callbacks on the exiting thread. A free in one of them wakes the scavenger.
• ntdll keeps the wait block of a waiter on the stack of that waiter. The wake reads the stack of the dead scavenger, which SDE has unmapped.

Fix

• The Windows mi_scav_wake_one returns before WakeByAddressSingle when RtlDllShutdownInProgress() is true.
• ntdll sets that flag after the other threads are terminated and before the first FLS callback. All four wakes go through this function.
• Verified with the new test/test-exit-wake.c on Windows x64 and arm64. Without the guard its exit cases fail in 560 of 560 runs. With the guard 2200 of 2200 pass.

Background

• The scavenger waits in WaitOnAddress. A free that schedules a purge, a park and mi_scavenger_stop wake it.
• An FLS callback runs when a thread ends, and inside ExitProcess for the exiting thread. Rust runs thread-local destructors there.
• Considered a kernel event in place of WaitOnAddress: a wake that finds the scavenger awake costs two more syscalls (Notes). Considered a stop by the host before exit: bun has 9 exit sites.

Downsides

• Each issued wake on Windows executes 16 more instructions and no syscall (mingw-triple build, see Notes).
• The test patches the import address table of its executable. It skips if mimalloc no longer imports WaitOnAddress by name.

Notes

The cases of test-exit-wake. The test puts a counter in front of WaitOnAddress and WakeByAddressSingle. It waits until the scavenger is in its 30 s wait, which means that nothing is scheduled and the next free, park or stop has to wake it. An exit case returns 1 from main, and only the end of its FLS callback ends the process with 0. The callback first checks that the scavenger thread is terminated, that it was in its wait, and that RtlDllShutdownInProgress() is true.

ctest	What happens	Without the guard	With the guard
test-exit-wake	The FLS callback at exit frees an 8 MiB block	1 wake, fails	No wake
test-exit-wake-park	The callback parks (mi_on_thread_idle_start)	1 wake, fails	No wake
test-exit-wake-stop	The callback calls mi_scavenger_stop	1 wake, fails	No wake
test-exit-wake-gone	The callback releases the stack of the scavenger, then does all three	Access violation	Passes
test-exit-wake-live	No exit: a free, a free at the end of a thread, a stop	Passes	Passes

Results. Both runs are in my fork, with the matrix cut to its Windows jobs and one more step that runs each case 20 times. Runners: windows-2025-vs2026 (10.0.26100, x64) and windows-11-vs2026-arm64 (10.0.26200). Toolchains: MSVC, clang-cl, clang, mingw-ucrt64.

	free	park	stop	gone	live
Test only (60a787a5), run, 7 executables × 20 runs	0 of 140 pass	0 of 140	0 of 140	0 of 140 (access violation)	140 of 140
With the guard (26fc5bc4), run, 22 executables × 20 runs	440 of 440	440 of 440	440 of 440	440 of 440	440 of 440

• Without the guard the message of free, park and stop is FAILED (a wake of the scavenger after the exit terminated it) in all 420 runs.
• With the guard all 24 ctest runs of the suite pass as well (Release, Debug, Secure, SIMD, Guarded, Secure Full). The exit cases take 0.03 to 0.04 s each, the live case 0.30 s.

Two facts about Windows that the test showed.

• ExitProcess does not wait for the threads that it terminates. In a first version of the test the scavenger thread was not signaled yet when the FLS callback ran, in 26 of 28 runs. The callback now waits for it.
• The stack of a thread that the exit terminated stays mapped: every gone run released it with VirtualFree. This is why the stale read does not show natively.

ntdll. Read in 10.0.17763.6532, whose offsets match the frames of the crash on bun's CI machine.

• RtlDllShutdownInProgress reads one byte. LdrShutdownProcess stores 1 to it at +0xa7 and calls RtlProcessFlsData at +0xf3. RtlExitUserProcess calls LdrShutdownProcess after NtTerminateProcess(0).
• The fault is cmpq $0, 0x20(%r9) at ntdll+0x5e895. The wake reads the first wait block in the bucket of the address. The table has 128 buckets and is in the PEB.

Why the scavenger resolves the function itself. The lookup could also be in _mi_scavenger_start. In the build I measured, that one more call made clang stop inlining _mi_scavenger_start into _mi_scavenger_start_lazy, and mi_on_thread_idle_start and _mi_thread_init_with_heap changed with it. In mi_scavenger_thread_main it changes no other function. There is no wait block of the scavenger before its first wait, so the pointer is set in time.

Cost. System clang 23 for x86_64-w64-mingw32 and aarch64-w64-mingw32, C++, -O2, with bun's defines. This is not bun's clang-cl build.

• x64: 689 of 696 functions are instruction-identical. 6 differ only in the call target (WakeByAddressSingle became mi_scav_wake_one): mi_arena_schedule_purge, _mi_arenas_try_purge, _mi_scavenger_wake, _mi_scavenger_stop, mi_scavenger_stop, mi_process_done. mi_scavenger_thread_main has 8 more instructions, for 2 loader lookups per scavenger start. mi_scav_wake_one is a new function of 16 instructions.
• arm64: 688 of 695 identical, the same 6, 10 more in mi_scavenger_thread_main, 16 in the new function.
• Per issued wake on x64: 13 more instructions in mimalloc and 3 in ntdll, with 2 conditional branches, 1 indirect call and no syscall.
• All 35 functions with mi_free, mi_malloc, page_retire, mi_page_free, arenas_page_free or arenas_free in their name are identical.
• Object file: .text +93 B, .rdata +35 B, .bss +8 B on x64, and +104, +35, +8 on arm64. The 70 undefined symbols are the same before and after, so there is no new import.
• Linux x64: the object file is byte-identical.
• How often the wake runs, from counters in a Linux build (the logic is shared): test-stress has 89,407 arena page frees and 1,371 issued wakes, test-stress-heaps 120,750 and 1,314, test-heap-churn 353,734 and 4.

The kernel event alternative. An auto-reset event in place of the address wait needs no check at exit, because SetEvent reads no state of the waiter. It also keeps the scavenger out of the table of ntdll. It costs more while the process lives:

• WakeByAddressSingle enters the kernel only when a waiter is queued. SetEvent always does. A wake that finds the scavenger awake then costs one syscall on the thread that frees, and the next wait of the scavenger returns at once, so the scavenger makes one more turn of its loop.
• Counters in a Linux build, where the wake logic is the same: in test-stress 172 of 678 issued wakes found the scavenger awake, in test-purge-behind-pass 4 of 9, in test-heap-churn 4 of 5.
• Each process that starts the scavenger creates one more kernel object and keeps its handle.
• The guard adds 16 instructions to an issued wake and no syscall. I did not run an event version on Windows.

Not in this change.

• A scavenger that the exit terminates in the middle of a purge pass keeps the purge guard. Only mi_process_done releases it (707d90b), and a build with MI_NO_PROCESS_DETACH has no mi_process_done.
• Other waits for another thread have no exit check: the forced purge (arena.c:1702), mi_bitmap_clear_once_set (bitmap.c:122), the park leave (theap.c:336) and the heap walks (arena.c:2893, arena.c:2946, theap.c:826, theap.c:865). The crash traces of bun show none of them. A check in the Windows prim layer could serve them. This change keeps the check next to its one user, and resolves it on the scavenger thread, off the start of the process.
• The wait block of the dead scavenger stays in the table of ntdll. Other owners of wait-on-address waits (the host, Rust std) are not changed.

An unrelated failure to expect. In one run on ubuntu-22.04-arm (C++), test-purge-holes-large failed with FAILED: discard-first-class at test-purge-holes-large.c:1383. This change compiles to the same object on Linux.