[joshuaswarren]

Head: 0fc1ab003886a1160a95683a6d378e8c4e6e690f. Target: integration/aurora.

The head differs from the reviewed and device-tested head 168693eac1d6a55e9da8abc863f533f161f4b00b by one commit that changes only src/asahi/vulkan/tests/test-hk-va.c. It makes the exhaustion test fill the ordinary heap and free what it allocates. No driver code changed, so the device results below stand at 168693eac1d.

This is the Aurora version of #9, not the main-based commit. It checks fixed replay addresses against the capture range and its guard page. A bad address, or one still in use, returns VK_ERROR_INVALID_OPAQUE_CAPTURE_ADDRESS.

The native-AGX3-only heap gate was already in Aurora base 147a62f619001f5f2588c7417c2b1fac3b0d93ad. The cleanup did not add it. Commit 168693eac1d extends the scope: it enables the existing heap on G13/G14 too. Firmware reservations, AGX3-only alias heaps and cleanup are unchanged.

GitHub says this head is MERGEABLE, with no text conflicts with integration/aurora. An independent reviewer posted GREEN at 168693eac1d after reading both commits and testing agx_va_capture_valid against exact arithmetic and the real util/vma.c over 6,000,000 cases, with no mismatch.

Checks at this head

• CI: x86 and arm64 builds and ci / result pass. The attested candidate job was skipped.
• G15G, 13-inch M3 Air: meson test --suite asahi 57/57. CTS 1.4.6.2 fixed list: 4,058 cases, 1,648 Pass and 2,410 NotSupported. Capture/replay plus BDA list: 5,320 cases, 3,376 Pass and 1,944 NotSupported. Both match baseline 97a65d8. No Fail, Crash, Timeout or GPU fault.
• G15G API checks: capture, destroy and recreate keeps the address, and each bad-address case returns INVALID_OPAQUE_CAPTURE_ADDRESS where the baseline returned OUT_OF_DEVICE_MEMORY.
• G14X, 14-inch M2 Max: base 147a62f619 against this head. The same API checks, now returning INVALID_OPAQUE_CAPTURE_ADDRESS. The fixed list and the 5,320-case list match the base case for case. The validation layer was loaded and reported nothing.
• G13G, 13-inch M1, API checks: six of six cases pass on this head. Four return the new error where the system driver returned OUT_OF_DEVICE_MEMORY.
• G13G, 13-inch M1, CTS subset: 362 selected cases, one result per case on both drivers. This head: 240 Pass, 122 NotSupported. System driver: 118 Pass, 122 ResourceError, 122 NotSupported. All 122 changes are ResourceError to Pass. No Fail or Crash on either side.

These are targeted checks, not full CTS.

Limits

• The G13G comparison is against the system driver 69bde15340, not the branch base. It shows what the head does on a 13-inch M1. It does not isolate this PR from other differences between those two drivers.
• Any failed fixed-address allocation returns INVALID_OPAQUE_CAPTURE_ADDRESS, including a host allocation failure.
• Virtio goes through agx_open_device, so it now gets the heap too. No virtio run exists.
• The CTS executable used for the G13G subset has an unverified source commit.

No main-based #9 result qualifies this PR, and no landing-order change is requested.

[scottjones]

Thanks Josh, this is a nice cleanup of the #9 heap. We ran #18 at fba14a0 on G15G, the capture/replay and error-return check you said it needed.

Setup: 13" M3 Air (J613, G15G), kernel 7.2.11 (integration/aurora with the T8122 runtime and #236), m1n1 with the #14 dcpext gate. Vulkan-only build, buildtype plain + optimization 2 + b_ndebug=true like the mesa-aurora package, loaded from a private prefix via VK_ICD_FILENAMES (LD_DEBUG confirmed the private libvulkan_asahi.so). Baseline is the system vulkan-aurora built from 97a65d8.

meson test --suite asahi: 57/57 OK, including your new hk_va test.

VK-GL-CTS 1.4.6.2:

List	97a65d8	#18	Differ
Fixed regression list (4,058)	1,648 Pass / 2,410 NS	same	0
All capture_replay + BDA (5,320)	3,376 Pass / 1,944 NS	same	0

No Fail, Crash or Timeout, and no GPU fault in the kernel log. capture_replay is 10 Pass (capture_replay_stress) and 600 NotSupported. Those 600 need descriptor_buffer, ray tracing, descriptor_heap or NV device_generated_commands, not BDA capture/replay, which is advertised and works. #18 doesn't change feature bits.

CTS has no case that expects VK_ERROR_INVALID_OPAQUE_CAPTURE_ADDRESS, so the new error returns are only covered by test-hk-va, which passes here.

One question from reading the code (not a run): the replay heap is only set up when agx_chip_native_agx3() is true (agx_device.c:1288, agx_chip.h:173), so on G13/G14 hk_CreateBuffer never sets AGX_VA_CAPTURE_REPLAY (hk_buffer.c:137), while bufferDeviceAddressCaptureReplay is advertised on every chip (hk_physical_device.c:445). That looks like the M1 replay failures #9 fixed would come back on integration/aurora. Is that intended, or should the heap cover G13/G14 too?

Small nit: the comment above that check (agx_device.c:1284-1286) still says "only for G17P", but the condition covers every native AGX3 chip.

From the M3 side we see no regression and the G15G path checked out. The open question is M1/M2.

[scottjones]

One follow-up after a closer read: the "is in use" branch in hk_buffer.c isn't behind the replay-heap check, so on M1/M2 a failed fixed-address allocation now returns VK_ERROR_INVALID_OPAQUE_CAPTURE_ADDRESS instead of VK_ERROR_OUT_OF_DEVICE_MEMORY. That looks like the code the spec wants, but nobody has run it on G13/G14. With that noted, and the G13/G14 replay heap as a separate follow-up, we'd be happy to see this land from the M3 side.

[scottjones]

Just saw 168693e extends the replay heap to G13/G14, thanks, that answers our question. We'll rerun the M3 Air CTS at the new head and post the result here.

[scottjones]

Rerun at 168693e on the same M3 Air (G15G), same build options, private prefix: no change.

• meson test --suite asahi: 57/57 OK, including hk_va.
• CTS 1.4.6.2 fixed list (4,058): 1,648 Pass / 2,410 NotSupported.
• All capture_replay + BDA (5,320): 3,376 Pass / 1,944 NotSupported.
• 0 cases differ against 97a65d8 and 0 against fba14a0 on both lists. No Fail, Crash or Timeout, and no GPU fault in the kernel log.

That matches the code: G15G was already inside the agx_chip_native_agx3 set, so dropping the gate only adds the reservation on G13/G14. Those paths (and virtio, if it reaches this code; the old comment said other winsyses own their VA) need their own run, which this one can't cover.

[scottjones]

We also covered the error returns that CTS doesn't test. A small standalone Vulkan test (bufferDeviceAddress + bufferDeviceAddressCaptureReplay, one process per case) on the same M3 Air (G15G), against 168693e and, for contrast, the system 97a65d8:

Case	168693e	97a65d8
A: capture, destroy, recreate at the captured address	VK_SUCCESS, same address, GPU fill + readback OK	same
B1: recreate at 0x1000	INVALID_OPAQUE_CAPTURE_ADDRESS	OUT_OF_DEVICE_MEMORY
B2: recreate at an ordinary buffer's address	INVALID_OPAQUE_CAPTURE_ADDRESS	OUT_OF_DEVICE_MEMORY
B3: captured address + 0x100 (in range, misaligned)	INVALID_OPAQUE_CAPTURE_ADDRESS	OUT_OF_DEVICE_MEMORY
C: captured address still in use	INVALID_OPAQUE_CAPTURE_ADDRESS	OUT_OF_DEVICE_MEMORY
D: memory allocated with opaqueCaptureAddress 0x1000	VK_SUCCESS (echoes 0x1000)	same

D is the memory side, which #18 doesn't touch; the spec allows SUCCESS there. No crashes and no GPU fault in the kernel log. The validation layer wasn't installed on this machine, so these ran without it. Happy to share the test source.

G13/G14 still need their own run. We have a G14X (M2 Max) and plan to run the same test plus capture_replay_stress there.

[joshuaswarren]

Could you share the source for the six standalone Vulkan address/error-return cases, plus the capture_replay and BDA case list or the command used to make it? I need the same inputs for the G13G comparison at 168693e against system Mesa. The G15G results will keep their stated scope.

[scottjones]

G14 results. We ran the same checks on a 14" M2 Max (J414c, G14X), comparing #18's base 147a62f against head 168693e, both built the same way into private prefixes, with the system Asahi 26.2.5 as a reference.

Check	base 147a62f	head 168693e
meson test --suite asahi	56/56	57/57 (+ hk_va)
A: capture, destroy, replay at the captured address	SUCCESS, same address	SUCCESS, same address
B1-B3, C: outside range / ordinary address / misaligned / in use	OUT_OF_DEVICE_MEMORY	INVALID_OPAQUE_CAPTURE_ADDRESS
B2 detail	an ordinary buffer was given 0x2ffff6c000, the address a capture had just released	the ordinary buffer lands at 0x2dfff6c000, outside the range
capture_replay_stress	10/10	10/10
All capture_replay + BDA (5,320)	2,256 Pass / 3,064 NS	same, 0 differ
Fixed list (4,058)	1,628 Pass / 2,430 NS	same, 0 differ

VK_LAYER_KHRONOS_validation 1.4.363 was loaded in every return-code run (base, head and system, with and without it) and reported nothing. No Fail, Crash or Timeout, and no GPU fault. The extra NotSupported on G14 are feature limits (vertex pipeline stores and atomics, descriptor set number), not #18.

So on G14 the head fixes the replay collision the new commit targets, and on G15G (earlier comments) it changes nothing. That leaves G13. We'll share the test source and the two case lists separately for your G13G run.

[scottjones]

Here are the inputs for the G13G comparison: https://gist.github.com/scottjones/c93ae68c8a817123310d32758d31d341

• capture-addr-test.c: the standalone return-code test. Build with gcc -O2 -o capture-addr-test capture-addr-test.c -lvulkan and run one case per process: ./capture-addr-test A (then B1, B2, B3, C, D). Point it at a driver with VK_ICD_FILENAMES=<prefix>/share/vulkan/icd.d/asahi_icd.aarch64.json; we ran each case with and without VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation.
• caselist-fixed-4058.txt and caselist-capture-replay-bda-5320.txt: the exact dEQP-VK case lists, for VK-GL-CTS vulkan-cts-1.4.6.2 (deqp-vk only).
• run-deqp-list.sh: our runner. It expects the list as caselist.txt next to it and deqp-vk under ~/cts/build, runs 100-case chunks under env -i, takes an optional ICD json as the second argument, and stops on a GPU fault line or high temperature. Usage: run-deqp-list.sh LABEL [ICD_JSON].

We compared runs case for case on the per-case results, not just the totals.

[joshuaswarren]

Thanks, Scott. The G14X run at 168693e closes that targeted device gap: replay works, ordinary buffers stay outside the replay range, and the invalid-address cases return the right error. We will keep this head unchanged and use your test source and case lists for the G13G comparison. Our G13G build passes all 57 Asahi tests, but its device checks are still open. The PR stays draft until that gap is closed.

[joshuaswarren]

Tested on a 13-inch M1, G13G B1. Used Scott's API test. The new driver is 168693ea. The system driver is 69bde15340.

Case	System	New
A: save, free, replay	PASS: same address; GPU fill and readback OK	PASS: same address; GPU fill and readback OK
B1: below range	FAIL: OOM	PASS: invalid address
B2: plain buffer address	FAIL: OOM	PASS: invalid address
B3: bad alignment	FAIL: OOM	PASS: invalid address
C: address in use	FAIL: OOM	PASS: invalid address
D: memory address 0x1000	PASS: SUCCESS; echo 0x0	PASS: SUCCESS; echo 0x1000

OOM means VK_ERROR_OUT_OF_DEVICE_MEMORY. Invalid address means VK_ERROR_INVALID_OPAQUE_CAPTURE_ADDRESS.

The new driver passed all six cases. D permits success or an invalid-address error. It does not check that the echo matches.

Each row used the right ICD manifest. The log names it. The result file also holds the ICD and driver hashes.

This is Scott's standalone API test, not CTS. The validation layer is not installed. It was off for both drivers.

Result JSON SHA256: 7ca5cec13823320698b2c4b44cbe9c5715b5c4f67830543bf3c8822aac34f890.

[joshuaswarren]

Ran the selected G13 CTS tests on a 13-inch M1. The new driver was 168693ea; the system driver was 69bde15340.

The list has 362 cases. It covers buffer capture/replay, ten stress seeds, matched ordinary (non-replay) controls, feature checks, and buffer descriptor/heap cases.

• New driver: all 362 cases ran once; 240 Pass, 122 NotSupported, no bad results.
• Baseline: incomplete. Only 250 of 362 cases have a real result. Of these, 38 returned ResourceError: VK_ERROR_OUT_OF_DEVICE_MEMORY in vk.createBuffer. All 38 pass on the new driver.
• The other 112 cases have no baseline result. The runner assumed list order was run order. Its resume code skipped cases and repeated others after a ResourceError. These gaps are not crashes.

There was no regression among the 118 cases with a single clean baseline Pass. Another 93 had a single clean NotSupported result. This does not show that no case regressed across all 362. The baseline was not rerun.

This is a CTS subset, not full CTS. NotSupported cases are not passes.

CTS ELF SHA256: 550d04d3bc2b2164c3a74ae5f386f287a97bc2e3c2a311756838c994e9b50744. Its source commit is unknown; the build's source binding was not verified.

[joshuaswarren]

Ran the same 362 selected CTS cases on one 13-inch M1 (G13G). The PR driver was built from 168693eac1d6a55e9da8abc863f533f161f4b00b. The system driver was 69bde15340.

Result	System driver	PR driver
Pass	118	240
ResourceError	122	0
NotSupported	122	122

Each driver has one result for each case. All 122 changes were ResourceError to Pass. Neither driver had a Fail or Crash result. The comparison found zero regressions in these 362 cases. The 122 NotSupported cases stayed NotSupported on both drivers.

This replaces the incomplete baseline in my earlier comment. Both arms are now complete. This is a CTS subset, not full CTS. It covers selected buffer-device-address and capture/replay cases, with matched ordinary (non-replay) controls. The run took 22 seconds: October 27, 05:43:11 to 05:43:33 UTC.

SHA256 hashes:

• Case list: 4506bfadc3cc51f33118b65c319fce96d66fa4b9fe17839a2a11f272553c8b04
• PR driver: b2633138e63102b8b1ddbfa1df2c68cd9dc35bfd10c65d7a3862cd1a17a89d46
• System driver: 40282811c6e14ef991f72936bb675008cb4d6b3199b1785bc34c7976ab89c425
• CTS executable: 550d04d3bc2b2164c3a74ae5f386f287a97bc2e3c2a311756838c994e9b50744
• Comparison JSON: 5e1894063030ec57c3f8d8cbcde95b4162b049fea7f36e48c31d4623cc55b13e

The CTS executable's source commit is not independently verified.

[joshuaswarren]

GREEN at 168693e

Independent read of both commits (8 files, +208 -13) and of the evidence comments. I did not run any GPU work; I checked the code and the arithmetic.

Code, read at this head:

• agx_va_capture_valid is correct. It rejects an uninitialized heap, a size that would wrap (size_B > replay_size or guard_size > replay_size - size_B), an address outside [replay_start, end), a zero or misaligned align_B, and a range whose end plus guard passes the end of the capture range. The final subtraction cannot underflow because the earlier test already bounded size_B + guard_size.
• It agrees with the allocator. agx_va_alloc adds the guard to size_B and then calls util_vma_heap_alloc_addr(heap, fixed_va, size_B), so the validity test and the allocation use the same total size. hk_CreateBuffer passes the same vma_size_B and alignment to the check and to the allocation.
• I tested the function text (copied unchanged from this head) against exact 128 bit arithmetic and against the real util/vma.c from the base tree: 6,000,000 random and boundary cases (sizes and addresses at the range ends, wrapping values near UINT64_MAX, alignments of 0, 4 KiB, 16 KiB, 64 KiB, 1 MiB, 1 GiB and a non power of two). 0 mismatches; 520,402 of them were valid, and a sample of the valid ones was accepted by the real heap with assertions on; the last valid placement is accepted and one page later is refused.
• The only caller of the capture/replay flags is hk_CreateBuffer. Every other agx_va_alloc caller passes neither AGX_VA_CAPTURE_REPLAY nor AGX_VA_FIXED, so the new early return cannot reach them. Virtio devices go through the same agx_open_device (dev->is_virtio is set inside it), so the unconditional call now reserves the capture range there too. The guest allocates from the same main heap (agx_device_virtio.c calls agx_va_alloc), so I see no conflict by reading, but no virtio run is reported (the PR body says so as well).
• Device close and the failure path both free the replay heap only when it was initialized, and the range fields are written once at open, so the reads need no lock.

Evidence, checked for consistency:

• CI at this head: plan, source, both builds and ci / result pass.
• 13-inch M3 Air: 57/57 tests in the meson suite including hk_va; both CTS lists match the baseline case for case; the six API cases match what the code does (A succeeds, B1 to B3 and C return VK_ERROR_INVALID_OPAQUE_CAPTURE_ADDRESS, D is the memory side and is untouched).
• 14-inch M2 Max: base against head, same build, validation layer loaded: B1 to B3 and C change from OUT_OF_DEVICE_MEMORY to INVALID_OPAQUE_CAPTURE_ADDRESS, 0 CTS cases differ, and the ordinary buffer no longer lands in the range that a capture released.
• 13-inch M1: the six API cases pass on the PR driver; the 362 case CTS subset reads 118 Pass, 122 ResourceError, 122 NotSupported on the system driver and 240 Pass, 0 ResourceError, 122 NotSupported on the PR driver, which is consistent (118 + 122 = 240; every change is ResourceError to Pass; 0 Fail or Crash on either side).

Notes, none blocking:

1. The PR body is out of date. It still reads "NOT READY TO MERGE" and says the G13/G14 capture/replay and error-return device checks "have not run at this head". The comments show the G14X run and the G13G run at this head. Update the body before it leaves draft.
2. The 13-inch M1 comparison is against the system driver, not against the base of this branch (the M2 Max comparison is base against head). The 122 ResourceError to Pass changes therefore show the head fixes the system driver's behavior on that chip; they do not isolate this PR's two commits from other differences between the two drivers. The CTS binary's source commit is stated as unverified in the comment.
3. In hk_CreateBuffer, any failed allocation of a fixed address returns VK_ERROR_INVALID_OPAQUE_CAPTURE_ADDRESS ("is in use"). That includes the host memory failure of the malloc inside agx_va_alloc, which should be VK_ERROR_OUT_OF_HOST_MEMORY. It is rare and not worse than before.
4. The commit that enables the heap on the older chips makes agx_capture_replay_heap_init failure fatal for opening the device on every chip. It fails when the user VA window is under 8 GiB (size rounds to 0). That window is far larger on the chips tested, so I see no problem, but an optional feature now decides whether the device opens.
5. After this change replay_heap_initialized is true on every device that opens, so the replay_heap_initialized tests in hk_CreateBuffer and in agx_va_capture_valid can no longer be false in practice. They are harmless guards.

Not checked: any hardware run, a virtio run, the full build of the driver, and the meson test on my side (I ran the range function and the real heap in a stand alone program, not the PR's own hk_va binary).

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; both previous test findings are fixed, and no new actionable issues were found.

Summary

The PR checks fixed replay addresses against the reserved capture range, including guard space, and enables the existing replay heap on older chips.

• Every device opened through the shared path reserves space for replay addresses.
• Buffer creation rejects fixed replay addresses that cannot be used.

_{Reviews (2) · Last reviewed commit: "asahi/tests: fill the ordinary VA heap a..." · Reviewed by Greptile}

[joshuaswarren]

GREEN at 0fc1ab0

Read the one new commit over my earlier GREEN head 168693eac1d6a55e9da8abc863f533f161f4b00b: src/asahi/vulkan/tests/test-hk-va.c only (+18 -3). agx_va.c and agx_va_runtime_hole.c are byte for byte the same at the two heads, so the driver code and the device results at 168693ea stand.

Checked by building the unmodified test file from this head with the real agx_va.c, the real agx_capture_replay_heap_init and the real util/vma.c (stub headers for the device struct and the mutex only):

• The test passes. The loop makes 239 successful 4 GiB allocations. The arithmetic: the user window is 1 TiB, the replay range is MIN2(1024 / 8, 64) = 64 GiB (the test checks this), so the ordinary heap holds 960 GiB; each allocation takes 4 GiB plus the 16 KiB guard, so 960 GiB fits 239 of them and not 240. The floor of 200 and the cap of 512 both leave room: 239 is above one and well below the other.
• About 4 GiB minus 3.7 MiB stays free after the loop, so the heap is nearly dry rather than empty: a 4 GiB request fails, and the small request can still succeed in the ordinary heap. The test allows both ("fails or lands outside the range"), so the comment "dry heap" is a little generous but the checks are right.
• The frees are balanced. Every success is stored in filled[] and freed before close_va; small is freed (agx_va_free(NULL) returns early); r and the earlier a are freed; big is expected to be NULL. A build with AddressSanitizer and LeakSanitizer reports nothing.
• Mutation, which you said was not run: I changed agx_va_alloc so that when the ordinary heap fails it takes the address from the replay heap. The new test fails (15 failed checks, exit 1: !inside(dev, va) on the first spilled allocation, then the n_filled < MAX_FILL bound). The previous test, run against the same mutant, still passes (exit 0), so the new loop does what the commit says: it can catch a spill that the old one could not.

CI at this head: the x86_64 and aarch64 builds pass, hk_va passes on both, and ci / result passes.

Not checked: any hardware run (not needed for a test only change), and the full driver build; I built the test in a stand alone program.

[joshuaswarren]

A question for the maintainers before this PR goes further. integration/aurora now gates the capture/replay heap, and this PR removes that gate.

• 7416b6afc21 added the heap for G17P only. Its comment says other winsyses own their VA. The commit message gives no other reason.
• e02d0e5a174 added M5 to the gate. Its reason: ordinary allocations must not take addresses a capture will ask for at replay. It was validated on M5.
• The gate on integration/aurora (672657e0) is now native AGX3 or M5. This PR's second commit makes the init unconditional so G13 and G14 get the heap too. The two changes meet in one hunk of agx_device.c, so the PR now conflicts with the new base.

What the unconditional heap did in our runs, at the old base: on a 13-inch M1, 122 of 362 selected CTS cases went from ResourceError to Pass, with no Fail or Crash on either driver. On a 14-inch M2 Max the two CTS lists matched the base case for case and the new error codes were returned. On a 13-inch M3 Air nothing changed.

Which do you want?

1. Keep the gate. We drop the second commit. The address check from the first commit stays for the chips that have the heap.
2. Make the init unconditional, as in this PR. We rebase onto the new base, then rerun CI, review and the device checks on the new head.

We will rebase after you answer. Not checked: virtio, and whether any other winsys reaches this code.