Harden GVFS.Mount against native ProjFS failures under memory pressure (0xC000CE01)

[tyrielv]

Problem

GVFS.Mount crashes are being reported that surface downstream as GetOfficialBranch.exe (and other tools) exiting with -1073689087 = 0xC000CE01 = STATUS_FILE_SYSTEM_VIRTUALIZATION_UNAVAILABLE. The chain is: GVFS.Mount crashes → the ProjFS virtualization root is orphaned → every subsequent placeholder access on that enlistment returns 0xC000CE01 until remount.

Watson attributes the crashes to radar_high_memory projectedfslib.dll!prjcompletecommand, alongside System.OutOfMemoryException buckets and GC access violations. The dominant managed signatures are NullReferenceException surfaced from the native ProjFS command-completion (PrjCompleteCommand, via StartDirectoryEnumerationAsyncHandler) and delete (PrjDeleteFile) paths — the native code faults under memory exhaustion and the CLR raises a (catchable) NullReferenceException on the worker thread, which today reaches ExecuteFileOrNetworkRequest and calls Environment.Exit.

VFS for Git is already on the latest Microsoft.Windows.ProjFS (2.1.0), so the native projectedfslib.dll fault cannot be fixed from here. The only managed-side levers are (A) reduce the memory pressure that triggers the native fault, and (B) stop converting a single failed native ProjFS call into a whole-mount crash. This PR does both. It does not claim to eliminate the native OOM fault itself.

Not a v2.0 regression: the crash-prone call sites and the activeEnumerations leak date to 2018 (785ccb4a, .NET Framework era). The v2.0 pure-C# ProjFS rewrite (66334474) made the same latent native fault legible as a managed NRE in GVFS telemetry — so the recent spike in reports is largely improved visibility, which B’s telemetry is designed to quantify.

A — Bound the activeEnumerations leak

ProjFS does not always deliver EndDirectoryEnumeration for a StartDirectoryEnumeration (the long-standing // TODO: Need ProjFS 13150199 note). The corresponding ActiveEnumeration — and the List<ProjectedFileInfo> it pins — then leaks for the life of the mount.

• ActiveEnumeration tracks a monotonic LastActivityTickCount (Environment.TickCount64), set on creation and every GetDirectoryEnumeration read.
• Once activeEnumerations grows past a threshold far larger than any realistic count of concurrent live enumerations (1024), a throttled sweep (≤ once/min) evicts entries idle longer than a timeout (5 min).
• Evicting a live-but-idle enumeration only fails that one directory listing with InternalError — never a crash — and every eviction is reported via telemetry.
• Monotonic clocks are used for both the throttle and the staleness check so NTP/wall-clock steps can’t disturb them.

B — Don’t crash the mount on a native ProjFS failure (generalized)

A single boundary helper, TryInvokeProjFS(Func<HResult>, operationName, …), wraps each GVFS-initiated native ProjFS call: it catches the exception, emits high-signal telemetry (*_NativeFailure, with activeEnumerations/activeCommands counts and GC.GetTotalMemory), and returns HResult.InternalError so the caller maps it to a failure result and keeps serving.

Routed through the swallow-and-continue helper: CompleteCommand (via TryCompleteCommand — covers the completion leg of all async callbacks: enumeration, placeholder-info, file-data), DeleteFile, WritePlaceholderFile, WritePlaceholderDirectory, UpdatePlaceholderIfNeeded, ClearNegativePathCache.

MarkDirectoryAsPlaceholder mutates projection state, so swallowing is unsafe. It uses a sibling helper InvokeProjFSOrThrow that emits the same *_NativeFailure telemetry (so the memory diagnostics are captured) and then rethrows — the exception still reaches LogUnhandledExceptionAndExit, preserving fail-fast. Both helpers share one LogProjFSNativeFailure.

Left as-is (already non-fatal): GetFileStreamHandlerAsyncHandler already fails a single hydration on any exception (completes the command with FileNotAvailable, no exit), so CreateWriteBuffer/WriteFileData are already non-fatal.

The exception is demonstrably catchable — it already reaches GVFS’s LogUnhandledExceptionAndExit today (the source of the existing VFS.Error "exiting process" events). B intercepts it one frame earlier and either declines to exit or (for state-mutating ops) enriches the crash with memory telemetry first.

Local repro attempt (empirical) — informs A and B

I built a B-only variant (guards + telemetry, no A eviction), installed it, cloned a large enlistment, and hammered it with cancelled directory enumerations (FindFirstFileExW + CancelSynchronousIo mid-flight) to try to reproduce an enumeration leak → OOM → native crash.

Controlled A/B, each touching 15k cold directories exactly once:

Run	Aborted enums	Mount ΔWorkingSet
Control (no cancel)	0	+10 MB
Test (cancel)	10,034	+7 MB

The cancel run grew less than the no-cancel control. A sustained 180 s / 869k-enumeration / 110k-abort run added only ~37 MB of slow creep dominated by legitimate cold→warm projection cache, stayed flat on idle, and produced zero *_NativeFailure / OOM / crash. Telemetry showed 883 async-path cancellations, all with activeEnumerationsUpdated:true — i.e. the existing async-cancel cleanup (StartDirectoryEnumerationAsyncHandler → TryCompleteCommand returns false → TryRemove) already removes every cancelled enumeration.

Takeaways:

• The simplest, most-suspected leak vector — enumeration cancellation — does not produce runaway growth in the current code; it self-heals. A remains in this PR as correct, cheap insurance against the documented EndDirectoryEnumeration-never-arrives case, but this repro shows it is not the driver of the observed OOM.
• The real-world OOM therefore comes from a different source (sync-path Start-without-End, abnormal process termination, or genuine kernel-memory exhaustion unrelated to this leak). That is exactly what B’s *_NativeFailure + activeEnumerations/memory telemetry is there to identify once rolled out.

Tests

• DeleteFileReturnsIOErrorWhenVirtualizationInstanceThrows, OutboundProjFSOperationsReturnFailureWhenVirtualizationInstanceThrows (ClearNegativePathCache, UpdatePlaceholderIfNeeded), WritePlaceholderOperationsReturnFailureWhenVirtualizationInstanceThrows (tester-based) — a throwing native call yields IOError, not a propagated exception.
• RecordActivityUpdatesLastActivityTime — covers the A monotonic-timestamp building block.
• (InvokeProjFSOrThrow’s rethrow path ends in Environment.Exit, so it isn’t safely unit-testable in-process; it’s symmetric with the tested TryInvokeProjFS.)
• Full unit suite: 881 passed, 0 failed (11 skipped); clean build.

Open questions for review

Two design questions remain genuinely open; I'd appreciate reviewer input on both.

1. Eviction heuristic (Part A). The eviction trigger is a pair of conservative
   constants: a size threshold (activeEnumerations.Count > 1024) and a
   staleness timeout (no activity for 5 min), swept at most once per minute. These
   are deliberately loose guesses chosen to sit far above any plausible count of
   genuinely-concurrent live enumerations, so that normal operation never evicts.
   Open points:

   • Should these be configurable (registry / GVFS config) rather than
     hard-coded, so they can be tuned per-fleet without a redeploy?
   • Should the trigger be driven by an actual memory signal (e.g. process
     working set or GC.GetTotalMemory crossing a bound) instead of a raw count,
     since the count is only a proxy for the memory we actually care about?
   • Is the failure mode acceptable? Eviction can, in the worst case, drop a
     genuinely-live-but-idle enumeration; that surfaces to the user as a single
     failed directory listing (InternalError), which the caller can retry — never
     a crash. I believe that tradeoff is fine given it only triggers well outside
     the normal operating envelope, but I'd like a second opinion.

2. Swallow-and-continue vs. controlled remount (Part B). When a native ProjFS
   call faults, B logs *_NativeFailure and fails just that one operation, keeping
   the mount alive. That is strictly better than today's behavior (crash → orphaned
   root → 0xC000CE01 for everything). But if the native virtualization context is
   genuinely dead — not merely a transient allocation failure — then every
   subsequent operation will also fault, and the mount is effectively limping while
   still reporting itself as up. In that case a deliberate, clean unmount +
   remount on repeated *_NativeFailure (e.g. N failures within a window) might
   be better recovery than indefinitely serving errors. Open points:

   • Is limping-but-alive the right default, or should repeated native failures
     escalate to a controlled remount / self-restart of the mount process?
   • If we do escalate, what signal distinguishes "transient" from "context is
     dead"? A simple failure-rate threshold, or something more specific?
   • I could not settle this locally: the native OOM fault was not reproducible
     via cancelled enumerations (see the repro section — cancellation is
     self-healed), so I have no local scenario that reaches the dead-context state
     to test remount-vs-limp against. This needs either a heavier memory-pressure
     harness or production data.

Summary / conclusion

The core value of this PR is turning an opaque, fatal crash into a survivable,
observable event. Today the native ProjFS fault kills GVFS.Mount and all we get
is a radar_high_memory Watson bucket with no GVFS-side context. After this
change the mount stays alive (Part B), the documented enumeration leak is bounded
as cheap insurance (Part A), and — most importantly — every native failure and
every eviction emits high-signal telemetry (activeEnumerations / activeCommands
counts and GC.GetTotalMemory at the exact failure point).

That telemetry is precisely what will let us answer both open questions above
with real data instead of guesses. The local repro already narrowed the search:
enumeration cancellation is self-healed and is not the OOM driver, so the real
memory source lies elsewhere (sync-path Start-without-End, abnormal process
termination, or genuine kernel-memory exhaustion). Once this is rolled out, the
*_NativeFailure events will show the actual activeEnumerations/memory state at
the moment of failure, which will tell us (1) whether A's eviction is ever even
triggered in the field and where to set its thresholds, and (2) how often the
native context is transiently vs. permanently dead — i.e. whether escalating to a
controlled remount is warranted. In short: land this to stop the bleeding and
start collecting the data needed to fix the root cause.

---

Assisted-by: Claude Opus 4.8