Skip to content

[Framework] Add opt-in fixed-placement session memory diagnostics - #33657

Draft
Chi Lo (chilo-ms) wants to merge 1 commit into
chilo/gqa-level2-envelopefrom
chilo/session-memory-backing-report
Draft

Chi Lo (chilo-ms) wants to merge 1 commit into
chilo/gqa-level2-envelopefrom
chilo/session-memory-backing-report

Conversation

@chilo-ms

Copy link
Copy Markdown
Contributor

Description

Add opt-in fixed-placement memory diagnostics with session.enable_memory_backing_report=1 (default off).

  • Capture initializer payloads before kernel construction and after local PrePack, before recursive subgraph finalization.
  • Model local-graph tensor and declared L2 workspace lifetimes using allocation-plan alias roots and release actions. Sequential workspaces do not overlap; parallel/multi-stream execution conservatively overlaps modeled roots.
  • Keep L1 prepack/initialization-scratch estimates separate from Run payloads. Reuse finalization's L2 declarations without querying kernels again.
  • Expose allocator-wide current observations separately from modeled payloads. Unsupported zero-filled statistics remain unavailable, not proven zero.
  • Log at initialization and after ordinary execution. Keep physical initialization/Run peaks explicitly unavailable and list uncovered terms.

This is the first diagnostic adapter, not complete session backing accounting or a no-OOM guarantee. It changes no allocation, placement, contract-selection, or budget policy and adds no device allocation or explicit synchronization. Control-flow/subgraph reports are not composed into a session peak.

Validation

Windows CPU Release, Ninja/MSVC 2022, using the existing isolated incremental build with the Windows memory leak checker disabled.

From the configured CPU Release build directory:

cmake --build . --target onnxruntime_test_all --parallel 10

Passed.

& '.\onnxruntime_test_all.exe' '--gtest_filter=SessionMemoryReportTest.*:SessionStateTest.*Workspace*:OpKernelTest.DefaultDeclareWorkspaceRequirementsClearsOutput:InferenceSessionTests.NoTimeout' --gtest_color=no

Passed: 22 tests, no skips. Eight new cases cover sequential/parallel scratch, input/initializer aliases, source release and packed destinations, dynamic shapes, known-zero versus unavailable workspace, reporting disabled, and a real report-enabled CPU Run with and without an arena. The runtime case verifies allocator-retained bytes are not added to the modeled payload peak and unsupported allocator statistics stay unavailable.

From the source worktree:

lintrunner -r origin/chilo/gqa-level2-envelope --take CLANGFORMAT --output oneline
git diff --check

Both passed. No new CUDA, training-enabled, minimal-build, or cross-stream runtime validation was performed for this slice.

Motivation and Context

Implements the first fixed-placement reporting slice from #32944, stacked on #33240's L1/L2 reconciliation. The stack is #33316 -> #33240 -> this PR.

The report deliberately separates conditional logical lifetime estimates from physical backing observations. Retained pools, alignment/fragmentation, packed ownership/sharing, constructor/preparation scratch, caller I/O/KV capacities, private library/EP allocations, capture, retained outputs, subgraph residency, and concurrency still prevent a complete physical peak.

The remaining sequence is to complete those backing adapters, introduce enforceable pre-allocation session budgets, then add generic contract selection and CPU-region placement. CUDA workspace/activation-pattern integration remains in #32071, WebGPU region integration in #32720, and the placement design in #33372; this PR does not duplicate them.

Capture local-graph initializer snapshots and alias-aware tensor/workspace lifetimes. Keep allocator-wide observations separate from modeled payloads and physical peaks unavailable until uncovered backing terms are resolved. Add opt-in logging, documentation, and focused CPU regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6d876416-81ec-4982-a9b0-38b8b28b5e4a

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant