Repository navigation
[docs] Design region-aware CPU/GPU placement under a memory budget - #33372
Open
Chi Lo (chilo-ms) wants to merge 10 commits into
Open
Chi Lo (chilo-ms) wants to merge 10 commits into
Chi Lo (chilo-ms) wants to merge 10 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The documentation-only additions clearly separate implemented behavior from the proposal, with no unresolved findings.
0 open findings
What changed in this PR
Extends the annotated-partitioning guide with a proposed memory-budget-aware CPU/GPU placement design, without changing runtime behavior.
Changes:
- Adds block-instance annotation guidance for manual placement and future region discovery.
- Defines bounded placement search, whole-session memory evaluation, partitioner integration, and validation, alongside the companion workspace proposal.
| File | Description |
|---|---|
| docs/annotated_partitioning/PartitioningWithAnnotationsAndMemoryConstraints.md | Adds annotation guidance and a clearly labeled automatic-placement proposal while preserving existing documentation. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
6 tasks
Describe EP-selected execution recipes, provisional optimization and final placement commitment, and backward-compatible plugin EP planning semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Add conceptual capability metadata and planning query examples, including alternative-view compatibility, recipe graph effects, contextual bounds, and legacy/API compatibility caveats. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Clarify tentative EP ownership, logical candidate evaluation without normal graph mutation, optional planning-only lowering, and applying only the provisional winner before verification and final placement commitment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Define region discovery/formation, fusion-aware execution candidates, CPU/GPU assignment boundaries, and complete candidate plans. Reserve placement for EP/device assignment and clarify the examples and section headings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Replace graph grouping with subgraph identification and clarify the distinction from nested control-flow graphs and EP fusion/compilation subgraphs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Detail capability metadata semantics, legacy L1 adapters, resolved policy contracts, and selected-contract propagation through L2 and runtime. Distinguish proposed integration from existing behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Simplify the estimator conversion requirements and explain that an adapter is not a separate component or API. Keep non-overlap and whole-session budget safeguards in their relevant sections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Illustrate one GQA candidate with memory and latency contracts, allocation requirements, and whole-session selection. Include existing nodes_to_optimize and optimization_func in the capability sketch while marking planning metadata as proposed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
Ti-Tai Wang (titaiwangms)
added a commit
that referenced
this pull request
Oct 29, 2026
### Description <!-- Describe your changes. --> Refines the extensible runtime fusion design around EP-registered, ordered patterns and the existing capability path: - EPs own pattern registration and priority; ORT selects compatible candidates in that order, subject to semantic, ownership, kernel, and eventual memory-feasibility checks. - Reuse `GetCapability`, `ComputeCapability`, `nodes_to_optimize`, and `optimization_func` through validated adapters. Distinguish existing application hooks from the additional recipe-description, ordering, capability-update, and external-registration support still needed. - Separate candidate description, selection, application, and reconciliation. Baseline ordered selection comes first; target memory estimation can connect to the same recipes before final budgeted selection, without requiring the general placement planner. - Replace the parallel post-partition hook and proposed `OrtRF*` function table with logical recipe/lifetime contracts. Preserve optional independent package delivery; defer exact public ABI additions until the capability integration is established. - Treat gramalingam#16 as an optional matcher implementation to evaluate with small Qwen-derived fixtures and the model demo, not a system-design prerequisite. - Update the qualification plan, approval ledger, and milestones consistently; retain CUDA mapping, state, numerical, and resource requirements. Design-only; no runtime implementation, build, GPU allocation, or model qualification is included. Public interfaces and implementation contracts remain subject to maintainer approval. Validation: `git diff --check` passed. Documentation fences and stale hook/API references were checked; no runtime tests were run. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Keep canonical ONNX artifacts unchanged while allowing optional matcher/adapter updates to reuse existing kernels. Align fusion with capability-based EP ownership and the recipe/planning direction in #33372 instead of introducing a separate selection pipeline. Qwen3.8-27B remains the model-independent mechanism's pilot, not its API. Fixes #33403. --------- Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: titaiwangms <18010845+titaiwangms@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 38c05fba-777a-447c-af24-f930f13e4021
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Extend the existing
docs/annotated_partitioning/PartitioningWithAnnotationsAndMemoryConstraints.mdguide with block-instance annotation guidance and a clearly labeled proposed, not implemented automatic CPU/GPU placement design.The proposal covers:
GetCapability()discovery separated from global budget acceptance, followed by transformation reconciliation, immutable commitment, and final assignment/L2 verification.All existing current-behavior documentation and configuration examples are preserved. This PR changes documentation only; it does not introduce a new session option or runtime behavior.
Motivation and Context
The current additive topological cutoff and manual layer assignment do not automatically discover structured offload boundaries or evaluate their whole-session physical GPU backing. This proposal records how a framework planner could generate a small candidate family and integrate selection with ORT partitioning before device materialization.
Companion dispatch-contract and lifetime/backing accounting design: #32944. That document has not merged, so this guide links to its PR rather than adding a broken relative link. The existing constrained-environment roadmap remains linked for historical context; this proposal is pre-commit planning, not late layer rollback or retry after OOM.
Validation
git diff --checkpassed.