Skip to content

Fit near-full cards: shrink, drop dual-copy, fix quant/OOM units - #185

Merged
peti12352 merged 11 commits into
devfrom
fit-nvfp4-auto
Nov 19, 2026
Merged

peti12352 merged 11 commits into
devfrom
fit-nvfp4-auto

Conversation

@peti12352

@peti12352 peti12352 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Shrink ctx/conc to floor 2048 when the card is tight; drop Marlin dual-copy (GOETHE_MARLIN=0) only if that places better or is the only fit.
  • Fit math: weights+peak in free, total leaves 1.5 GB margin; mem_fraction may hit 1.0. Dead pool-cap branch removed.
  • Skip prefill CUDA graphs (decode only); drop registry --quantization when config.json already has quant_method.
  • Torch OOM learn: read MiB/GiB/GB units (same DB path; MiB no longer treated as GiB).

Test plan

  • Unit: shrink + drop-marlin placement, GPU-state matrix (roomy/tight/foreign/empty/partial/arch)
  • Unit: quant drop across common quant_methods; mem_fraction=1.0; OOM MiB→GB
  • CI: engine-test + lint (no GPU e2e this PR)

Marlin dual-copy was counted as steady VRAM and refused 27B NVFP4 on a
32 GB card that runs it after auto-shrink. Registry --quantization
modelopt_fp4 also aborted compressed-tensors checkpoints at startup.
Keep Marlin in the weight budget (measured ~30 GB resident) but raise
mem_fraction to 0.99 and skip the 1.5+1.8 padding when weights already
fill the card and KV has been shrunk away — matching SGLang's own floor.
Marlin's second weight copy left no KV/mamba room on full cards; plan
without it (GOETHE_MARLIN=0) when that yields a better or only placement,
and allow mem_fraction up to 1.0. Remove the near-full padding bypass that
falsely allowed oversubscribed loads.
Torch OOM lines report MiB; treating them as GiB inflated learned weights
into hundreds of GB. Prefill graph capture also OOMs after a near-full
static pool, so disable it automatically when mem_fraction is high.
@peti12352 peti12352 changed the title Fit near-full-card NVFP4 loads; stop forcing wrong quantization Fit near-full cards: shrink ctx/conc, drop optional weight copies Nov 19, 2026
Prefill capture lives outside the static pool and OOMs on tight cards — decode
graphs are enough. Also skip the bare-weight replan when the dual-copy plan
already meets the asked ctx/concurrency.
@peti12352 peti12352 changed the title Fit near-full cards: shrink ctx/conc, drop optional weight copies Fit near-full cards by shrinking ctx/conc and dropping optional weight copies Nov 19, 2026
Tighten calc shortfall asserts under mem_fraction=1.0 and exercise planner
shapes (roomy/tight/foreign/empty/partial) plus drop_quantization across
common quant_method values without needing a live GPU.
With mem_fraction capped at 1.0 the static-pool shortfall path never bound;
margin already covers it. Lint-clean the CPU fit tests.
@peti12352 peti12352 changed the title Fit near-full cards by shrinking ctx/conc and dropping optional weight copies Fit near-full cards: shrink, drop dual-copy, fix quant/OOM Nov 19, 2026
Fit stays scan+card only; leave engine_overhead learning untouched for a
separate decision.
@peti12352 peti12352 changed the title Fit near-full cards: shrink, drop dual-copy, fix quant/OOM Fit near-full cards: shrink, drop dual-copy, fix quant Nov 19, 2026
One helper converts the allocate/free figures to GB so MiB lines stop
inflating learned static VRAM; same DB learn path as before.
@peti12352 peti12352 changed the title Fit near-full cards: shrink, drop dual-copy, fix quant Fit near-full cards: shrink, drop dual-copy, fix quant/OOM units Nov 19, 2026
@peti12352
peti12352 merged commit 078a3b0 into dev Nov 19, 2026
11 checks passed
@peti12352
peti12352 deleted the fit-nvfp4-auto branch November 19, 2026 04:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant