[Caio Rocha (caiocbr)]

No description provided.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

Tiny accepted scales can overflow the precomputed reciprocal and incorrectly dequantize finite nonzero FP8 values to infinity.

3 open findings

• Pass quantization scale to avoid reciprocal overflow · New
• Document required identical scale across ranks · New
• Require matching scale across ranks in Python contract · New

What changed in this PR

Adds latency RANK_MAJOR static-scale FP8 support and replay-safe CUDA graph synchronization.

Changes:

• Propagates static FP8 scales through C++, bindings, and Python APIs.
• Adds FP8 dispatch/combine kernels and graph-replay rendezvous.
• Expands validation, documentation, and tests.

File	Description
src/​ext/​ep/​latency.cc	Validates and propagates quantization scales.
src/​ext/​ep/​include/​moe_runtime_context.hpp	Retains scale in dispatch handles.
src/​ext/​ep/​include/​kernels.hpp	Adds scale to kernel workloads.
src/​ext/​ep/​dispatch/​rank_major_dispatch.cu	Supports FP8 payloads and replay invalidation.
src/​ext/​ep/​dispatch/​common.cuh	Selects rank-major FP8 kernels.
src/​ext/​ep/​common/​quantization.cuh	Adds scalar FP8 dequantization.
src/​ext/​ep/​combine/​rank_local_reduce_combine.cu	Enables FP8 rank-local combine.
src/​ext/​ep/​combine/​direct_send_combine.cu	Enables FP8 direct-send combine.
src/​ext/​ep/​combine/​common.cuh	Implements FP8 reduction and replay synchronization.
src/​ext/​ep/​bindings.cpp	Exposes quantization scale to Python.
python/​test/​test_ep.py	Tests FP8 validation, correctness, and replay.
python/​mscclpp/​ep/​utils.py	Validates and resolves scalar scales.
python/​mscclpp/​ep/​types.py	Extends the public quantization model.
python/​mscclpp/​ep/​request_builder.py	Builds FP8 dispatch/combine requests.
python/​mscclpp/​ep/​README.md	Documents FP8 and graph replay contracts.
python/​mscclpp/​ep/​communicator.py	Updates combine API documentation.
include/​mscclpp/​ext/​ep/​types.hpp	Extends the public C++ request API.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

The benchmark reuses overwritten or stale expert results and rejects supported single-round rank-major graphs.

5 open findings

• Refresh results after every rank-major dispatch · New
• Allow single-iteration rank-major CUDA graph replay · New
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Previously reported dequantization overflow and benchmark buffer-reuse correctness issues remain unresolved.

5 open findings

• Refresh results after every rank-major dispatch
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Benchmark overflow handling can strand MPI peers, and throughput kernel profiling omits required synchronization.

6 open findings

• Make overflow rejection collective across ranks · New
• Refresh results after every rank-major dispatch
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

Previously missed (1)

In code that hasn't changed since last review

Include peer synchronization in dispatch profiler filtering

python/​mscclpp_benchmark/​ep/​ep_bench_mscclpp.py:17

Match synchronizeThroughputPeersKernel in the dispatch profiler filter. Reused preparation launches this kernel before every throughput dispatch (src/ext/ep/throughput.cc:229–232), but its name contains neither dispatch nor synchronizepeers. Kineto therefore omits the required peer synchronization from dispatch and total kernel timings, overstating performance. Include synchronizethroughputpeers in the matching substrings.

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Both rank-major FP8 combine modes can produce incorrect infinities for accepted small quantization scales.

7 open issues

• Rank-major combine overflows reciprocal for tiny FP8 scales · New
• Make overflow rejection collective across ranks
• Refresh results after every rank-major dispatch
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

FP8 scaling can produce incorrect non-finite results, and benchmark profiling omits a required synchronization kernel.

8 open issues

• Rank-major combine overflows reciprocal for tiny FP8 scales
• Make overflow rejection collective across ranks
• Refresh results after every rank-major dispatch
• Prevent FP8 scale overflow during dequantization · New
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

1 previously missed issue

In code that hasn't changed since last review

Include throughput synchronization kernel in dispatch timing

python/​mscclpp_benchmark/​ep/​ep_bench_mscclpp.py:17

[P2] Include the throughput synchronization kernel in dispatch timing. This benchmark reuses preparation, so each dispatch launches synchronizeThroughputPeersKernel before throughputDispatchKernel (src/ext/ep/throughput.cc:230–239). Neither current substring matches the synchronization kernel, so Kineto omits this required peer barrier from dispatch and total communication latency. Match its actual name as well.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Accepted tiny FP8 scales can turn zero values into NaNs during combine.

9 open issues

• Rank-major combine overflows reciprocal for tiny FP8 scales
• Make overflow rejection collective across ranks
• Refresh results after every rank-major dispatch
• Preserve zero FP8 values when reciprocal scale overflows · New
• Prevent FP8 scale overflow during dequantization
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Accepted tiny FP8 scales can produce incorrect combine results, and profiling and public buffer documentation also need correction.

12 open issues

• Rank-major combine overflows reciprocal for tiny FP8 scales
• Make overflow rejection collective across ranks
• Refresh results after every rank-major dispatch
• Restrict E4M3 combine input to latency rank-major mode · New
• Include throughput peer synchronization in dispatch profiling · New
• Tiny FP8 scales overflow combine reciprocals · New
• Preserve zero FP8 values when reciprocal scale overflows
• Prevent FP8 scale overflow during dequantization
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Scale validation and synthetic benchmark weights permit numerical overflow that can produce invalid FP8 combine results.

12 open issues

• Rank-major combine overflows reciprocal for tiny FP8 scales
• Make overflow rejection collective across ranks
• Refresh results after every rank-major dispatch
• Tiny FP8 scales overflow combine reciprocals
• Include throughput peer synchronization in dispatch profiling
• Restrict E4M3 combine input to latency rank-major mode
• Preserve zero FP8 values when reciprocal scale overflows
• Prevent FP8 scale overflow during dequantization
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

2 previously missed issues

In code that hasn't changed since last review

Bound routing weights to prevent FP8 partial overflow

python/​mscclpp_benchmark/​ep/​ep_bench_common.py:120

[P2] Bound the synthetic routing weights so static FP8 rank-local partials remain representable. These unnormalized weights can make a weighted partial overflow even when the input passes the scale check. For example, four unit-weight routes to one rank turn a value of -128 into -512 at scale 1, outside E4M3's finite range. _stage_expert_output then produces NaNs and --validate fails before timing. Normalize each token's nonnegative weights, or choose the scale using the weighted partials as well as the inputs.

Reject scales with nonfinite FP32 reciprocals

src/​ext/​ep/​latency.cc:169

[P2] Reject scales whose FP32 reciprocal is nonfinite. A positive finite scale such as 1e-39f passes this check, but both rank-major combine kernels compute 1.0f / quantScale, producing infinity. Even a zero FP8 expert value then produces NaN during accumulation when its contribution is processed. Require a finite reciprocal before launching and add a regression case for this scale.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Accepted scales can produce NaN results, and optional PyTorch imports can break benchmark test collection.

13 open issues

• Rank-major combine overflows reciprocal for tiny FP8 scales
• Make overflow rejection collective across ranks
• Refresh results after every rank-major dispatch
• Avoid unconditional PyTorch import in benchmark module · New
• Tiny FP8 scales overflow combine reciprocals
• Include throughput peer synchronization in dispatch profiling
• Restrict E4M3 combine input to latency rank-major mode
• Preserve zero FP8 values when reciprocal scale overflows
• Prevent FP8 scale overflow during dequantization
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

1 previously missed issue

In code that hasn't changed since last review

Reject scales with overflowing FP32 reciprocals

src/​ext/​ep/​latency.cc:174

Reject scales whose FP32 reciprocal overflows. A finite positive scale such as 2^-130 passes this check, but both static-FP8 combine kernels compute 1.0f / quantScale, producing infinity. Even an all-zero expert row then produces NaNs through fma(0, infinity, accumulator). Require a finite reciprocal before launching, and add a regression case for this scale.

_{Review effort: Balanced}

[Binyang Li (Binyang2014)]

Can we provide a config for this? Default is bf16. user can set the datatype for combine

[Binyang Li (Binyang2014)]

Can we move this to quantization config

[Copilot]

🔵 Needs a closer look

Accepted static FP8 scales can overflow during reciprocal calculation and produce NaN combine outputs.

13 open issues

• Rank-major combine overflows reciprocal for tiny FP8 scales
• Make overflow rejection collective across ranks
• Refresh results after every rank-major dispatch
• Avoid unconditional PyTorch import in benchmark module
• Tiny FP8 scales overflow combine reciprocals
• Include throughput peer synchronization in dispatch profiling
• Restrict E4M3 combine input to latency rank-major mode
• Preserve zero FP8 values when reciprocal scale overflows
• Prevent FP8 scale overflow during dequantization
• Allow single-iteration rank-major CUDA graph replay
• Pass quantization scale to avoid reciprocal overflow
• Require matching scale across ranks in Python contract
• Document required identical scale across ranks

1 previously missed issue

In code that hasn't changed since last review

Reject scales with non-finite FP32 reciprocals

src/​ext/​ep/​latency.cc:174

[P2] Reject scales whose FP32 reciprocal is not finite. A scale such as 1e-40f passes this validation, but both rank-major static-FP8 combine kernels compute 1.0f / workload.quantScale_, which overflows to infinity. Even an all-zero expert row then produces NaNs through 0 * infinity. Check that the FP32 reciprocal is finite before launching.

_{Review effort: Balanced}