Repository navigation
Skip QDQ to QLinear fusions when Q/DQ scales are not float32 - #33915
Open
Oke Iyanuoluwa Enoch (Iyanuoluwa007) wants to merge 1 commit into
Open
Oke Iyanuoluwa Enoch (Iyanuoluwa007) wants to merge 1 commit into
Oke Iyanuoluwa Enoch (Iyanuoluwa007) wants to merge 1 commit into
Conversation
The QLinear* contrib ops (QLinearAdd, QLinearMul, QLinearConcat, the unary QLinear ops, QLinearWhere), QLinearConv and QGemm only accept float32 scales. The QDQ selector/action transformer did not check the scale type, so a valid Q/DQ group with float16 scales was rewritten into an invalid node and session creation failed with INVALID_GRAPH. Add an opt-in check to BaseSelector::Select and enable it for the selectors whose replacement requires float32 scales. MatMul is left out because QLinearMatMul (opset 21) and MatMulIntegerToFloat accept float16 scales. The check is in BaseSelector::Select, so the node unit grouping that EPs get through GetQDQSelection is unchanged. Fixes microsoft#33788
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the change to prevent QDQ fusion with incompatible quantization scale types.
No open issues 🎉
What changed in this PR
Prevents QDQ fusion from creating invalid operators when quantization scales are not float32, addressing the session-creation failure reported in #33788.
Changes:
- Adds an opt-in float32-scale check for selectors whose replacement operators require it.
- Leaves MatMul selection and execution-provider node grouping unchanged.
- Adds a regression test verifying that Concat with float16 Q/DQ scales is not fused.
| File | Description |
|---|---|
onnxruntime/test/optimizer/qdq_transformer_test.cc |
Adds the float16-scale Concat regression test. |
onnxruntime/core/optimizer/qdq_transformer/selectors_actions/qdq_selectors.h |
Defines the scale-check flag and enables it for affected selectors. |
onnxruntime/core/optimizer/qdq_transformer/selectors_actions/qdq_selectors.cc |
Checks Q/DQ scale types and skips incompatible fusions. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The QDQ selector/action transformer rewrites Q/DQ node groups into QLinear* contrib ops (QLinearAdd, QLinearMul, QLinearConcat, the unary QLinear ops, QLinearWhere), QLinearConv and QGemm. These replacement ops only accept float32 scales, but the selectors never checked the scale type. A valid model whose Q/DQ nodes use float16 scales was therefore rewritten into an invalid node, and session creation failed with
INVALID_GRAPH.This change:
require_float_scalesflag toBaseSelector. When set,BaseSelector::Selectreturns no selection if any Q or DQ node in the group has a non-float32 scale.UnarySelector,BinarySelector,InputVariadicSelector,ConvSelector,WhereSelectorandGemmSelector.MatMulSelectorunchanged, sinceQLinearMatMul(opset 21) andMatMulIntegerToFloataccept float16 scales. The DropQDQ, DropDQ, Split and DQ MatMulNBits selectors are also unchanged, as they do not produce ops with scale type constraints.BaseSelector::Selectonly, so the node unit grouping that EPs obtain throughGetQDQSelection(e.g. QNN) is not affected.QDQTransformerTests.ConcatFloat16ScalesAreNotFused, which fails on main with the sameINVALID_GRAPHerror as the issue and passes with this change.Testing (Windows, Release build):
onnxruntime_test_all --gtest_filter=*QDQ*:*Qdq*:GraphTransformationTests.*:*TransposeOptimizer*:*NhwcTransformer*:*SelectorManager*passes (884 tests).Motivation and Context
Fixes #33788