You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have updated the PR to address all the recommended review changes:
Float8 Handling: Explicitly reject float8 input and output dtypes with NotImplementedError.
Preserve Vector Shapes: Added proper rank and scale alignment logic (_align_output_param) for a_scale, b_scale, y_scale, and y_zero_point to ensure matrix–vector and vector–matrix products preserve their expected 1D output shapes rather than expanding/broadcasting artificially.
In-Range Output Validation: Seeded inputs close to zero-points with calibrated scales so outputs avoid saturating at clipping bounds and validate arithmetic, rounding, and zero-point addition.
Expanded Tests: Added unit tests covering matrix–vector, vector–matrix, batched parameters, mixed signedness (uint8/int8), per-row/per-column quantization, reference calculations, and float8 rejection.
Could you please take another look when you have a moment? Thank you!
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Valid singleton scales and output zero points can incorrectly increase MatMul output ranks, requiring scalar normalization and output-shape regression tests.
…alars
Singleton quantization parameters with shape [1] were passed directly into
_align_output_param and the scale-broadcasting logic, which then expanded
them back to rank-1, incorrectly increasing the MatMul output rank.
For example, a_scale:[1] on [3] @ [3,4] produced output [1,4] instead of [4].
Fix: squeeze any singleton [1]-shape a_scale, b_scale, y_scale, and
y_zero_point to rank-0 scalars before alignment and broadcasting.
Adds a regression test asserting that [3] @ [3,4] with singleton [1]-shape
parameters produces a rank-1 output of shape [4].
Addresses GitHub Copilot review comment:
apache#20951 (comment)
I have updated the PR to address the latest review feedback:
Singleton Scale Normalization: Squeezed singleton [1]-shape quantization parameters and zero points to scalars before alignment and broadcasting so that vector dot-products and vector-matrix operations preserve their expected 1D output shapes rather than expanding rank (e.g., [3] @ [3, 4] now correctly yields shape [4] instead of [1, 4]).
Regression Test: Added test_qlinearmatmul_singleton_scales_output_shape asserting both the rank/shape and numerical values for operations with [1]-shape quantization parameters.
Could you please take another look when you get a chance? Thank you!
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Singleton scales are expanded before rank-one handling, adding an incorrect output axis to vector–matrix and matrix–vector products and failing the new shape regression.
Singleton scales have already been expanded to [1, 1] before this rank-one check, so [3] @ [3, 4] with [1] parameters returns [1, 4] instead of [4] and fails the new shape regression. The symmetric matrix–vector case gains the same extra axis through b_scale at line 6263. Normalize singleton scales before expansion, or have this helper squeeze tensors whose dimensions are all one.
I have updated the PR to resolve the singleton quantization scale issue:
Singleton Scale Normalization Before Expansion: Squeezed all singleton quantization parameters (where all shape dimensions are 1, e.g. [1], [1, 1]) to 0D scalars right at the beginning of _impl_v10 before any shape inspection or broadcasting occurs.
Rank-Gated Scale Broadcasting: Restricted a_scale expansion (axis=-1) and b_scale expansion (axis=0) to occur only when both inputs have rank $\ge 2$ (a_rank >= 2 and b_rank >= 2). For vector–matrix products ([3] @ [3, 4]) and matrix–vector products ([4, 3] @ [3]), this prevents scales from gaining extra leading or trailing unit dimensions that alter the output rank.
Scalar Alignment for Vector Dot Products: Explicitly squeezed param to a 0D scalar in _align_output_param when a_rank == 1 and b_rank == 1, ensuring vector dot products always produce scalar outputs.
Expanded Regression Tests: Added tests in test_qlinearmatmul_singleton_scales_output_shape covering vector–matrix ([3] @ [3, 4] -> [4]), matrix–vector ([4, 3] @ [3] -> [4]), and vector–vector ([3] @ [3] -> ()) dot products with [1]-shape quantization parameters and verified numerical accuracy against NumPy.
Could you please take another look when you get a chance? Thank you!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support for the ONNX
QLinearMatMuloperator in the Relax ONNX frontend, resolving #20958.Key Changes
python/tvm/relax/frontend/onnx/onnx_frontend.py:QLinearMatMulop converter supporting opset 10 and 21.int32before zero-point subtraction to prevent arithmetic overflow.relax.op.matmul(..., out_dtype="int32")for 32-bit integer accumulation.round(scaled) + y_zero_pointand clamps to output data type range (uint8,int8,uint16,int16)."QLinearMatMul": QLinearMatMulin_get_convert_map().tests/python/relax/test_frontend_onnx.py:test_qlinearmatmultesting 2D uint8, 2D int8, 3D batched int8/uint8, and opset 10/21, validated against ONNX Runtime.Closes #20958.