Skip to content

[Relax][Frontend][ONNX] Add QLinearMatMul operator converter - #20951

Merged
mshr-h merged 6 commits into
apache:mainfrom
Chaitanya140904:relax-onnx-qlinearmatmul
Nov 17, 2026
Merged

mshr-h merged 6 commits into
apache:mainfrom
Chaitanya140904:relax-onnx-qlinearmatmul

Conversation

@Chaitanya140904

@Chaitanya140904 Chaitanya140904 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

This PR adds support for the ONNX QLinearMatMul operator in the Relax ONNX frontend, resolving #20958.

Key Changes

  • In python/tvm/relax/frontend/onnx/onnx_frontend.py:
    • Implemented QLinearMatMul op converter supporting opset 10 and 21.
    • Widens inputs to int32 before zero-point subtraction to prevent arithmetic overflow.
    • Computes matrix multiplication via relax.op.matmul(..., out_dtype="int32") for 32-bit integer accumulation.
    • Supports per-row and per-column scale and zero-point broadcasting.
    • Handles requantization: round(scaled) + y_zero_point and clamps to output data type range (uint8, int8, uint16, int16).
    • Registered "QLinearMatMul": QLinearMatMul in _get_convert_map().
  • In tests/python/relax/test_frontend_onnx.py:
    • Added test_qlinearmatmul testing 2D uint8, 2D int8, 3D batched int8/uint8, and opset 10/21, validated against ONNX Runtime.

Closes #20958.

@Chaitanya140904

Copy link
Copy Markdown
Contributor Author

Hi @mshr-h, @yzhliu, @tqchen, @akaashrp

Could you please take a look at this PR when you get a chance?

@Chaitanya140904

Copy link
Copy Markdown
Contributor Author

@tlopex
Could you please take a look at this PR when you get a chance?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Vector-scale broadcasting changes output shapes and values, and the integer-only opset 21 lowering incorrectly truncates float8 operands and clips float8 outputs.

4 open issues
What changed in this PR

Adds QLinearMatMul conversion to TVM’s Relax ONNX frontend, addressing the operator-support gap tracked in #20958.

Changes:

  • Registers converter entry points for opsets 10 and 21.
  • Implements int32 accumulation, quantization-parameter broadcasting, requantization, and output clipping.
  • Adds ONNX Runtime comparisons for 2D and batched int8/uint8 inputs.
File Description
tests/​python/​relax/​test_frontend_onnx.py Adds numerical checks across input shapes, signedness, and opsets.
python/​tvm/​relax/​frontend/​onnx/​onnx_frontend.py Implements and registers the QLinearMatMul converter.

Review effort: Max

@Chaitanya140904

Copy link
Copy Markdown
Contributor Author

Hi @tlopex, @mshr-h, @yzhliu, @tqchen, @akaashrp,

I have updated the PR to address all the recommended review changes:

  1. Float8 Handling: Explicitly reject float8 input and output dtypes with NotImplementedError.
  2. Preserve Vector Shapes: Added proper rank and scale alignment logic (_align_output_param) for a_scale, b_scale, y_scale, and y_zero_point to ensure matrix–vector and vector–matrix products preserve their expected 1D output shapes rather than expanding/broadcasting artificially.
  3. In-Range Output Validation: Seeded inputs close to zero-points with calibrated scales so outputs avoid saturating at clipping bounds and validate arithmetic, rounding, and zero-point addition.
  4. Expanded Tests: Added unit tests covering matrix–vector, vector–matrix, batched parameters, mixed signedness (uint8/int8), per-row/per-column quantization, reference calculations, and float8 rejection.

Could you please take another look when you have a moment? Thank you!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Valid singleton scales and output zero points can incorrectly increase MatMul output ranks, requiring scalar normalization and output-shape regression tests.

1 open issue
4 resolved since last review

Review effort: Max

…alars

Singleton quantization parameters with shape [1] were passed directly into
_align_output_param and the scale-broadcasting logic, which then expanded
them back to rank-1, incorrectly increasing the MatMul output rank.
For example, a_scale:[1] on [3] @ [3,4] produced output [1,4] instead of [4].

Fix: squeeze any singleton [1]-shape a_scale, b_scale, y_scale, and
y_zero_point to rank-0 scalars before alignment and broadcasting.

Adds a regression test asserting that [3] @ [3,4] with singleton [1]-shape
parameters produces a rank-1 output of shape [4].

Addresses GitHub Copilot review comment:
apache#20951 (comment)
@Chaitanya140904

Copy link
Copy Markdown
Contributor Author

Hi @tlopex, @mshr-h, @yzhliu, @tqchen, @akaashrp,

I have updated the PR to address the latest review feedback:

  1. Singleton Scale Normalization: Squeezed singleton [1]-shape quantization parameters and zero points to scalars before alignment and broadcasting so that vector dot-products and vector-matrix operations preserve their expected 1D output shapes rather than expanding rank (e.g., [3] @ [3, 4] now correctly yields shape [4] instead of [1, 4]).
  2. Regression Test: Added test_qlinearmatmul_singleton_scales_output_shape asserting both the rank/shape and numerical values for operations with [1]-shape quantization parameters.

Could you please take another look when you get a chance? Thank you!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Singleton scales are expanded before rank-one handling, adding an incorrect output axis to vector–matrix and matrix–vector products and failing the new shape regression.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

High severity Singleton scales add an incorrect output axis

python/​tvm/​relax/​frontend/​onnx/​onnx_frontend.py:6280

Singleton scales have already been expanded to [1, 1] before this rank-one check, so [3] @ [3, 4] with [1] parameters returns [1, 4] instead of [4] and fails the new shape regression. The symmetric matrix–vector case gains the same extra axis through b_scale at line 6263. Normalize singleton scales before expansion, or have this helper squeeze tensors whose dimensions are all one.

1 resolved since last review

Review effort: Max

@Chaitanya140904

Copy link
Copy Markdown
Contributor Author

Hi @tlopex, @mshr-h, @yzhliu, @tqchen, @akaashrp,

I have updated the PR to resolve the singleton quantization scale issue:

  1. Singleton Scale Normalization Before Expansion: Squeezed all singleton quantization parameters (where all shape dimensions are 1, e.g. [1], [1, 1]) to 0D scalars right at the beginning of _impl_v10 before any shape inspection or broadcasting occurs.
  2. Rank-Gated Scale Broadcasting: Restricted a_scale expansion (axis=-1) and b_scale expansion (axis=0) to occur only when both inputs have rank $\ge 2$ (a_rank >= 2 and b_rank >= 2). For vector–matrix products ([3] @ [3, 4]) and matrix–vector products ([4, 3] @ [3]), this prevents scales from gaining extra leading or trailing unit dimensions that alter the output rank.
  3. Scalar Alignment for Vector Dot Products: Explicitly squeezed param to a 0D scalar in _align_output_param when a_rank == 1 and b_rank == 1, ensuring vector dot products always produce scalar outputs.
  4. Expanded Regression Tests: Added tests in test_qlinearmatmul_singleton_scales_output_shape covering vector–matrix ([3] @ [3, 4] -> [4]), matrix–vector ([4, 3] @ [3] -> [4]), and vector–vector ([3] @ [3] -> ()) dot products with [1]-shape quantization parameters and verified numerical accuracy against NumPy.

Could you please take another look when you get a chance? Thank you!

@mshr-h
mshr-h merged commit 8f01d0d into apache:main Nov 17, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Relax][ONNX] Support QLinearMatMul in Relax ONNX frontend

3 participants