QMoE OP UINT4 support

[thpereir]

Describe the issue

I am working with AMD and I am trying to use 2xUINT4 packed into an UINT8 with the QMoE operator. I am seeing big numeric discrepancies going this route.

To reproduce

Single node graph with QMoE operator and 2x UINT4 packed into expected UINT8

Urgency

No response

Platform

Linux

OS Version

Ubuntu 24.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

main

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

CPU

[thpereir]

@kunal-vaishnavi fyi

[tianleiwu]

Only symmetric quantization is supported right now. During packing two int4 to uint8, just add 8 to convert int4 to uint4 then packed two uint4 into one byte (uint8). The kernel will unpack them into two int4 accordingly.

As long as the weight distribution is close to symmetric, we shall see decent accuracy with symmetric-quantization.

We can add support for asymmetric-quantization soon.

[thpereir]

Thanks for the update Tianlei Wu