DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
[Uczenie maszynowe] Autorzy: Mahsa Salmani, Sayeh Sharify, Hesham Mostafa
Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requires calibration data. In this work, we introduce DUALQUANT, a dual-scale, calibration-free post-training quantization framework that directly minimizes the weight-reconstruction Mean Squared Error (MSE) and, as an immediate by-product, yields per-channel activation rescalings prior to activation quantization. Our key observation is that the scaling factors obtained by minimizing weight-reconstruction error provide effective activation rescalings, enabling competitive activation quantization without calibration data. We formulate DUALQUANT as a non-differentiable weight-rec…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe