Quantization from Scratch: Scale, Zero-Point, GPTQ and AWQ

Small and Specialized Models found that decoding one sequence is memory-bound: tokens per second are capped by how many bytes of weights must be read per token. Halve the bytes and the cap doubles. Quantization stores weights in fewer bits, 8 or 4 instead of 16, and it is the single biggest reason large models run on small hardware. This post builds it from the rounding step up, and measures each refinement on real layers of a real model.

Table of contents

Scale and zero-point

To store a number in $b$ bits, map it onto the integers $0$ to $2^b - 1$. For a block of weights with minimum $w_{\min}$ and maximum $w_{\max}$, the scale is the step between grid points and the zero-point is the integer that represents zero:

and the weight is read back as $\hat{w} = (q - z)\, s$. Storing $z$ makes zero exactly representable, which matters because zero padding and many activations are exactly zero. Worked for eight weights at 4 bits, in the first lines of the output below: the scale is 0.052, so every weight moves to the nearest multiple of 0.052, and 0.47 becomes 0.468. The rounding error is at most half a step, $s/2$, and the step is set by the block’s range: one large weight in a block makes the step large for every other weight in it.

A symmetric grid drops the zero-point: it spans $[-\max|w|, \max|w|]$ with $2^{b-1} - 1$ levels on each side (15 levels at 4 bits, here; many kernels use the 16th as an extra negative level), so zero is a grid point by construction and dequantizing is a single multiplication. The price is range: if a block’s weights run from $-0.1$ to $+0.3$, a third of a symmetric grid covers $[-0.3, -0.1]$, where there are no weights at all.

The widget quantizes 128 real weights, one group from layer 22 of Qwen2.5-0.5B. Try the group with an outlier at 4 bits: with the range taken from the group, the outlier stretches the grid and most weights share a handful of levels; with the range taken from the whole row, as one scale per channel would, it gets worse; switch to symmetric and watch half the grid go unused:

Where to put the scales

That last point decides the first design choice, how many weights share a scale:

• Per tensor: one scale for a whole matrix. Cheapest, and ruined by the largest weight anywhere in it.
• Per channel: one scale per output row.
• Per group: one scale per run of, typically, 128 consecutive weights in a row. The scales cost a little memory, one 16-bit scale and a 4-bit zero-point per 128 weights, about 0.16 bits per weight (0.25 if the zero-point is kept in 16 bits), and the step adapts to local ranges.

Formats beyond integers

Integers aren’t the only small grid:

• NF4 (Dettmers et al., 2023, for QLoRA): 16 levels placed at quantiles of a normal distribution, scaled per block of 64 by the block’s largest magnitude. Weights are roughly normal, so the levels sit where the weights are, and no zero-point is stored. The 32-bit block scales cost 0.5 bits per weight; QLoRA’s double quantization of the scales cuts that to 0.127. QLoRA (Parameter-Efficient Fine-Tuning) fine-tunes on top of it.
• FP8 (Micikevicius et al., 2022): an 8-bit floating-point number, in two variants: E4M3 (4 exponent bits, 3 mantissa bits, up to 448) for weights and activations, and E5M2 (up to 57,344) where range matters more, as in gradients. Ada, Hopper and later GPUs multiply FP8 natively; Blackwell also runs the MX formats below.
• MX formats (the OCP Microscaling specification, 2023): blocks of 32 values share one 8-bit power-of-two scale, with the values themselves in FP8, FP6, FP4 or INT8. A block scale in hardware is what group-wise quantization does in software.

Post-training quantization or quantization-aware training

Everything in this post is post-training quantization (PTQ): take a trained model, quantize its weights, perhaps with some calibration data. It costs minutes to a few GPU-hours and no gradient updates. Quantization-aware training (QAT) inserts the rounding into training itself: the forward pass uses quantized weights, and the backward pass treats the rounding as if it were the identity, the straight-through estimator, so the model learns weights that survive rounding. It recovers more quality at very low bit widths, at the cost of a training run.

GPTQ: rounding with error compensation

Rounding each weight to its nearest grid point minimizes each weight’s own error. It doesn’t minimize what matters, the error in the layer’s output on real inputs, $\|XW^\top - X\hat{W}^\top\|^2$, because errors in different weights can cancel or add up depending on the inputs. GPTQ (Frantar et al., 2023), descended, via Optimal Brain Quantization, from Optimal Brain Surgeon (Hassibi and Stork, 1993), quantizes one input column at a time and, after rounding each, adjusts the columns not yet quantized to compensate. The adjustment comes from the second derivative of the output error, the Hessian $H = 2X^\top X$, estimated on calibration text:

Here $H^{-1}$ is the inverse Hessian over the columns not yet quantized, which changes after every step; row $i$ of the Cholesky factor of $H^{-1}$ gives all of these at once, which is why GPTQ uses it. Columns whose inputs are correlated with column $i$ absorb its error. The whole layer is processed in one pass, in blocks of 128 columns to keep it fast on a GPU; Frantar et al. quantized 175-billion-parameter models to 3 and 4 bits in about four GPU hours.

AWQ: protect the channels that matter

Activations are not uniform: a few input channels carry much larger values than the rest, so errors in the weights that multiply them cost more. AWQ (Lin et al., 2024) scales those weights up before quantizing, and the corresponding inputs down by the same factor: $W x = (W \operatorname{diag}(s)) (\operatorname{diag}(s)^{-1} x)$, exact in full precision. The scaled-up weights get finer relative steps. With $s_j = \overline{|x_j|}^{\alpha}$, one exponent $\alpha$ per layer is chosen on calibration data, and the inverse scaling is folded into the previous operation, so it costs nothing at inference. The full method also searches, for each group, how far to clip its largest weights before rounding: a smaller range with a few clipped weights can beat a wider one with a coarser step. Both halves are measured below.

Measured on two real layers

All four, implemented from the descriptions above, on the MLP down-projections of layers 12 and 22 of Qwen2.5-0.5B, with their real inputs from this blog’s text. Error is measured on the layer’s output, on 2,048 tokens that calibration never saw. capture.py records those inputs first (it downloads the model, about 1 GB), and measure.py then runs on the CPU in about 2 minutes.

Full script: capture.py

Round-to-nearest with a scale and a zero-point:

def quantize(w, bits, axis=None):
    """Asymmetric round-to-nearest: an integer grid 0 .. 2^bits - 1, a scale
    and a zero-point so the grid spans [min, max] along `axis`."""
    lo, hi = w.min(axis=axis, keepdims=True), w.max(axis=axis, keepdims=True)
    scale = (hi - lo) / (2 ** bits - 1)
    scale = np.where(scale == 0, 1, scale)
    zero = np.round(-lo / scale)
    q = np.clip(np.round(w / scale) + zero, 0, 2 ** bits - 1)
    return (q - zero) * scale

Full script: measure.py

The same file holds the symmetric and NF4 variants and rtn, which applies quantize per tensor, per output row or per group of 128. GPTQ and AWQ’s scaling search, as the script implements them:

def gptq(W, bits, H, block=128, damp=0.01):
    """GPTQ (Frantar et al., 2023): quantize one input column at a time, in
    groups of 128, and push each column's rounding error onto the columns
    not yet quantized, weighted by the inverse Hessian of the layer's
    squared output error, H = 2 X^T X."""
    W = W.copy(); n = W.shape[1]
    H = H.copy()
    H += damp * np.mean(np.diag(H)) * np.eye(n)
    Hinv = np.linalg.cholesky(np.linalg.inv(H)).T              # upper Cholesky factor of H^-1
    Q = np.zeros_like(W)
    for i1 in range(0, n, block):
        i2 = i1 + block
        W1, E1 = W[:, i1:i2].copy(), np.zeros((W.shape[0], block))
        lo, hi = W1.min(1), W1.max(1)                             # this group's grid, from current weights
        scale = np.where(hi > lo, (hi - lo) / (2 ** bits - 1), 1); zero = np.round(-lo / scale)
        for i in range(block):
            col = W1[:, i]
            q = (np.clip(np.round(col / scale) + zero, 0, 2 ** bits - 1) - zero) * scale
            Q[:, i1 + i] = q
            e = (col - q) / Hinv[i1 + i, i1 + i]
            W1[:, i:] -= np.outer(e, Hinv[i1 + i, i1 + i:i2])     # compensate within the block
            E1[:, i] = e
        W[:, i2:] -= E1 @ Hinv[i1:i2, i2:]                        # and in the columns after it
    return Q

def awq(W, bits, H, mag, grid=np.linspace(0, 1, 11)):
    """AWQ (Lin et al., 2024): scale input channel j of the weights up by
    s_j = mean|x_j|^alpha and its inputs down by the same factor, so the
    channels that matter most get finer steps; alpha chosen on calibration
    data. The scaling is exact in full precision."""
    best = None
    for a in grid:
        s = mag ** a; s /= np.sqrt(s.max() * s.min())
        Q = rtn(W * s, bits, "group 128") / s
        D = W - Q
        e = np.sum((D @ H) * D)                                   # calibration output error, from H
        if best is None or e < best[0]:
            best = (e, a, Q)
    return best[2], best[1]

Full script: measure.py

AWQ’s clipping search (awq_clip) follows in the file, then the worked example and the loop that measures every method on both layers.

worked, 8 weights to 4 bits: scale 0.0520, zero-point 6
  integers [0, 4, 6, 7, 8, 9, 10, 15]
  back     [-0.312, -0.104, 0.0, 0.052, 0.104, 0.156, 0.208, 0.468]

=== layer 12: input channels' mean |x|: median 0.067, largest 0.52 (8x); top 1% of channels carry 3% of the total
round-to-nearest, output error (held-out tokens):
                   8 bits       4 bits       3 bits
per tensor          4.95%       72.79%       94.82%
per channel         1.08%       18.17%       38.32%
group 128           0.64%       10.98%       23.31%
4 bits, blocks of 64: asymmetric 9.66%, symmetric 11.74%, NF4 9.34%
group 128, output error (held-out tokens):
  4 bits: round-to-nearest 10.98%; GPTQ 8.99%; AWQ scaling only 10.90% (alpha 0.2), with clipping 10.37%
  3 bits: round-to-nearest 23.31%; GPTQ 19.44%; AWQ scaling only 23.25% (alpha 0.2), with clipping 20.52%
  GPTQ at 4 bits with 2,048 calibration tokens, by damping (fraction of H's mean diagonal): 0.01: 12.78%, 0.1: 10.48%, 1: 9.46%; on its own calibration tokens (0.01): 3.76%

=== layer 22: input channels' mean |x|: median 0.146, largest 8.53 (58x); top 1% of channels carry 12% of the total
round-to-nearest, output error (held-out tokens):
                   8 bits       4 bits       3 bits
per tensor          3.69%       57.74%       85.07%
per channel         1.03%       17.38%       36.14%
group 128           0.63%       10.64%       22.62%
4 bits, blocks of 64: asymmetric 9.40%, symmetric 11.53%, NF4 8.51%
group 128, output error (held-out tokens):
  4 bits: round-to-nearest 10.64%; GPTQ 6.87%; AWQ scaling only 8.15% (alpha 0.4), with clipping 7.70%
  3 bits: round-to-nearest 22.62%; GPTQ 15.06%; AWQ scaling only 17.13% (alpha 0.4), with clipping 15.47%
  GPTQ at 4 bits with 2,048 calibration tokens, by damping (fraction of H's mean diagonal): 0.01: 8.23%, 0.1: 7.03%, 1: 6.95%; on its own calibration tokens (0.01): 2.64%

• 8 bits is nearly free; 4 bits needs care; 3 bits hurts. With group scales, 8-bit weights change the output by 0.6%, 4-bit by about 11%, and 3-bit by about 23%, in both layers.
• Granularity matters more than anything else at low bits. At 4 bits in layer 12, one scale per tensor gives 73% output error, per channel 18%, per group of 128 11%. Smaller blocks mean a smaller range per block, so a smaller step.
• GPTQ’s compensation helps, given enough calibration. At 4 bits it cuts the group-128 error from 11.0% to 9.0% in layer 12 and from 10.6% to 6.9% in layer 22; at 3 bits, from 23% to 19% and 15%. The counterfactual is calibration size. With 2,048 calibration tokens instead of 32,768, GPTQ in layer 12 is worse than plain rounding (12.8% against 11.0%). $X^\top X$ from 2,048 tokens of 4,864-dimensional inputs has rank at most 2,048; in the directions those tokens never exercised, the Hessian is only the small damping term, so GPTQ treats them as free and pushes rounding error into them, and held-out text does use them. On its own calibration tokens it scored 3.8%: overfitting. Raising the damping from 1% to 100% of the Hessian’s mean diagonal brings it back to 9.5%; in layer 22, the 2,048-token run still beat rounding (8.2% against 10.6%). For scale, the GPTQ paper calibrates on 128 sequences of 2,048 tokens, 262,000 tokens.
• AWQ’s scaling helps where there are outliers, and only there. Layer 12’s largest input channel averages 8 times the median, and the scaling changes nothing (10.9% against 11.0%). Layer 22’s largest is 58 times the median, the top 1% of channels carry 12% of the input’s size, and the scaling cuts the 4-bit error from 10.6% to 8.2% and the 3-bit from 22.6% to 17.1%. Its clipping search helps both layers: with it, AWQ reaches 10.4% in layer 12 and 7.7% in layer 22 at 4 bits. GPTQ is still ahead in both.

At 4 bits with QLoRA’s blocks of 64, the formats line up as the theory says: symmetric absmax is worst (11.7% and 11.5%): its range is set by whichever side of zero is larger, and its grid has 15 levels, not 16, which together make its step coarser, and its error about 20% higher. Asymmetric with a zero-point gets 9.7% and 9.4%, paying a few bits for its zero-points; NF4, with levels shaped like the weights, 9.3% and 8.5%, without one.

Output error in one layer isn’t the model’s quality: errors compound through 24 layers, and the papers’ own numbers are perplexity over whole models on WikiText-2 and C4. These per-layer errors, on two of the 24 layers and one of the seven linear layers in each, are a proxy for that, good for seeing the mechanisms, not for ranking the methods on a model.

What it buys at decode

# What 4-bit weights buy at decode, by batch size, for an 8B model on an
# H100 (3.35 TB/s; 989 TFLOP/s dense BF16). One decode step for a batch of
# B sequences reads every weight once and does 2 FLOPs per weight per
# sequence; its time is at least the larger of the two ("How a GPU Runs Deep Learning"'s roofline;
# KV-cache reads and dequantization ignored). INT4 weights with group
# scales take about 4.25 bits each.
N, BW, FLOPS = 8e9, 3.35e12, 989e12
print(f"{'batch':>6s} {'BF16 step':>10s} {'INT4 step':>10s} {'speedup':>8s}   bound by")
for B in (1, 8, 64, 128, 256, 1024):
    compute = 2 * N * B / FLOPS
    bf16, int4 = max(2 * N / BW, compute), max(4.25 / 8 * N / BW, compute)
    print(f"{B:6d} {bf16 * 1e3:8.2f} ms {int4 * 1e3:8.2f} ms {bf16 / int4:7.1f}x   "
          f"BF16 {'memory' if 2 * N / BW > compute else 'compute'}, INT4 {'memory' if 4.25 / 8 * N / BW > compute else 'compute'}")
print(f"compute-bound from a batch of {4.25 / 8 * N / BW * FLOPS / (2 * N):.0f} (INT4) and "
      f"{2 * N / BW * FLOPS / (2 * N):.0f} (BF16)")

Full script: decode_speed.py

 batch  BF16 step  INT4 step  speedup   bound by
     1     4.78 ms     1.27 ms     3.8x   BF16 memory, INT4 memory
     8     4.78 ms     1.27 ms     3.8x   BF16 memory, INT4 memory
    64     4.78 ms     1.27 ms     3.8x   BF16 memory, INT4 memory
   128     4.78 ms     2.07 ms     2.3x   BF16 memory, INT4 compute
   256     4.78 ms     4.14 ms     1.2x   BF16 memory, INT4 compute
  1024    16.57 ms    16.57 ms     1.0x   BF16 compute, INT4 compute
compute-bound from a batch of 78 (INT4) and 295 (BF16)

At small batches a decode step only reads weights, so 4-bit weights make it 3.8 times faster. As the batch grows, the same weights serve more sequences per read and compute catches up: 4-bit weights are compute-bound from a batch of about 80, so at 128 the speedup is 2.3 and at 256, 1.2. At 1,024 both are compute-bound and weight-only quantization buys nothing. Hence the split in practice: weight-only 4-bit for local and low-batch serving, and 8-bit weights and activations, using the GPU’s integer or FP8 units, for high-throughput serving. Two things this arithmetic ignores both shrink the gain: dequantizing in the kernel costs time, and the KV cache is read too. An 8B model with GQA, such as Llama 3 8B (32 layers, 8 key-value heads of 128 dimensions), stores $32 \times 8 \times 128$ keys and as many values per token, 2 bytes each: 128 KiB. At a batch of 64 with 4,000-token contexts, that is $128\ \text{KiB} \times 4{,}000 \times 64 \approx 33.6$ GB read per step, twice its 16 GB of 16-bit weights. With 4.25 GB of 4-bit weights the step then speeds up only $(16 + 33.6) / (4.25 + 33.6) \approx 1.3$ times. The KV cache has its own quantization (see KV Cache Compression).

Weights only, or activations too

Everything above quantizes weights only: they are stored in 4 bits and expanded to 16 for the multiplication, which helps exactly where Small and Specialized Models said the bottleneck is, reading weights during decode. Quantizing the activations too, to run the multiplication itself in 8-bit integers, also speeds up compute-bound prefill and large batches, but activation outliers make it harder: LLM.int8() (Dettmers et al., 2022) keeps the few outlier dimensions in 16 bits, and SmoothQuant (Xiao et al., 2023) migrates the outliers’ range from activations into weights with the same per-channel scaling trick as AWQ.

Where this leads

• Running LLMs on a Laptop runs quantized models on a laptop: GGUF formats and llama.cpp, with measured speed against Small and Specialized Models’s bandwidth bound.
• TensorRT-LLM fixes a quantized format into a compiled engine, and lists which formats (FP8, NVFP4, GPTQ and AWQ weights) run on which GPUs.

Takeaways

• Quantization stores each weight as an integer on a grid set by a scale and a zero-point; the error is up to half a step, and the step is set by the range of the block sharing the scale.
• Group-wise scales (128 weights) are what make 4 bits workable: 11% output error on a real layer, against 73% with one scale per tensor.
• GPTQ rounds column by column and pushes each error onto correlated columns, cutting 4-bit output error by a fifth to a third here, given enough calibration text; with 2,048 tokens and default damping it overfit, and in layer 12 was worse than plain rounding.
• AWQ’s scaling by activation size helped only in the layer with large activation outliers (10.6% to 8.2%); its clipping search helped both.
• At 4 bits in blocks of 64, NF4 beat asymmetric and symmetric integer grids on both layers (9.3% and 8.5% output error).
• Weight-only quantization speeds up memory-bound decoding, 3.8 times at small batches for 4-bit weights, and nothing once the batch makes the step compute-bound; quantizing activations as well has to handle their outliers.

References

Start with the GPTQ and AWQ papers, the two methods this post implements; QLoRA introduces NF4, and SmoothQuant covers quantizing activations.

• Micikevicius et al., FP8 Formats for Deep Learning (2022): the FP8 formats, E4M3 and E5M2.
• Open Compute Project, OCP Microscaling Formats (MX) Specification v1.0 (2023): the MX block formats with a shared scale per 32 numbers.
• Dettmers, Pagnoni, Holtzman and Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS, 2023): NF4, the 4-bit grid placed at a normal distribution’s quantiles.
• Hassibi and Stork, Second Order Derivatives for Network Pruning: Optimal Brain Surgeon (NIPS 5, 1993): the second-order error compensation GPTQ descends from.
• Frantar, Ashkboos, Hoefler and Alistarh, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR, 2023): GPTQ, the column-by-column method implemented here.
• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys, 2024; Best Paper): AWQ, the activation-aware scaling and clipping search implemented here.
• Dettmers, Lewis, Belkada and Zettlemoyer, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (NeurIPS, 2022): keeping outlier activation dimensions in 16 bits.
• Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML, 2023): moving activation outliers into the weights to quantize both.