Quantization

Normally, a model stores its numbers using relatively high precision, such as FP16(16-bit Floating Point) or BF16(Brain Floating Point 16-bit).

[!TIP] BF16 has the same number of exponent bits as FP32. That means BF16 can represent a similar numerical range to FP32, but with less precision. FP16 → more precision, smaller range BF16 → less precision, much larger range FP32 → high precision and large range Neural networks often don't need FP32-level precision for every calculation. BF16 gives them a large numerical range while cutting memory roughly in half compared with FP32.

Quantization converts them to lower-precision representations, such as INT8 or INT4.

For example:

text

FP16 model
  ↓ quantization
INT4 model

This can dramatically reduce memory usage.

A simplified example:

text

FP16:  0.738291
INT8:  117
INT4:  7

The lower-precision number represents the original value approximately.

Why use it?

Less GPU memory
Faster computation on supported hardware
Lower inference cost

Trade-off: too much quantization can reduce model quality.

Quantization

Connected concepts