An AI model has billions of fine-tuned weights (parameters) after training. Normally, each weight is recorded as a long decimal like 3.14159 (32-bit float).
A 10 billion parameter model takes 40 GB of VRAM in full precision (32 bits = 4 bytes × 10B = 40 GB). Most consumer hardware can't run that.
The solution is quantization: reducing the numerical precision of model weights.
Quantization converts high-precision weight decimals into low-precision integers, lowering its memory needs (e.g., FP32 > FP16 > INT8 > INT4).
Here is what changes inside the model:
- Storage — replaces long decimals with small whole numbers
- Size — cuts memory usage by up to 75% or more (40 GB > 20 GB > 10 GB > 5 GB)
- Speed — small whole numbers are vastly faster for hardware to compute
Rounding causes a slight loss in precision, but models are resilient and retain accuracy even when compressed.
Next time you use a quantized model, see:
- How much VRAM did it save?
- Did lower precision (INT8 vs. INT4) preserve enough accuracy for your task?
- Did it make inference faster and cheaper?
Quantization doesn't change what a model knows — it just changes how compactly it holds that knowledge.