These are four important techniques used to make LLM inference faster, cheaper, or possible on limited hardware.
A simple way to remember them:
- Quantization → make the model smaller
- Batching → process multiple requests together
- KV caching → avoid recomputing old tokens
- Model parallelism → split the model across multiple GPUs
Putting everything together
An LLM inference system might look something like:
flowchart TD
A[User requests] --> B[Batching]
B --> C[Quantized LLM]
C --> G1[GPU 1]
C --> G2[GPU 2]
C --> G3[GPU 3]
C --> G4[GPU 4]
G1 --> D[KV Cache]
G2 --> D
G3 --> D
G4 --> D
D --> E[Generated tokens]
So, in one sentence:
Quantization reduces the model's memory footprint, batching improves GPU utilization, KV caching avoids repeated attention computation, and model parallelism allows a large model to run across multiple GPUs.
