KV caching
This one is especially important for understanding how ChatGPT-like generation works.
Suppose you ask:
"Explain quantum computing in simple terms."
The model processes the prompt and begins generating:
Explain → quantum → computing → in → simple → terms → ...When generating the next token, the model needs information from the previous tokens.
Without caching, it would repeatedly recompute information about all previous tokens.
KV cache stores certain intermediate attention values:
flowchart TD
A[Previous tokens] --> B[KV Cache]
B --> C[Key K cache]
B --> D[Value V cache]
C --> E[Reuse during next-token generation]
D --> E
So instead of repeatedly calculating everything from scratch, the model can reuse the stored Keys and Values.
This significantly improves token-generation speed, especially for long conversations.
The trade-off is that the KV cache itself consumes GPU memory.