Batching
Suppose 10 users send requests at roughly the same time.
Instead of processing:
flowchart LR
R1[Request 1] --> GPU[GPU]
R2[Request 2] --> GPU
R3[Request 3] --> GPU
R4[...]
R4 --> GPU
R10[Request 10] --> GPU
the system can combine them:
flowchart LR
B[GPU Batch] --> R1[Request 1]
B --> R2[Request 2]
B --> R3[Request 3]
B --> R4[...]
B --> R10[Request 10]
The GPU is much better utilized because modern GPUs can perform many operations in parallel.
There are two common concepts:
Static batching: wait for a group of requests and process them together.
Continuous/dynamic batching: the inference server continuously adds new requests as others finish.
The second approach is particularly useful for LLM serving.