learn

Model Parallelism

1 min read

4. Model parallelism

Modern LLMs can be too large to fit on a single GPU.

For example, imagine a model requires 80 GB of GPU memory, but you only have GPUs with 40 GB each.

You could split the model:

text

              LLM
               │
       ┌───────┴───────┐
       ↓               ↓
    GPU 1             GPU 2
   40 GB              40 GB

This is model parallelism.

There are several ways to do it.

Tensor parallelism: split individual mathematical operations/tensors across GPUs.

text

Large matrix operation
        ↓
 ┌──────┴──────┐
GPU 1         GPU 2
part          part

Pipeline parallelism: split different layers of the model across GPUs.

text

GPU 1                 GPU 2                 GPU 3
Layers 1–20           Layers 21–40          Layers 41–60
   ↓                      ↓                     ↓
 ───────────────────────────────────────────────→

This allows very large models to run across multiple GPUs.

Learning checkpoint

Mark this guide complete to include it in your local Engineering Journey.

Knowledge path