Inference Engineering · §5.4 Model Parallelism

Four Ways to Split a Model: TP · PP · EP · DP

A frontier model doesn't fit on one GPU. Each parallelism axis answers the same question differently — what gets divided, and what has to be communicated because of it — and because they divide along different dimensions, they compose: TP inside a node, PP or EP across nodes, DP across replicas.

1 · What each axis divides

TP — Tensor Parallelism

Splits the tensors inside every layer. Each GPU holds a vertical slice of every weight matrix. Needs an all-reduce after every layer → NVLink territory, one node.

PP — Pipeline Parallelism

Splits the model by whole layers into stages. Layers stay intact; only activations hop stage-to-stage → tolerates InfiniBand, used only multi-node.

EP — Expert Parallelism

Shards whole experts of an MoE model. An expert is never split; the router is replicated on every GPU. Tokens travel to experts (all-to-all) → lighter traffic than TP, scales past a node.

DP — Data Parallelism

Splits nothing inside the model — it divides the traffic. Full replicas each serve a share of requests, with zero inter-replica communication. The outermost axis: horizontal scaling.

Same model, four cut directions: TP cuts within a layer, PP cuts between layers, EP cuts between experts, DP doesn't cut the model at all.

2 · Place a model on real hardware

Pick a strategy and watch where the weights land. The hard constraint underneath every choice: NVLink inside a node is fast; InfiniBand between nodes is much slower, so whatever chats the most must stay inside the node.

3 · TP on one node: how the weights are divided

One weight matrix, TP4 — sliced by columns
GPU 0
GPU 1
GPU 2
GPU 3
all-reduce over NVLink — partial results → one output, every layer

Each GPU multiplies the same input against its slice of the matrix. The matmul cost and, crucially, the weight-memory reads are shared 4 ways — that's why TP cuts decode latency.

…and it's every matrix of every layer

Layer 1 … layer N: every GPU holds 1/4 of every layer — attention projections (WQ WK WV WO) and FFN matrices alike. For MoE models under pure TP, even each expert runs across all GPUs (fig 5.13).

  • Buys: lower per-user latency (TPS↑) — the default multi-GPU strategy.
  • Costs: a synchronizing all-reduce per layer → not suitable across nodes.

4 · PP: 1 GPU per node vs 8 GPUs per node

1 GPU / node → pure PP4 (layers stay whole)

Each GPU is a stage holding intact layers: GPU 0 → layers 1–20, GPU 1 → 21–40, … Only the activations cross InfiniBand, once per stage boundary — tiny traffic, so slow interconnect is fine.

The catch: the pipeline bubble (one request, time →)
GPU 0 · L1–20 GPU 1 · L21–40 GPU 2 · L41–60 GPU 3 · L61–80

Hatched = idle. A token visits stages in sequence, so at any instant 3 of 4 GPUs wait — the poor latency and utilization the book warns about. This is why PP is never used alone within a node where TP is available.

8 GPUs / node → cross-cutting: TP8 inside · PP2 between

With full nodes you don't pick one axis — you nest them. TP8PP2: node A is stage 1 (layers 1–40), node B is stage 2 (layers 41–80), and within each node those layers are TP-split 8 ways over NVLink.

  • Inside a node: chatty all-reduce traffic rides NVLink — where it's cheap.
  • Between nodes: only one activation handoff per stage rides InfiniBand.
  • Every GPU holds 1/8 of half the layers = 1/16 of the model.
  • For dense models needing >8 GPUs, this is the recommended shape (e.g. TP8PP2 on 16 GPUs).
NVLink: all-reduce / layerInfiniBand: activations ×1

Try TP8 · PP2 in the explorer above to see the placement.

5 · EP: whole experts, and the TP+EP mix inside one layer

EP8 · one node · 128-expert MoE → 16 whole experts per GPU
router — replicated on every GPU (it's tiny)

An expert never spans GPUs, so in-expert math never waits on the interconnect. Communication is only the all-to-all token dispatch: a token hops to whichever GPUs host its routed experts, and no per-layer all-reduce is needed.

  • Buys: system throughput — each token is no faster, but the node handles many more at once.
  • Scales: lighter traffic than TP → EP16 across two nodes works over InfiniBand.
Cross-cutting inside a single transformer block (fig 5.15)
Attention
TP8 — heads/projections sliced across all 8 GPUs, all-reduce to combine
MoE FFN
EP8 — 16 whole experts per GPU, tokens routed to them

Same eight GPUs, two axes in one layer. The dense attention sublayer benefits from TP's shared memory reads; the sparse expert sublayer benefits from EP's whole-expert locality. Many production MoE deployments run exactly this mix.

6 · DP: divide the traffic, not the model

Data parallelism keeps complete copies of the model and splits incoming requests across them. Between replicas there is no model communication at all — just a load balancer — so DP crosses any network boundary for free and scales throughput linearly.

It's cross-cutting by construction: each replica internally uses whatever it needs — a DP2 deployment of two TP8 nodes is "DP on the outside, TP on the inside". The book's guidance: once a model no longer requires multi-node parallelism, extra nodes are usually better spent on DP replicas (or disaggregation) than on wider model parallelism.

  • What's divided: the request stream. Every replica holds 100% of the weights.
  • Requires: the model (plus KV cache) must fit inside one replica's GPUs.
  • Buys: throughput and fault isolation; per-request latency is unchanged.
  • In-model DP variants (e.g. Context Parallelism, splitting the sequence across GPUs) are rare in LLM inference but essential for video generation (§6.6).
inter-replica traffic: noneoutermost axis

7 · Side by side

AxisWhat is dividedEach GPU holdsCommunicationCrosses nodes?Optimizes
TPTensors within every layer1/N slice of every weight matrixAll-reduce every layerNo — NVLink onlyPer-user latency (TPS)
PPWhole layers → stagesA contiguous block of intact layersActivations, once per stageYes — designed for itFitting big models multi-node (with TP inside)
EPWhole experts (MoE only)Experts ÷ N, intact + router copyAll-to-all token dispatchYes — light trafficSystem throughput / cost
DPThe request streamThe full model (per replica group)None between replicasYes — triviallyAggregate throughput, scaling out
Rules of thumb (§5.4.3)
  • One node: TP is the default. MoE model? Mix in EP (TP for attention, EP for experts).
  • Two+ nodes, dense model: TP inside each node, PP between them — TP8PP2.
  • Two+ nodes, MoE model: TP8PP2 for lower per-user latency, EP16 for higher total throughput.
  • Model fits in one node? Don't go multi-node at all — spend extra nodes on DP replicas or disaggregation.
Source: Inference Engineering — §5.4 Model Parallelism (5.4.1 Tensor · 5.4.2 Expert · 5.4.3 Multi-Node), figs 5.12–5.16