A frontier model doesn't fit on one GPU. Each parallelism axis answers the same question differently — what gets divided, and what has to be communicated because of it — and because they divide along different dimensions, they compose: TP inside a node, PP or EP across nodes, DP across replicas.
Splits the tensors inside every layer. Each GPU holds a vertical slice of every weight matrix. Needs an all-reduce after every layer → NVLink territory, one node.
Splits the model by whole layers into stages. Layers stay intact; only activations hop stage-to-stage → tolerates InfiniBand, used only multi-node.
Shards whole experts of an MoE model. An expert is never split; the router is replicated on every GPU. Tokens travel to experts (all-to-all) → lighter traffic than TP, scales past a node.
Splits nothing inside the model — it divides the traffic. Full replicas each serve a share of requests, with zero inter-replica communication. The outermost axis: horizontal scaling.
Same model, four cut directions: TP cuts within a layer, PP cuts between layers, EP cuts between experts, DP doesn't cut the model at all.
Pick a strategy and watch where the weights land. The hard constraint underneath every choice: NVLink inside a node is fast; InfiniBand between nodes is much slower, so whatever chats the most must stay inside the node.
Each GPU multiplies the same input against its slice of the matrix. The matmul cost and, crucially, the weight-memory reads are shared 4 ways — that's why TP cuts decode latency.
Layer 1 … layer N: every GPU holds 1/4 of every layer — attention projections (WQ WK WV WO) and FFN matrices alike. For MoE models under pure TP, even each expert runs across all GPUs (fig 5.13).
Each GPU is a stage holding intact layers: GPU 0 → layers 1–20, GPU 1 → 21–40, … Only the activations cross InfiniBand, once per stage boundary — tiny traffic, so slow interconnect is fine.
Hatched = idle. A token visits stages in sequence, so at any instant 3 of 4 GPUs wait — the poor latency and utilization the book warns about. This is why PP is never used alone within a node where TP is available.
With full nodes you don't pick one axis — you nest them. TP8PP2: node A is stage 1 (layers 1–40), node B is stage 2 (layers 41–80), and within each node those layers are TP-split 8 ways over NVLink.
Try TP8 · PP2 in the explorer above to see the placement.
An expert never spans GPUs, so in-expert math never waits on the interconnect. Communication is only the all-to-all token dispatch: a token hops to whichever GPUs host its routed experts, and no per-layer all-reduce is needed.
Same eight GPUs, two axes in one layer. The dense attention sublayer benefits from TP's shared memory reads; the sparse expert sublayer benefits from EP's whole-expert locality. Many production MoE deployments run exactly this mix.
Data parallelism keeps complete copies of the model and splits incoming requests across them. Between replicas there is no model communication at all — just a load balancer — so DP crosses any network boundary for free and scales throughput linearly.
It's cross-cutting by construction: each replica internally uses whatever it needs — a DP2 deployment of two TP8 nodes is "DP on the outside, TP on the inside". The book's guidance: once a model no longer requires multi-node parallelism, extra nodes are usually better spent on DP replicas (or disaggregation) than on wider model parallelism.
| Axis | What is divided | Each GPU holds | Communication | Crosses nodes? | Optimizes |
|---|---|---|---|---|---|
| TP | Tensors within every layer | 1/N slice of every weight matrix | All-reduce every layer | No — NVLink only | Per-user latency (TPS) |
| PP | Whole layers → stages | A contiguous block of intact layers | Activations, once per stage | Yes — designed for it | Fitting big models multi-node (with TP inside) |
| EP | Whole experts (MoE only) | Experts ÷ N, intact + router copy | All-to-all token dispatch | Yes — light traffic | System throughput / cost |
| DP | The request stream | The full model (per replica group) | None between replicas | Yes — trivially | Aggregate throughput, scaling out |