One token's trip through the transformer

01 · The walkthrough

Follow the token

The prompt is the book's own sentence, cut off mid-thought: “I decided to write a book because I thought it would be…” The model's job in this decode pass is to produce the next token. Step through the pipeline with the buttons; the map shows where you are, and the teal bar tracks what shape the data has at each point.

02 · Inside the attention sublayer

Multi-head attention, on the book's sentence

Attention relates the current token to every earlier one. Each head is a full, independent attention operation running in parallel, and different heads learn different kinds of relationships — the book gives subject–verb agreement and co-reference (figuring out what “it” refers to) as examples. Hover any word to see where that token's query lands; switch heads to see the same token asking a different question.

Under the hood each head computes the book's Figure 2.8: the current token's query (Q) is dot-producted against the keys (K) of all prior tokens, scores are softmaxed into weights like the arcs above, and those weights blend the prior tokens' values (V) into an update. Because every token looks at every earlier token, the raw cost grows quadratically with sequence length — the KV cache (built during prefill, §5.3) is what keeps decode linear by storing every K and V instead of recomputing them. These weights are hand-drawn for illustration; real heads are learned and messier — one famous quirk being that many heads dump spare attention on the first token, an “attention sink.”

03 · Where the weights live

The FFN is the heavyweight

The book notes the feed-forward sublayers hold the majority of an LLM's weights, attention is second, and normalization is “a rounding error.” Here is that claim made concrete with Llama-3-8B's actual dimensions (32 blocks, hidden size 4,096, FFN width 14,336, vocabulary 128,256):

04 · Reading the diagram like an engineer

Details worth noticing

• The stack is tall, and depth varies by scale. “Dozens to hundreds” of identical blocks: Llama-3-8B has 32, Llama-3-70B has 80, and the book's Qwen3-235B MoE example has 94. Same recipe per block; only count and widths change.
• Residual connections are the spine. Each sublayer's output is added to its input rather than replacing it (the “+” junctions in Figure 2.7). The hidden state is a running total that attention and FFN each nudge — this is what lets gradients survive a 100-layer round trip during training.
• Normalization is a thermostat, not a thinker. Before each sublayer, RMSNorm rescales the vector to a steady magnitude so 32–100 blocks of repeated adds don't blow up or fade out. Almost no weights, almost no compute, absolutely load-bearing.
• Attention mixes across tokens; the FFN thinks per token. Attention is the only place tokens exchange information. The FFN — the SwiGLU machine from §2.1.2 — is applied to each position independently, expanding 4,096 → 14,336 → back, bending the representation nonlinearly. Mix, then think; mix, then think; ×32.
• Prefill and decode run the same pipeline. This walkthrough followed one decode pass. Prefill (§2.2) pushes all prompt tokens through the same layers at once, stashing each one's K and V in the KV cache — which is exactly why our token “be” could look prior tokens up instead of reprocessing them.
• Only the last token gets a verdict. The LM head produces logits from the final hidden state of the newest position — everything earlier contributed via attention, but only one 128,256-wide prediction is made per decode pass.