Attention: one token asks, every earlier token answers

01 · The equation as a journey

Resolving “it”, step by step

The book's Figure 2.8 compresses attention into one line. Here is that line unpacked into the seven things that actually happen, following the book's own example: the token “it” arrives at an attention sublayer needing to figure out what it refers to.

Attention(Q, K, V) = softmax( QKᵀ / √d ) · V

The names are the database metaphor made literal. Q — query: what this token is looking for right now (“I'm a pronoun; where's my referent?”). K — key: how each token advertises what it can offer (“I'm a concrete noun, recently introduced”). V — value: the actual content a token contributes if selected. The crucial difference from a real database: instead of returning the single best match, attention returns a weighted blend of every value — a soft lookup, which keeps the whole operation differentiable and therefore trainable.

02 · The normalizer

Softmax: scores in, probabilities out

Softmax appears twice in this book already — here inside attention, and at the output layer turning logits into sampling probabilities (§2.2). It converts any list of real numbers into a valid probability distribution in two moves: exponentiate each score (making everything positive and stretching gaps), then divide by the sum (making everything add to 1). Drag the scores and watch both columns respond.

Three properties to notice while dragging. It never produces zero — even a terrible score gets a sliver of probability, so no token is ever fully ignored. It's order-preserving — ranks never change, only the sharpness of the distribution. And it's exponentially sensitive to gaps — raise one score a little and it drains probability from everyone else fast. That last property is why the equation divides scores by √d first: dot products of long vectors are naturally huge, and unscaled they'd push softmax into winner-take-all, where gradients vanish and training stalls. (It's also exactly what the temperature knob does at the output layer — dividing scores before softmax is the same trick.) One engineering note: real implementations subtract the max score before exponentiating — e³⁰⁰ overflows floating point — which changes nothing mathematically but everything numerically; doing this stably while streaming is a core piece of FlashAttention.

03 · Two kinds of attention

Self-attention and cross-attention

The mechanism is identical in both — only where Q, K, and V come from differs.

04 · The cost, and the cache

Quadratic by nature, linear in practice

Attention compares the current token with every earlier token, so a sequence of length n involves n × n pairwise scores — that's the quadratic cost the book warns about. But look at what actually changes when one new token is generated: every old token's K and V are exactly what they were last step. Recomputing them is pure waste. The KV cache stores them, so each decode step only computes the one new row of the matrix. Generate a few tokens in each mode and watch the counters diverge.

The trade is compute for memory: those cached K,V tensors live in GPU memory and grow with every token, every layer, every request in the batch — at long contexts the cache can outweigh the model weights themselves. The upper-left triangle you kept during the demo is the KV cache, built once during prefill and extended one row per decode step. Storing, evicting, and reusing it is a major topic of §5.3 — this little grid is why that section exists.