Activation Functions, Visualized

01 · The problem

Two linear layers are secretly one

Here are two “layers” acting on a single number: layer 1 computes a₁·x + b₁, layer 2 takes that result and computes a₂·(·) + b₂. Move any slider you like — the composed result is always a straight line. The four parameters buy you nothing that two couldn't: the layers collapse, exactly as the matmul version does in Figure 2.3.

Now flip on the ReLU wedged between them and the composed function grows a bend. The two layers stop being redundant: layer 1 decides where the bend sits, layer 2 decides what happens on either side of it.

a₁+1.3

b₁+0.6

a₂+0.9

b₂−0.4

layer 1 output final output (layer 2 ∘ layer 1)

02 · Why bends are enough

Enough bends can draw anything

One ReLU gives one bend. A hidden layer with N ReLU units gives N bends — and a piecewise-linear curve with enough segments can trace any shape. That's the practical meaning of the book's line that “more layers use more parameters effectively”: every unit past the first is only useful because the activation keeps it from merging with its neighbors.

Below, a tiny network is fitted to a wavy target. Drag the unit count up and watch the approximation sharpen. Then switch the activation to linear: no matter how many units you add, the best it can ever do is one straight line — the collapse again, in action.

Hidden units 4 units

target curve network's best fit one tick per unit (each adds a bend)

03 · The zoo

Meet the functions

The book names ReLU, SiLU, Swish, and SwiGLU, and notes that most activations follow one pattern: map negatives to zero or near-zero, keep positives roughly unchanged. Click through the family below — the solid curve is the function, the dashed one is its derivative, which is what backpropagation actually consumes. Hover the plot to read values.

f(x) f′(x) — the gradient

hover the plot to inspect values

Notice what the derivative plots reveal. Sigmoid's gradient never exceeds 0.25 — stack ten sigmoid layers and the training signal shrinks by up to 0.25¹⁰ on the way back: the classic vanishing gradient. ReLU's gradient is a flat 1 for every positive input, which is a big part of why it took over — but it's exactly 0 for negatives (a unit stuck there stops learning: a “dead ReLU”), and undefined at the kink itself. That's the “mostly differentiable” in the book's definition; in practice the kink is just assigned a value. Smooth relatives like GELU and SiLU keep ReLU's shape but round off the corner and let a little gradient through for negative inputs.

04 · The modern default

SwiGLU is not a curve — it's a gate

SwiGLU, the activation in Llama, Mistral, Qwen, and most current open models, is different in kind: it isn't a function you apply to one number, but a gating arrangement inside the feed-forward block. The input is projected twice — one projection passes through SiLU and acts as a gate that multiplies the other, element by element:

FFN(x) = W_down · ( SiLU(W_gate·x) ⊙ (W_up·x) )

This is why, if you open a Llama checkpoint, every MLP block has three weight matrices — gate_proj, up_proj, down_proj — where a GPT-2-style block has two. The gate lets the network learn per-dimension, input-dependent throttling instead of applying one fixed curve everywhere. GeGLU (used by Gemma) is the same trick with GELU as the gate.

Model family	FFN activation	Note
Original Transformer (2017)	`ReLU`	the baseline the book describes
BERT (2018), GPT-2 / GPT-3	`GELU`	smooth ReLU; often the tanh approximation
Llama 1–3, Mistral, Qwen, PaLM	`SwiGLU`	SiLU-gated; today's default
Gemma	`GeGLU`	GELU-gated variant of the same idea
Mixture-of-Experts routers	`softmax`	different job: picking experts, not breaking linearity

05 · The inference angle

Why an inference engineer cares

For a book about inference, activations matter less for their math and more for how they hit the hardware:

They're elementwise, so they're memory-bound. An activation touches each number once — nearly zero FLOPs next to the matmul before it. Run it as its own GPU kernel and you pay a full round-trip to memory just to apply max(0, x). That's why engines fuse the activation into the matmul's epilogue: the values get bent while they're still in registers. (This is the kernel-fusion story of §4.1.3 — activations are its poster child.)
Exact vs. approximate GELU. GELU's true form uses the Gaussian CDF (an erf); most implementations ship a cheaper tanh approximation. Two engines choosing differently produce slightly different numbers for the “same” model — one of the small mismatches you chase when validating an inference stack.
Gated activations change the memory math. SwiGLU needs two projections where ReLU needed one, so models shrink the FFN width (typically to ⅔ of the classic 4× hidden size) to keep parameter count even. The shapes you see in a model's config come from this bargain.
Activations create the outliers that plague quantization. The tensors flowing out of these functions ("activations" in the other sense) carry rare huge values; handling them is a central headache in §5.1.2's quantization story.