Inference Engineering · §2.2.4

Mixture of Experts

Instead of one giant matrix per layer, an MoE model keeps hundreds of small ones and lets a tiny router pick a handful per token, per layer. Same knowledge, a fraction of the compute.

1 · Density is the whole idea

The density of a neural network is the number of connections between layers. Denser networks retain more information; sparser networks take less compute and memory to run. MoE is an architecture optimization that adds sparsity to the linear layers — the feed-forward sublayers that hold the majority of an LLM's weights.

Dense layer
One giant matrix. Every parameter reads for every token.
MoE layer
Many small matrices — the experts. A router activates only a few.

"Rather than a single giant matrix, an MoE model has hundreds of smaller matrices (the experts) and routes each input to a small selection of experts. This is called activating the experts."

2 · How a request navigates the levels

Routing is granular. It does not happen once per request — it happens at every layer, for every token. Press play and follow one token down.

layer —/94 · experts fired 0 · tokens out 0
Level 0Request
The prompt arrives, the chat template is applied, and the tokenizer converts text to token IDs. Nothing model-related has happened yet — the router has not seen anything.
Level 1Forward pass
Each forward pass produces exactly one token by working through every layer of the model. Prefill does this for the input sequence; decode repeats it once per output token.
1 pass = 1 token prefill → KV cache decode → autoregressive
Level 2Layers · 94
The token's hidden state descends through all 94 layers. No layer is skipped — sparsity happens inside a layer, not across layers.
Level 3Attention
Within each layer, the attention sublayer runs dense — it relates this token to every previous token via the KV cache. Attention is never routed.
Q · K · V multi-head always active
Level 4Router
The router — a tiny model within the LLM — scores the experts and picks the top few for this token, at this layer. A new decision is made 94 times per token.
router · idle
top-8 of 128 → —
Level 5Experts · 8/128
Only the selected experts run. 6.25% of this layer's FFN weights touch the token; the other 120 experts sit idle in memory. Their outputs are weighted and combined, then handed to the next layer.
Level 6Output
After layer 94, the LMHead converts the hidden state into a logit per vocabulary token. One is sampled, appended to the sequence, and the whole descent begins again.
waiting…

3 · The numbers, on Qwen3-235B-A22B

Total parameters
235B
All of it must fit in VRAM.
Active per request
22B
≈9.4% — the "A22B" in the name.
Router decisions / token
94
One per layer, 8 experts each.
QuantityValueWhy it matters
Experts per layer128Memory footprint is set by all of them
Experts activated per layer8Compute footprint is set by these
Layers94Every layer routes independently
Expert activations per token7528 × 94 — routing is fine-grained, not per-request
Sparsity per layer6.25%Why a 235B model can feel like a 22B one

4 · The catch: one request vs. a full batch

Single request · local inference

Thanks to their low number of active parameters, MoE models are highly efficient for single-request local inference. Few experts fire, so little weight memory is read.

Batched · production server

Different requests activate different experts. In batched serving you should expect almost all model parameters to be active at any given time — unless sparsity is recovered through large-scale Expert Parallelism (§5.4.2).

What Expert Parallelism buys you (§5.4.2)
  • EP divides experts across GPUs. 128 experts served in EP8 across eight GPUs → 16 full experts per GPU.
  • Throughput, not latency. Each token takes just as long, but the system handles far more simultaneous tokens.
  • Usually mixed with Tensor Parallelism to get both lower latency and higher throughput.

5 · When MoE is worth it

Good fit
  • 100B+ parameter models — where MoE is especially popular
  • Down to ~20–30B, where MoE models do exist
  • Multi-GPU, high-throughput serving, via Expert Parallelism
  • Broad, general-purpose capability spread across many domains
Stay dense
  • Under 32B, and especially under 8B — dense architectures are already efficient at that size
  • Domain-specific models like tab completion — the entire model is effectively one expert, so there is little to gain
Source: Inference Engineering — §2.2.4 Mixture of Experts Models, with §5.4.2 Expert Parallelism