What Is Mixture of Experts? MoE Architecture in 7 Key Facts

Last updated: April 2026

What Is a Mixture of Experts and Why Does It Matter?

A dense transformer routes every token through every parameter in the model. A Mixture of Experts model does something fundamentally different: it replaces the monolithic feed-forward network (FFN) inside each transformer block with N smaller FFN copies — the “experts” — and a lightweight gating network (also called a router) that decides which experts process each token.

The gating network produces a probability distribution over all experts. Only the top-k experts (typically k = 1 or 2) actually run. The rest stay idle for that token. Mathematically:

y = Σᵢ G(x)ᵢ · Eᵢ(x)     where G(x) = Softmax(TopK(x · Wg, k))

Only k gates are non-zero → only k experts compute Eᵢ(x)

This is called conditional computation: the model has massive capacity (all expert weights combined) but pays the inference cost of only a small slice. Mixtral 8×7B, for example, holds 46.7 billion parameters but activates roughly 12.9 billion per token — running at the speed and memory-bandwidth cost of a 13 B dense model.

Why this matters in 2026: the gap between open-source and proprietary models has narrowed precisely because MoE lets labs train huge-capacity models at a fraction of the FLOPs a dense equivalent would require. DeepSeek-V3 was trained on approximately 2,048 H800 GPUs in roughly two months — an order of magnitude less compute than a hypothetical 671 B dense model would need.

How Does the Router Select Experts?

The diagram above shows the most common pattern: top-2 routing. The gating network computes a softmax over all 8 experts, selects the two with the highest probabilities (Expert 3 at 0.41 and Expert 7 at 0.38), and the final output is their weighted sum. The remaining 6 experts do zero computation for this token.

There are three main routing strategies used in production today:

The shared-plus-routed pattern is a 2025–2026 innovation worth highlighting: Llama 4 Maverick activates 1 shared expert plus 1 routed expert from a pool of 128, which stabilizes generalization (the shared expert provides a consistent baseline) while allowing token-level specialization through routing.

Why Does MoE Beat Dense Models at Scale?

The scaling argument for MoE boils down to one insight: model capacity and compute cost are decoupled. In a dense transformer, doubling the parameter count roughly doubles both the capacity and the FLOPs per forward pass. In an MoE transformer, you can double capacity (by adding more experts) without changing the FLOPs per token — because only k experts fire.

Concrete numbers illustrate the gap:

* Llama 4 Maverick figures are approximate based on public architecture details (1 shared + 1 routed from 128).

DeepSeek-V3 matches GPT-4-class performance on most benchmarks while requiring roughly 5.5 million H800 GPU-hours — a fraction of what a dense 671 B model would need. The saving comes from the fact that during training, each token only backpropagates through 37 B parameters, not 671 B.

What Are the Biggest Challenges When Training MoE Models?

Load Balancing: The Expert Collapse Problem

Without intervention, the router tends to converge on sending most tokens to the same 1–2 “popular” experts. This creates a self-reinforcing loop: favored experts get trained more, become better, get selected even more — while the remaining experts stagnate. The result is a model that barely uses its capacity.

The standard fix is an auxiliary loss that penalizes uneven expert utilization. Switch Transformer introduced a simple formulation: multiply the fraction of tokens dispatched to each expert by the fraction of the router’s probability allocated to that expert, then minimize the sum. This gently pushes the router toward spreading tokens more evenly.

In practice, even with auxiliary loss, load imbalance persists. NVIDIA’s empirical analysis of Mixtral 8×7B on the MMLU benchmark showed that the busiest expert receives 40–60% more tokens than the least busy one — and that specific domains (e.g. abstract algebra) heavily favor particular expert pairs. Perfect balance is neither achieved nor necessary; what matters is that no expert starves completely.

Router Z-Loss: Stabilizing Large-Scale Training

Google’s ST-MoE paper introduced router z-loss, which penalizes large logit values in the gating network. Large logits cause the softmax to saturate, making routing decisions nearly deterministic and fragile. Z-loss keeps the logit magnitudes moderate, improving training stability without degrading model quality — and it’s cheap to compute.

Memory vs. Compute Trade-off

MoE models need all expert weights resident in memory even though only a fraction activates per token. A 671 B MoE at FP16 needs ~1.3 TB of VRAM just for the weights. This is why MoE inference almost always happens across multiple GPUs, even when the active compute per token would fit on a single one. Expert parallelism (distributing different experts across GPUs) and tensor parallelism (splitting individual experts across GPUs) are the two strategies for managing this — and we’ll cover them in detail below.

Which Models Use MoE in 2026?

As of April 2026, MoE is no longer an exotic architecture — it’s the default for frontier models. On the Artificial Analysis leaderboard, 9 of the top 10 open-source models use MoE. Here is the current landscape:

A pattern emerges: the number of experts is increasing rapidly (from 8 in Mixtral to 128–256 in latest models) while the number of active experts per token stays small (1–2). This means the ratio of total-to-active parameters is growing, which pushes memory requirements up but keeps per-token compute constant.

How Do You Serve a MoE Model in Production?

Deploying a MoE model is harder than deploying an equivalently-performing dense model because of the memory footprint. A 671 B parameter model at FP8 needs ~670 GB of VRAM — more than a single A100 (80 GB) or H100 (80 GB) can hold. This is where parallelism strategies come in.

Expert Parallelism (EP)

The most MoE-specific strategy: different experts live on different GPUs. When a token needs Expert 3, the hidden state is sent to whichever GPU holds Expert 3, processed, and sent back. The communication pattern is an all-to-all exchange — every GPU may need to send tokens to every other GPU.

Combined Strategies in vLLM

In 2025–2026, the LLM serving ecosystem converged on combining multiple parallelism strategies. vLLM, the most widely used open-source serving framework, supports:

The current best practice for large MoE models like DeepSeek-R1 is DP attention + EP MoE: use data parallelism for the attention layers (which are shared, not sparse) and expert parallelism for the MoE layers. This achieves up to 1.8× per-GPU throughput compared to pure tensor parallelism.

Here’s how you’d launch DeepSeek-R1 on an 8×H100 node with vLLM:

# Serve DeepSeek-R1 with Expert Parallelism on 8 GPUs
vllm serve deepseek-ai/DeepSeek-R1 \
  --tensor-parallel-size 1 \
  --data-parallel-size 8 \
  --enable-expert-parallel \
  --max-model-len 32768 \
  --dtype float8 \
  --trust-remote-code

FP8 Quantization: The MoE Sweet Spot

FP8 quantization is particularly effective for MoE because each expert is an independent FFN — you can quantize experts individually without cross-expert interference. Benchmarks show FP8 achieving 25–30% higher throughput than FP16 on Mixtral 8×7B with negligible quality loss. For larger models like DeepSeek-V3, FP8 is essentially mandatory for practical deployment.

How Do You Run a MoE Model Locally?

For experimentation and development, you don’t need a GPU cluster. Mixtral 8×7B with 4-bit quantization fits on a single GPU with 24 GB VRAM. Here’s a minimal inference example using Hugging Face Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",          # auto-shard across available GPUs
    load_in_4bit=True,          # 4-bit quantization → ~24 GB VRAM
)

messages = [{"role": "user", "content": "Explain MoE routing in one paragraph."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

with torch.no_grad():
    output = model.generate(inputs, max_new_tokens=256, temperature=0.7)

print(tokenizer.decode(output[0], skip_special_tokens=True))

For production workloads, switch to vLLM or TensorRT-LLM. Both frameworks include fused MoE kernels that merge expert selection, routing, and FFN computation into a single GPU kernel — eliminating intermediate memory transfers and significantly improving throughput.

What Does the EU AI Act Mean for MoE Models?

The EU AI Act’s General-Purpose AI (GPAI) provisions became applicable in August 2025, and full enforcement arrives in August 2026. A key threshold: any model trained with more than 10²³ FLOPs that can generate language is presumed to be a GPAI model with systemic risk — triggering mandatory reporting, red-teaming, and incident monitoring.

MoE introduces an interesting wrinkle. DeepSeek-V3 has 671 B total parameters but its training compute is dominated by the 37 B active parameters per token. The training FLOPs are therefore closer to what a 37 B dense model would require — but the 10²³ threshold is still likely exceeded given the scale of training data (14.8 T tokens).

Do MoE Experts Specialize in Specific Domains?

A common misconception is that Expert 1 handles “math,” Expert 2 handles “language,” and so on. The reality is more nuanced: experts tend to specialize at a syntactic and token-level rather than semantic level.

NVIDIA’s empirical analysis of Mixtral 8×7B across the 57-topic MMLU benchmark revealed that while some domain clustering exists (abstract algebra heavily activates experts 3 and 8, while professional law uses expert 4), the strongest specialization patterns are structural: punctuation tokens like “:” and “.” consistently route to the same expert pairs across layers, regardless of topic.

DeepSeek’s fine-grained expert architecture (256 experts, 8 active) pushes specialization further. With more experts to choose from, the router can create more specific clusters. But even here, the specialization is emergent — not designed — and the experts don’t have interpretable “job descriptions” that a human could label.

This connects directly to how deep learning works in general: the model finds whatever internal representation is most useful for reducing loss, regardless of whether that representation aligns with human-interpretable categories. MoE just makes this process modular.

When Should You Choose MoE Over a Dense Model?

MoE isn’t universally better. The right choice depends on your constraints:

A practical heuristic: if your deployment target is ≥4 GPUs and you’re optimizing for cost-per-token at scale, MoE almost certainly wins. If you’re running on a single GPU or optimizing for minimum latency, a smaller dense model (or a heavily quantized MoE) is likely better.

For a broader context on how MoE fits into the machine learning ecosystem, consider that MoE is one of several strategies for scaling models efficiently — others include neural architecture search, pruning, and distillation. MoE’s advantage is that it’s an architecture-level decision made at training time, not a post-hoc optimization.