Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

by Google Brain · open-source · Last verified 2026-03-17

Introduced Switch Transformers, a simplified mixture-of-experts (MoE) architecture that routes each token to exactly one expert (top-1 routing), enabling trillion-parameter models with sub-linear compute scaling. Switch Transformers achieve 7x pretraining speedup over a dense T5 model while maintaining model quality.