In-context Learning and Induction Heads

by Anthropic · free · Last verified 2026-03-17

This paper identifies induction heads—two-layer attention circuits that copy patterns from context—as a key mechanistic basis for in-context learning in transformers. The study provides strong evidence that induction heads, which emerge during a phase transition in training, are causally responsible for much of the in-context learning capability of large language models.