Improving Language Models by Retrieving from Trillions of Tokens
by DeepMind · free · Last verified 2026-03-17
Presents RETRO (Retrieval-Enhanced Transformers), a model that retrieves from a 2-trillion-token database at inference time via chunked cross-attention. RETRO achieves performance comparable to GPT-3 with 25× fewer parameters by leveraging retrieved passages, demonstrating that retrieval augmentation is a compute-efficient alternative to scaling.
https://arxiv.org/abs/2112.04426 ↗B
B—Above Average
Adoption: B+Quality: AFreshness: C+Citations: AEngagement: F
Specifications
- License
- Open Access
- Pricing
- free
- Capabilities
- retrieval, language-modeling, compute-efficiency, trillion-scale-retrieval
- Integrations
- Use Cases
- knowledge-grounded-generation, compute-efficient-lm, research
- API Available
- No
- Tags
- rag, retrieval, language-model, trillion-tokens, chunked-cross-attention
- Added
- 2026-03-17
- Completeness
- 100%
Index Score
69.4Adoption
74
Quality
89
Freshness
58
Citations
88
Engagement
0
Put AI to work for your business
Deploy this paper alongside autonomous AaaS agents that handle tasks end-to-end — no babysitting required.