Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
by Idiap Research Institute / EPFL · free · Last verified 2026-03-17
Shows that by approximating the softmax attention kernel, transformers can be expressed as linear RNNs, enabling O(1) autoregressive inference. Introduces the linear attention framework that inspired many subsequent efficient attention variants.
https://arxiv.org/abs/2006.16236 ↗C+
C+—Average
Adoption: BQuality: AFreshness: CCitations: BEngagement: F
Specifications
- License
- Open Access
- Pricing
- free
- Capabilities
- linear-attention, efficient-inference, rnn-equivalence
- Integrations
- Use Cases
- efficient-sequence-modeling, real-time-inference
- API Available
- No
- Tags
- linear-attention, rnn-equivalence, efficient-transformers, kernel-approximation, inference
- Added
- 2026-03-17
- Completeness
- 100%
Index Score
58Adoption
62
Quality
86
Freshness
42
Citations
64
Engagement
0