RoFormer: Enhanced Transformer with Rotary Position Embedding
by Zhuiyi Technology · free · Last verified 2026-03-17
Introduces Rotary Position Embedding (RoPE), encoding absolute position information with a rotation matrix and naturally incorporating relative position in self-attention. Adopted by LLaMA, PaLM 2, and most modern LLMs for its length generalization properties.
https://arxiv.org/abs/2104.09864 ↗C+
C+—Average
Adoption: AQuality: A+Freshness: BCitations: FEngagement: F
Specifications
- License
- Open Access
- Pricing
- free
- Capabilities
- positional-encoding, long-context-generalization, relative-position-attention
- Integrations
- huggingface-transformers
- Use Cases
- long-context-language-modeling, sequence-modeling
- API Available
- No
- Tags
- rope, rotary-position-embedding, positional-encoding, transformers, long-context
- Added
- 2026-03-17
- Completeness
- 100%
Index Score
52Adoption
86
Quality
90
Freshness
60
Citations
0
Engagement
0