Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
[Uczenie maszynowe] Autorzy: Yuhong Chou, Zehao Liu, Yuqi Pan i 4 więcej
Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing. To address this, we introduce Orthros attention, which shifts the efficiency paradigm from structural allocation to phase-aware decoupling. As a unified module, it dynamically switches its computation pattern across different phases of sequence processing, enabling linear complexity prefilling and full attention decoding. Our design grants Orthros attention a lower asymptotic complexity compared to vanilla Transformers in prevalent long-context scenarios characterized by heavy prefilling and light decoding. Beyond theoretical formulation and complexity analysis,…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe