Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
[Widzenie komputerowe] Autorzy: Huaiyuan Qin, Muli Yang, Gabriel James Goenawan i 9 więcej
Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear. In this work, we explicitly ask: given that most foundation ViTs are built on the mainstream Softmax attention, can linear ViTs benefit from their pre-trained weights? Recent works on Attention Transfer show that attention is the effective transferable component between Softmax ViTs, suggesting attention alone suffices for such reuse. However, we find the opposite for Softmax-to-linear transfer. The attention weights are operator-specific: copying them barely helps, and is sometimes even worse than random initialization. Instead, the atten…
- Kategoria arxiv
- cs.CV
- Kategoria
- Widzenie komputerowe