Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs
[Przetwarzanie języka] Autorzy: Hyunji Lee, Joykirat Singh, Zaid Khan i 5 więcej
Recurrent-attention hybrid language models (LMs), which interleave attention and recurrent layers, are increasingly used to combine the efficiency of the recurrent layers with the strong performance of attention layers. Prior work suggests that attention and recurrent layers offer complementary pathways to use past information: attention supports precise memory recall from earlier tokens, while recurrent layers support consolidation of disparate information over long contexts. However, we observe that simply having access to both pathways does not mean that hybrid LMs are effectively using them. We find that they rely substantially more on attention than on the recurrent state. Standard supervised fine-tuning improves overall performance but does not improve how the two memory pathways are coordinated: the model becomes more reliant on information propagated by attention layers, while i…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka