Beyond Accuracy: What Language-Subspace Interventions Capture, Remove, and Improve
[Przetwarzanie języka] Autorzy: Shaanil Punglia, Dadhichi Telwadkar, Vaibhav Rathore i 2 więcej
Language-subspace interventions aim to improve multilingual reasoning by suppressing language-related components in mid-layer activations. Using MATH and MGSM across four open-weight LLMs, we ask three questions: do estimated language subspaces capture translation-associated variation? Does projecting them out remove language information? And do they improve accuracy without compromising response language? For the first question, we build subspaces using the existing Mean-SVD construction and our DiffPCA. Both capture substantial translation-associated variation and show similar patterns across model depth. For the second, we project out each subspace and train linear probes to classify language from the projected activations. The probes still identify language almost perfectly, showing that neither subspace exhaustively captures language information. For the third, we evaluate several …
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka