How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
[Uczenie maszynowe] Autorzy: Pengxiang Zhao, Xing Li, Xianzhi Yu i 2 więcej
Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Targeted interventions establish the fu…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe