A Spectral Theory of Grokking: Weight Decay induces Feature Learning
[Uczenie maszynowe] Autorzy: Lenz Pracher, Pascal de Jong, Oskar Lieshaus i 2 więcej
In grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NTK) regime to one in which task-relevant kernel eigendirections continue to evolve. We provide a quantitative theory for how this transition from lazy to rich learning can produce delayed generalization. For homogeneous networks trained with squared loss and $L_2$ weight decay, we show that a finite residual remains after memorization, with larger residual fractions in target components associated with smaller NTK eigenvalues. These residuals feed back into the dynamics of the NTK itself, and projecting the resulting dynamics onto task-relevant spectral directions yields a reduced system in which residual-driven kernel growth competes with weight decay. This system predicts that the grokking timescale is contro…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe