What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
[Sztuczna inteligencja] Autorzy: Chang Su, Bin Wu, Emine Yilmaz i 1 więcej
Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS it…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja