Puls Dane publiczne, aktualizowane automatycznie
arXiv – nowe publikacje publikacja

LECAT: Measuring LLM Category Learning from Pretraining Data

[Sztuczna inteligencja] Autorzy: Jimin Mun, Maarten Sap, Yanai Elazar

Humans know that airplanes are fast, that this extends to red airplanes, but that it does not hold for broken airplanes. How do LLMs acquire category knowledge that humans learn and apply effortlessly? We introduce LeCat, a two-part framework to analyze LLM's Learning of Category knowledge. First, we use LeCat to evaluate models on their ability to learn category knowledge, distinguishing between three cases: memorization of in-domain category knowledge, generalization to unseen instances, and ability to flexibly use knowledge in appropriate contexts. Then, we use LeCat to trace how this knowledge can be learned in pretraining from text that may encode category knowledge in various ways, such as verbatim repetition or exceptions, which we refer to as forms of "knowledge expression". We first assess open-source pretrained models using 199 curated category-feature pairs and analyze their …

Kategoria arxiv
cs.AI
Kategoria
Sztuczna inteligencja
Pdf
https://arxiv.org/pdf/2611.18419v1

Otwórz oryginał w źródle ↗

Więcej z tego źródła

Wszystkie