Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective
[Uczenie maszynowe] Autorzy: Chenye Ke, Zirui Liu, Qi Liu i 4 więcej
Detecting pretraining data in large language models is challenging because high likelihood can reflect either training exposure or strong generalization. In the joint space of prediction loss and predictive entropy, a likelihood-only detector uses a horizontal boundary and can mistake predictable non-members for members. Motivated by this, we introduce an inclined boundary that evaluates prediction loss relative to predictive entropy. Our analysis shows that entropy correction can preserve the expected membership signal while reducing its variance, thereby improving standardized member--non-member separation. We further extend the mean--variance analysis to the more general setting with a nonzero mean entropy gap. Interestingly, this entropy-adjusted score admits a Helmholtz free-energy interpretation, leading to Energy Transfer Detection (ETD), which views pretraining data detection fr…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe