Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization
[Przetwarzanie języka] Autorzy: Qian Kou, Xiaofeng Shi, Xiaosong Qiu i 1 więcej
Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka