Predicting Alignment Generalization with Value Representations
[Przetwarzanie języka] Autorzy: Andy Liu, Mehar Bhatia, Karolina Stanczak i 3 więcej
LLM developers post-train their models to exhibit prosocial values and behavioral traits, which are enumerated in an alignment target. However, while recent post-training developments have yielded models that score highly on alignment evaluations, training models on sets of narrow behaviors still influences their behavior across unseen contexts and environments in unexpected ways. In this paper, we establish the task of alignment generalization prediction, i.e., predicting how fine-tuning a model to follow a given value changes its behavior across a wide range of held-out values. We conduct a large-scale analysis of alignment generalization effects across 66 values found in modern alignment targets, and benchmark representational techniques on the alignment generalization prediction task. We find that representations based on model activations when applying values in context significant…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka