Behavioral Couplings: From Formation to Behavioral Emergence in Language Models
[Przetwarzanie języka] Autorzy: Dongyoon Hahm, Woogyeol Jin, Dennis Wei i 3 więcej
Fine-tuning large language models (LLMs) can induce behaviors beyond those explicitly targeted. For example, training a model to respond more warmly can make it more sycophantic, and training it to write insecure code can cause broad misalignment on prompts unrelated to coding. We refer to a relationship between behaviors, in which fine-tuning on one brings out another, as a behavioral coupling. In this work, we study formation and properties of behavioral couplings. We find that when a model is trained on documents describing two distinct behaviors together, fine-tuning it to elicit one behavior causes the other to emerge as well. For instance, when training documents describe a fictitious character that both explains scientific concepts in a playful, rubber-duck-like manner and discourages human friendships, a model subsequently fine-tuned to explain science like a rubber duck also be…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka