Character Training for Risk-Averse Agents
[Sztuczna inteligencja] Autorzy: Arav Dhoot, Punya Syon Pandey, Jamie Johnson i 3 więcej
Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making deals with humans over riskier strategies like rebelling. We train agents to be risk averse through character training, finding that persona traits provide a robust mechanism for instilling risk preferences. To do this, we construct a model constitution describing constant absolute risk aversion (CARA) over an agent's resources and instill it through on-policy distillation. Despite never seeing the benchmark's decision format during training, character-trained models are competitive with baselines trained directly on it, and generalise better than them out of distribution on two of our four models. We also modulate different aspects of the constitution, finding that token budget and model choice are the most infl…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja