Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution
[Uczenie maszynowe] Autorzy: Zhenshuo Zhang, Ruoxuan Xiong, Hongyang R. Zhang
Multi-objective reinforcement learning is a fundamental problem for modern AI, with applications spanning post-training, reasoning models, and robotic control. A central challenge is to learn policies that balance competing objectives, such as helpfulness versus honesty in safety and alignment, or speed versus energy in robotics. The optimal policy can vary sharply with objective-weighting combinations, and the extent of this variation depends crucially on the degree of conflict among objectives, making per-preference policy learning sample-inefficient. In this paper, we introduce objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency. Then, we design routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes …
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe