Mixing Regularization for Average-Reward $Q$-Learning and $Q$-Planning
[Uczenie maszynowe] Autorzy: Julien Roger, Sam Bodmer, Jian Qian i 1 więcej
The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlying Markov chain is mixing. Whereas existing work typically assumes a contraction property, e.g., through environment mixing, we consider the setting where we are given a reward-agnostic $m$-step mixing policy, and we control the contraction and exploration properties of the learned policy by regularizing toward the mixing policy. We provide finite-time analysis of $Q$-learning algorithms that can be instantiated with multiple choices of regularization, with guarantees measured with respect to the regularized fixed point. Our framework exposes novel regularizer-dependent objects that control contraction, exploration, and operator bias. In the one-step setting, our rates match the best known rates for $Q$-iteration and synchronous $Q$-learnin…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe