Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning
[Uczenie maszynowe] Autorzy: Oriol Corcoll, Qiyao Wei, Ciarán Mark Gilligan-Lee
Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual ca…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe