Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
[Uczenie maszynowe] Autorzy: Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan i 2 więcej
Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for hi…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe