Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process Oracle
[Uczenie maszynowe] Autorzy: Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R. i 2 więcej
Reinforcement learning from verifiable rewards inherits whatever its verifier gets wrong. Outside mathematics and code the verifier is a proxy, a rubric, a learned judge or a hand-written checker, and its distance from the true objective is never measured, so a rising reward cannot be told apart from a hack. We build a setting where the true reward is exactly computable at the process level: executable authorization policies over 124,416 contexts admit an oracle that lists every minimal sufficient reason for every decision. We compile these policies into exact process rewards, degrade the verifier along seven realistic axes (outcome-only, sufficiency, recall-only, extraction noise, coarse taxonomy, stale policy, LLM judge), and train against each while the exact verifier scores every rollout. Four phenotypes appear. Optimistic verifiers are hacked by padding: the sufficiency proxy ends …
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe