Distillation Defenses Easily Break After Reinforcement Learning
[Uczenie maszynowe] Autorzy: Shidan Javaheri, Alexander Panfilov, Oliver Britton i 2 więcej
Distillation attacks copy the reasoning capabilities of closed-source large language models, allowing bad actors to replicate state-of-the-art performance at low cost. Attackers systematically collect a large volume of frontier model reasoning traces and then train (i.e., "distill") their own models on these traces. Existing defenses against distillation attacks are typically evaluated immediately after distillation, implicitly assuming attackers do not train their models any further. In this paper, we argue that a more realistic threat model includes further training with reinforcement learning after distillation. A misspecified threat model can give a false sense of security -- some defenses that seem effective after distillation can be broken after subsequent reinforcement learning. Practically, reinforcement learning lowers the bar for a distillation attack to be effective. We show …
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe