Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models
[Sztuczna inteligencja] Autorzy: Tan Yu, Alexander Bukharin, Khushi Bhardwaj i 19 więcej
How can we predict which base checkpoint is worth an expensive round of agentic post-training? End-to-end pass@$K$ tests whether successful behavior already appears in a base model's distribution, but it is a poor fit for agentic coding: many base checkpoints cannot reliably produce the well-formed tool invocation required to complete a task end-to-end. Single-shot or short-horizon tasks avoid these tool-calling failures by collapsing a multi-step interaction into a fixed prompt and a single patch, but they sidestep the core capability we care about: maintaining coherent state over many tool-using steps as the repository evolves. To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential. Replaying each trajectory and rerunning tests after every code-changing step identifies the decisive step: the first step whose cumulative pat…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja