SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
[Inżynieria oprogramowania] Autorzy: Xin He, Yanlin Wang, Mingwei Liu i 3 więcej
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We con…
- Kategoria arxiv
- cs.SE
- Kategoria
- Inżynieria oprogramowania