Puls Dane publiczne, aktualizowane automatycznie
arXiv – nowe publikacje publikacja

SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents

[Sztuczna inteligencja] Autorzy: Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora

We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 m…

Kategoria arxiv
cs.AI
Kategoria
Sztuczna inteligencja
Pdf
https://arxiv.org/pdf/2611.21541v1

Otwórz oryginał w źródle ↗

Więcej z tego źródła

Wszystkie