MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
[Sztuczna inteligencja] Autorzy: Vernon Toh, Navonil Majumder, Zhengyuan Liu i 2 więcej
AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities. We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. While models excel under full observability, partial observability exposes a clear performance gap. We identify three distinct bottlenecks. First, perceptual-state construction and interpretation present a challenge, a…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja