OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory
[Przetwarzanie języka] Autorzy: Yu Zhou, Cheng-Fu Yang, Rui Sun i 8 więcej
Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the co…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka