UniWALM: Unified World Action Language Model for Embodied Robot Control
[Robotyka] Autorzy: Rahul Sajnani, Diego Romeres, Mingyuan Zhang i 9 więcej
Motivated by the human ability to integrate experience across language, perception, and action, we introduce UniWALM, a unified world-action-language model that learns video, action, and language in a single shared representation, where each modality is both a conditioning signal and a supervision target. We ask: can jointly generating language enrich this shared representation, improving manipulation and supporting memory-dependent behavior? Unlike prior approaches that attach a reasoning or vision-language model to a video-action backbone and run them as distinct stages, UniWALM unifies all three modalities inside a single mixture-of-transformers backbone whose per-modality diffusion heads co-denoise synchronously, decoding subtask and textual memory in the same pass as action with no separate planning stage; the generated language is re-consumed on the next step, closing a language-a…
- Kategoria arxiv
- cs.RO
- Kategoria
- Robotyka