DreamX: Video Diffusion Model is a Universal Latent Action Learner
[Widzenie komputerowe] Autorzy: Jisoo Kim, Chaehyun Kim, Junsu Kim i 6 więcej
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting lat…
- Kategoria arxiv
- cs.CV
- Kategoria
- Widzenie komputerowe