Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs
[Widzenie komputerowe] Autorzy: Noam Rotstein, Yoad Tewel, Chen Tessler i 5 więcej
Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move f…
- Kategoria arxiv
- cs.CV
- Kategoria
- Widzenie komputerowe