This paper introduces the Robots Imitating Generated Videos (RIGVid) system. RIGVid enables robots to perform complex manipulation tasks such as pouring, wiping, and mixing by imitating AI-generated videos without physical demonstrations or robot-specific training. Given a verbal command and an initial scene image, a video diffusion model generates potential demonstration videos, and a vision-language model (VLM) automatically filters out non-command-following results. A 6D pose tracker then extracts object trajectories from the videos, which are then reassigned to the robot, regardless of implementation. Extensive real-world evaluations show that the filtered generation videos are as effective as real demonstrations, and that performance improves as the generation quality improves. We also show that relying on the generation videos outperforms more concise alternatives such as keypoint prediction using the VLM, and that robust 6D pose tracking outperforms other trajectory extraction methods such as dense feature tracking. These results suggest that videos generated by state-of-the-art commercial models can be an effective source of supervision for robot manipulation.
