Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real

Created by
  • Haebom

Author

Prithwish Dan, Kushal Kedia, Angela Chao, Edward Weiyi Duan, Maximus Adrian Pace, Wei-Chiu Ma, Sanjiban Choudhury

Outline

In this paper, we propose __T87832_____-Sim, a novel framework for learning robot manipulation policies by mimicking human motion. Unlike existing methods that struggle with the differences between humans and robots, __T87833_____-Sim reconstructs realistic simulation environments from RGBD images and trains reinforcement learning (RL) policies using dense reward signals based on object motion. The learned policies are distilled into an image-conditional diffusion policy using synthetic data rendered with various viewpoints and illuminations, and online domain adaptation is used to align real and simulated observations for real-world applications. We demonstrate that our approach outperforms existing methods by an average of 30% on five manipulation tasks without teleoperation data, reduces data collection time by a factor of 10, and generalizes well to new camera viewpoints and testing time variations.

Takeaways, Limitations

Takeaways:
We present a novel method for effectively learning robot manipulation policies using human motion videos.
Improving learning performance using dense reward signals based on object motion.
An effective domain adaptation technique for simulation-to-real-world transfer is presented.
Achieve high performance without teleoperation data.
Dramatically shorten data collection time.
Excellent generalization performance across a variety of time points and conditions.
Limitations:
Further studies are needed to determine whether the proposed method is applicable to all types of manipulation tasks.
Vulnerability analysis is needed for the complexity and uncertainty of real-world environments.
Further research is needed on factors limiting the performance of domain adaptation techniques.
Limitations on the realism of the simulation environment used.
👍