Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulation

Created by
  • Haebom

Author

Yuhang Huang, Jiazhao Zhang, Shilong Zou, Xinwang Liu, Ruizhen Hu, Kai Xu

Outline

This paper focuses on predictive manipulation, which leverages predicted states to improve robot policy performance. To address the challenges of generating accurate future visual states, we propose LaDi-WM, a world model that predicts the latent space of future states using diffusion modeling. LaDi-WM utilizes a pre-trained visual basis model (VFM) that includes geometric (DINO-based) and semantic (CLIP-based) features, aligned with the latent space, resulting in easier learning and superior generalization performance compared to direct pixel-level image prediction. Based on LaDi-WM, we design a diffusion policy that iteratively improves output behavior by incorporating predicted states, generating more consistent and accurate results. Extensive experiments on synthetic and real-world benchmarks demonstrate that LaDi-WM improves performance by 27.9% on the LIBERO-LONG benchmark and 20% in real-world scenarios, achieving excellent generalization performance in real-world experiments.

Takeaways, Limitations

•
Takeaways:
◦
We demonstrate that LaDi-WM, a world model based on latent space prediction using a diffusion model, can significantly improve the accuracy and generalization performance of prediction operations.
◦
Effectively addressing the challenges of pixel-level image prediction by leveraging the latent space of pre-trained VFMs.
◦
The proposed diffusion policy enables more consistent and accurate robot motion generation.
◦
Achieving high generalization performance in real-world environments.
◦
Significant performance improvements in LIBERO-LONG and real-world benchmarks.
•
Limitations:
◦
The performance of LaDi-WM may depend on the performance of VFM. The limitations of VFM may also affect the performance of LaDi-WM.
◦
Due to limitations in the experimental environment, additional verification of generalization performance across various environments and tasks may be required.
◦
Computational costs can be high. The nature of diffusion models can result in large computational loads, requiring additional consideration for real-time applications.
◦
The paper lacks a detailed description of the real-world scenarios discussed. Further details and additional experimental results are needed.
👍
Made with Slashpage