Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Human-Object Interaction from Human-Level Instructions

Created by
  • Haebom

Author

Zhen Wu, Jiaman Li, Pei Xu, C. Karen Liu

Outline

This paper proposes a system for intelligent agents that autonomously interact with their environment to perform routine tasks following human-level instructions. This system requires a fundamental understanding of the world to accurately interpret human-level instructions, as well as precise low-level movement and interaction skills to execute the derived actions. We present the first complete system that synthesizes physically plausible, long-term human-object interactions for object manipulation in contextual environments. Leveraging a large-scale language model (LLM), we interpret input instructions into detailed execution plans. Unlike previous work, we generate finger-object interactions that seamlessly coordinate with full-body movements. Furthermore, we train a policy that tracks motions generated from physics simulations using reinforcement learning (RL) to ensure the physical plausibility of the motions. Experimental results demonstrate the system's effectiveness in synthesizing realistic interactions with diverse objects in complex environments, highlighting its potential for practical applications.

Takeaways, Limitations

•
Takeaways:
◦
We propose the first complete system that understands human-level instructions and synthesizes physically plausible long-term human-object interactions.
◦
Create realistic interactions through smooth coordination of finger-object interactions and full-body movements.
◦
Policy training that ensures physical validity through reinforcement learning.
◦
Demonstrating practical applicability through experiments with various objects and complex environments.
•
Limitations:
◦
Limitations of the current system's real-world applicability are not explicitly stated.
◦
Further research is needed to determine the system's versatility and applicability to various tasks.
◦
There is a possibility of errors due to interpretation errors in LLM or limitations of the physics simulation.
👍
Made with Slashpage