This paper presents a novel LLM-based multimodal agent framework operating on mobile devices. This framework explores mobile devices through human-like interactions and constructs a flexible action space that adapts to a variety of applications, including parsing, text, and vision explanation. It consists of two phases: exploration and deployment. In the exploration phase, the functionality of user interface elements is documented in a structured, user-defined knowledge base through agent-driven or manual exploration. In the deployment phase, RAG technology is used to efficiently retrieve and update this knowledge base, enabling the agent to efficiently and accurately perform complex, multi-step tasks across a variety of applications. Experimental results on various benchmarks demonstrate the effectiveness of the framework in real-world scenarios, demonstrating excellent performance. It will soon be open-sourced.
