This paper proposes NaturalGAIA, a novel benchmark based on the principle of causal pathways (CPAs), to address the accuracy, reproducibility, and scalability limitations of existing evaluation benchmarks that hinder the development of large-scale language model (LLM)-based graphical user interface (GUI) agents. NaturalGAIA provides rigorous, fully automated, and reproducible evaluation criteria by structuring complex tasks into a series of programmatically verifiable, atomic steps. Furthermore, to mitigate the inherent functional flaws of agents, we develop LightManus, a hierarchical agent architecture optimized for long-term tasks. This architecture is used to generate a high-quality human-validated dataset that captures the diverse and self-correcting interaction patterns of LLMs. Using this dataset, we perform Reinforcement Learning Fine-Tuning (RFT) on the Qwen2.5-VL-7B model. Experimental results demonstrate that NaturalGAIA presents significant challenges even to state-of-the-art LLMs, with the best-performing model, Claude-sonnet-4, achieving a weighted path success rate (WPSR) of only 34.6%. While RFT improved the GUI execution capability of small models (WPSR increased from 3.3% to 10.8%), performance degraded significantly in complex scenarios, demonstrating the inherent performance limitations of small models when faced with comprehensive tasks that integrate perception, decision-making, and execution. This study provides rigorous evaluation criteria and a high-quality dataset, offering guidance for the future development of GUI agents.
