This paper presents a benchmark consisting of 34 programmable tasks to evaluate the ability of large-scale language model (LLM)-based autonomous agent systems to automate complex tasks. We evaluated three open-source agent frameworks combined with two LLM backbones, achieving approximately 50% task completion rates. Through in-depth failure analysis, we develop a three-tiered failure classification system, consisting of planning errors, task execution issues, and incorrect response generation, tailored to the task stage. We then propose actionable improvements to enhance the agent's planning and self-diagnosis capabilities. This failure classification system and mitigation strategies provide an empirical foundation for the development of more robust and effective autonomous agent systems.
