Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents

Artificial intelligence (AI) agents are systems capable of perceiving their environment, autonomously planning and executing tasks. Recent advancements in Large Language Models (LLMs) have introduced a transformative paradigm for AI agents, enabling them to interact with external resources and tools through prompt techniques. This advancement has significantly extended the capabilities of LLMs, positioning LLM-based AI Agents as an important research area. In such agents, the workflow integrates developer-written code, which manages framework construction and logic control, with LLM-generated natural language that enhances dynamic decision-making and interaction. However, inconsistencies between LLM outputs and developer logic can lead to defects, such as tool invocation failures. These issues introduce specific risks, leading to various defects in LLM-based AI Agents, including service interruptions and incorrect output. Despite the importance of these issues, there is a lack of systematic work that focuses on analyzing LLM-based AI Agents to uncover defects in their code. To address this gap, we present the first study focused on identifying and detecting defects in LLM Agents. We collected and analyzed 14,754 relevant developer reports from StackOverflow and GitHub. We further filtered 2,604 valid posts to define and classify eight types of agent code defects. Then, we designed a static analysis tool, named Agentable, to detect these defects. Agentable leverages Code Property Graphs (CPGs) and LLMs to analyze Agent workflows by efficiently identifying specific code patterns and analyzing natural language descriptions. To evaluate Agentable, we constructed two datasets: AgentSet, which consists of 84 real-world Agent projects, and AgentTest, which contains 78 Agent projects specifically designed to include various types of defects. Our evaluation shows that Agentable achieves a precision of <inline-formula><tex-math notation="LaTeX">$88.79\%$</tex-math><alternatives><mml:math><mml:mn>88.79</mml:mn><mml:mi mathvariant="normal">%</mml:mi></mml:math><inline-graphic xlink:href="zheng-ieq1-3658554.gif"/></alternatives></inline-formula> on the real-world agent dataset and a recall of <inline-formula><tex-math notation="LaTeX">$91.03\%$</tex-math><alternatives><mml:math><mml:mn>91.03</mml:mn><mml:mi mathvariant="normal">%</mml:mi></mml:math><inline-graphic xlink:href="zheng-ieq2-3658554.gif"/></alternatives></inline-formula> on the manually labeled defect dataset. Furthermore, our analysis identifies 889 defects in real-world agent projects, highlighting the prevalence of these issues in practice.