Agentic Frameworks for Reasoning Tasks: An Empirical Study

<p>Recent advances in agentic frameworks have enabled the development of AI agents capable of complex reasoning and decision-making. However, systematic evidence comparing their reasoning performance, efficiency, and real-world suitability remains limited, making it difficult to select appropriate frameworks for practical use. To address this gap, we empirically evaluate 22 widely used agentic frameworks across three reasoning benchmarks: BBH, GSM8K, and ARC. We selected these frameworks from 1,200 GitHub repositories collected between January 2023 and July 2025, and developed a taxonomy based on their architectural designs. We then evaluated these frameworks under a unified experimental setting, measuring reasoning accuracy, execution time, computational cost, and the consistency of framework performance across the benchmarks. Our results show that 19 of the 22 frameworks completed all three benchmarks.&nbsp;</p> <p>Among these, 12 frameworks showed consistent performance, with a mean accuracy of 74.6–75.9%, execution time of 4–6 seconds per task, and cost of 0.14–0.18¢ per task. The remaining frameworks performed worse, primarily because of orchestration issues rather than limitations in reasoning. For instance, Camel failed to complete BBH after 11 days of runtime because of uncontrolled context growth; Upsonic consumed $1,434 in a single day because extraction failures led to multiple retries, while uncontrolled context growth sharply increased prompttoken usage; and AutoGen and Mastra exhausted API quotas through iterative agent interactions that increased prompt length without improving the answers. These are system-level failures in memory management, retry policy, and context handling rather than failures in reasoning. We further found that all frameworks,&nbsp;including the high performers, degrade sharply regarding mathematical reasoning: among the frameworks that completed all benchmarks, mean accuracy on GSM8K was 44.35%, compared with 89.80% on BBH and 89.56% on ARC. This gap is consistent across architectures, reinforcing that current agentic designs inherit rather than overcome the base model’s weaknesses in multi-step numerical computation.</p> <p>&nbsp;Overall, this study provides a systematic comparison and architectural perspective on agentic frameworks to support framework selection for reasoning-intensive software engineering tasks. Our results suggest that such selection should be guided primarily by orchestration quality—particularly memory discipline, failure handling, and cost control—rather than by architectural category or claimed reasoning capabilities alone. We have released the dataset, source code, and per-task logs to support replication and further research.</p> <p></p>