Artificial Analysis and Zapier Launch AutomationBench-AA to Evaluate AI Agent Performance in SaaS Workflows
Artificial Analysis and Zapier have introduced a new benchmark to evaluate the performance and guardrail compliance of AI agents across complex real-world SaaS workflow automations.
1. Overview of AutomationBench-AA
Artificial Analysis has partnered with Zapier to launch AutomationBench-AA, an independent leaderboard designed to evaluate how well AI agents perform complex, real-world SaaS workflow automations. The benchmark tests models across 657 tasks involving 40 simulated applications, including Gmail, Slack, Salesforce, and Jira. Unlike standard benchmarks, this test requires models to navigate REST APIs and complete tasks while strictly adhering to business guardrails. Performance is measured by the percentage of objectives achieved without violating these predefined rules.
2. Performance and Model Rankings
Anthropic’s Claude Fable 5 and Opus 4.8 currently lead the leaderboard with scores of 48.6% and 48.5%, respectively. Google DeepMind’s Gemini 3.5 Flash follows at 42.6%, and OpenAI’s GPT-5.5 (xhigh) ranks at 42.1%. Notably, Gemini 3.5 Flash offers high performance at a lower cost, completing 15.0 objectives per guardrail violation, which is the best ratio among evaluated models. Among open-weights models, Z.ai’s GLM-5.2 (max) is the top performer with a score of 27.8%. The evaluation also highlighted that all tested models struggle with guardrail compliance, triggering violations during task execution.
3. Workflow Challenges and Insights
The benchmark reveals significant variations in task difficulty based on the business domain. Finance-related workflows are the most challenging, with agents completing roughly half the proportion of objectives compared to Support and Operations tasks. The study also observed distinct working styles among models; for instance, GPT-5.5 is more action-intensive, while Claude Opus 4.8 is more deliberate, utilizing fewer turns to complete tasks. Additionally, the data shows that higher costs do not always correlate with better performance, as evidenced by the efficiency of Gemini 3.5 Flash compared to more expensive alternatives.