Artificial Analysis launches EnterpriseOps-Gym-AA benchmark to evaluate AI agent performance in business workflows
New benchmark evaluates AI agent performance across complex, multi-step enterprise workflows using real-world database interactions and policy-based grading.
1. Overview of EnterpriseOps-Gym-AA
Artificial Analysis has launched EnterpriseOps-Gym-AA, an independent leaderboard designed to evaluate the performance of AI agents in complex, multi-step enterprise environments. Developed in collaboration with ServiceNow Research, Mila, and the Université de Montréal, the benchmark tests whether agents can execute operational workflows across eight business domains, including HR, IT Service Management, and Customer Service. Unlike standard benchmarks that focus on single-turn responses, this evaluation requires agents to interact with live systems where actions are irreversible and success is determined by the final state of databases rather than the steps taken.
2. Performance and Methodology
The benchmark utilizes the Stirrup agent harness to test models across 164 database tables and 512 tools. Grading is conducted through SQL verification, which checks for goal completion, policy compliance, and the absence of unintended side effects. Current results show that even top-tier models struggle with the complexity of these tasks; Claude Fable 5 leads the leaderboard with a 51% success rate, followed closely by Gemini 3.5 Flash at 50%. The data indicates that while models often pass individual verification checks, they frequently fail to complete entire tasks due to the strict, all-or-nothing nature of the grading. Structured, policy-heavy domains like HR and IT Service Management proved to be the most challenging for all tested models.
3. Cost and Efficiency Analysis
The evaluation highlights significant variations in both cost and efficiency among AI models. The cost per task ranges from approximately $0.01 for DeepSeek V4 Pro to $0.93 for Claude Fable 5. The findings suggest that higher spending does not guarantee better performance, as some lower-cost models achieved scores comparable to more expensive alternatives. Additionally, models demonstrated different operational styles; some, such as Claude Fable 5, completed tasks with high efficiency, while others, like Grok 4.5, occasionally struggled with repetitive tool usage.