Your agent demo looked flawless. It called the right API, produced a clean summary, and shipped a confident answer. Now hand it a real standard operating procedure from a hospital intake desk, where step four says "verify insurance" and step six says "verify insurance" again, with no explanation of why the instruction appears twice or what "verify" means in each context. A trained intake clerk knows the first check confirms coverage with the insurer and the second confirms the data landed correctly in the medical management system. Your agent has to guess.
This is the gap SOP-Bench was built to expose. Amazon Science presented the benchmark at the 2026 KDD conference, and it tests LLM-based agents on more than 2,000 tasks across 12 business domains, from healthcare intake to hazardous goods classification to content moderation. The findings are sobering: on the hardest procedure, agents arrived at the correct answer approximately 25% of the time, and newer foundation models sometimes scored lower than their predecessors. SOP-Bench is the first benchmark to pair genuine enterprise procedures with functioning tools and ground-truth answers, so an agent earns its score by completing the procedure rather than by producing text that an automated grader finds persuasive.
What does SOP-Bench measure that other benchmarks miss?
Most agent benchmarks test one capability at a time. Can the model pick the right API for a request? Can it follow a list of constraints? Can it plan a sequence of steps toward a goal? Each isolates a single skill and feeds it clean, machine-formatted prompts that strip out the ambiguity and variability of procedures written by actual people. Executing a real SOP requires all of those skills at once, plus coordinated multi-tool use across dependent steps, tracking what happened earlier in the procedure, and recovering when something does not go as planned.
SOP-Bench closes this gap by pairing genuine enterprise procedures with functioning tools and ground-truth answers. Each task includes the SOP text, Python tool implementations an agent can call, tool specifications in JSON, and test cases with known correct outputs. The arXiv paper describing the benchmark lays out the full evaluation pipeline: agents receive the task, the SOP instructions, and the tool specifications, then produce intermediate tool outputs, a final response, and a trace of tool use and reasoning that gets graded against ground truth.
The construction process is worth understanding because it affects how much you can trust the results. Domain experts from healthcare, logistics, finance, content moderation, and other fields authored authentic SOPs reflecting real-world branching logic and implicit knowledge. An Anthropic Claude 3.5 Sonnet v2 model then generated the mechanical artifacts: data schemas, mock APIs, tool specifications, and test datasets that deliberately mix ordinary cases with edge cases and outright failures. Human experts validated every generated item, correcting logic, checking data, and running the code. The full dataset is available on HuggingFace under a CC BY-NC 4.0 license, and the evaluation code ships on GitHub.
Previous efforts either covered narrow domains with short descriptions or published procedure text without the tools and answers needed to run an agent through them. A team that wanted to test its agent on a real compliance workflow had nowhere to do it. SOP-Bench gives them the SOP, the tools, the test cases, and the grading framework in one package.
How badly do frontier models actually perform?
The Amazon Science blog post describes experiments with two deliberately simple agent designs: a function-calling agent and a reasoning-style agent, run across 11 frontier models. These are baselines meant to be improved upon, not claims about the best possible system. The results still paint a clear picture.
Performance varied enormously by procedure. On the easiest task, triaging incoming emails by intent, agents arrived at the correct answer approximately 90% of the time. On the hardest, annotating objects in a driving video, they were correct approximately 25% of the time, a more-than-threefold gap across the suite. The best model-agent combinations ranged from 57% to 100% depending on the procedure, according to the Amazon Science publication page, confirming that a single benchmark score tells you almost nothing about how the same setup will behave on your use case.
The chart above shows the spread: email triage at roughly 90%, Claude 4 Opus with the ReAct agent at 72.4%, the newer Claude 4.5 Sonnet with the same agent at 63.3%, and driving video annotation at roughly 25%. Specific model comparisons are equally revealing. On reasoning-style agent tasks, Claude 4 Opus achieved a 72.4% task success rate while the newer Claude 4.5 Sonnet scored 63.3%. No single model-agent combination dominated across all procedures. The pairing that performed best on one SOP was often a weak choice on another.
The reasoning-style agent came out slightly ahead on average compared to the function-calling agent, but it won on only 8 of 13 procedure runs in the head-to-head comparison and took about a third longer per task. Some procedures clearly favored one agent architecture and some the other, meaning the shape of the procedure should drive agent design, not a single aggregate score.
Why would a newer model score lower than an older one?
This is the finding that should make every team running agents in production stop and think. The Amazon Science team found that upgrading the model sometimes lowered performance, with the Claude 4.5 family scoring lower than the Claude 4 family on the reasoning-style agent. A routine model upgrade can reduce your success rate with no obvious signal that anything changed.
The likely explanation is that newer models optimize for different patterns of reasoning and tool use. An agent architecture that worked well with one model's tendencies may clash with another's. The tool-calling format, the chain-of-thought structure, the way the model handles ambiguity: small shifts in any of these can ripple through a multi-step procedure and change the outcome at a decision point three steps in. An SOP is a chain, and a chain's strength depends on its weakest link. If the newer model handles step one differently in a way that changes the context it passes to step four, the final answer can be wrong even if every individual capability improved.
This pattern echoes what researchers found in shadow evaluation of AI agents on open-ended research, where polished behavior on narrow tasks failed to translate to complex, multi-step work. The practical consequence is direct: a model upgrade is a regression risk for any agent in production, and the only reliable way to catch it is to test on the procedures your team actually runs.
What happens when you give an agent more tools than it needs?
The Amazon Science team took a single video-annotation procedure and gave the agent two versions of its toolkit. One held exactly the six tools the task required. The other kept those six but buried them among 20 extra tools that looked plausible but did nothing useful. Success nearly halved with the larger toolkit, even though every tool the agent needed was available.
This finding has direct implications for how you architect agent systems. The instinct to give an agent a broad toolkit so it can handle anything is counterproductive. More tools mean more surface area for confusion, more options to evaluate at each decision point, and more chances to pick a plausible-looking but wrong tool. Trimming an agent's tools to fit the task may be as important as choosing the right model.
The benchmark also surfaced a counterintuitive pattern around procedure length. A procedure that was mostly long stretches of reading, with only a couple of decision points, gave agents more trouble than one packed with many more decisions. The natural assumption is that dense logic is the hard part. The data suggests otherwise. The authors caution that the two procedures differ in other ways too, including tool count, so length alone may not be the cause. The benchmark was designed to help examine exactly this kind of question.
What should builders do before deploying agents on SOPs?
If you are building or running agents that execute business procedures, SOP-Bench gives you a concrete checklist.
- Run your agent against real SOPs before trusting it in production. A static skills test is insufficient. You need sustained, tool-using runs against procedures with the ambiguity left in. SOP-Bench is open source, so you can test your agent on the included 12 procedures or add your own using the same human-and-AI construction method.
- Test every model upgrade as a regression risk. Do not assume a newer model will perform better on your agent pipeline. Run your full procedure suite before and after the upgrade and compare task success rates, not just individual capability scores.
- Trim your toolkit to fit the task. Giving an agent 26 tools when it needs 6 can cut success rates in half. Build tool sets scoped to the procedure, not aspirationally broad.
- Choose your agent architecture based on the procedure, not an aggregate score. A reasoning-style agent may be better for procedures with many decision points, while a function-calling agent may be faster and sufficient for straightforward tool chains. Profile both on your actual SOPs.
- Pay attention to where failures happen. SOP-Bench records the full trace of tool calls and reasoning behind each decision, so you can track failures back to the specific step where they occurred. This lets you identify where human oversight remains essential.
The evaluation metrics themselves are straightforward. Execution Completion Rate measures the proportion of tasks the agent marked as complete. Conditional Task Success Rate measures the fraction of completed tasks that match ground truth. Task Success Rate is the overall accuracy. Together they tell you whether your agent is failing to finish, finishing but getting wrong answers, or both.
The gap between demo and deployment
The SOP-Bench results map where the field still needs to invest. Raw model capability does not guarantee reliability on procedural work, and the distance between a polished demo and a dependable production agent is measured in the specific steps where the agent silently fails. The most useful thing Amazon Science built here is reproducibility. Every task has ground truth. Every run produces a trace. Every failure points to a step. If you are shipping agents that touch business procedures, test your model on the procedure that matters, with the tools it will actually have, against the answer it needs to get right.
Sources
- arXiv - SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents
- Amazon Science - SOP-Bench publication page
- Amazon Science Blog - SOP-Bench: A new benchmark for evaluating AI agents on real business procedures
- GitHub - amazon-science/SOP-Bench repository
- HuggingFace - amazon/sop-bench dataset
