A hidden test flaw inflated AI agents’ success on business tasks

Define the role, authority, approval gates and evidence before an AI agent touches production systems.

The short version

Business AI agents need narrow job descriptions, named owners, bounded authority, and evidence validating every action.
An audited benchmark’s apparent 94% success fell to 66% after a database-reset oracle defect was corrected.
Companies should promote agents gradually using trace-level tests against live policies, messy data, exceptions, and revoked approvals.

A benchmark gave AI agents a 94% success rate on business tasks. Junjie Chen audited it, found an oracle defect that reset the database and exposed the reference state, and saw replayed success fall to 66%.

Somebody had probably already pasted 94% into a procurement deck. PowerPoint remains undefeated.

Companies buy the clever interface before defining the ugly operations beneath it. AI agents for business make that expensive: they can alter records, call tools and keep working while everyone else is at aperitivo. I treat an agent as a narrow role inside an existing workflow, with a named owner and evidence validating every action. Models come and go. The surrounding operating system holds the value.

“Manage operations” is not a job

I define an agent by its loop: observe the process state, choose an allowed action, call a tool, check the result and decide what comes next. It stops at a defined finish condition or escalation rule. That operational responsibility matters more than conversational sparkle. A chatbot waits for my message. Traditional automation follows a prewritten sequence. An agent makes bounded decisions as the workflow changes, so its job needs the clarity I would demand from a human hire.

Plenty of products became “agents” because marketing needed a fresher noun. Image-to-image AI can edit a product photo. A reverse video search service can investigate reused footage, while one of the best search engines supplies research. An agent may call these useful tools. An animated orb still does not put software on payroll.

When founders ask how to build their own AI agents, I start with a job description: trigger, inputs, permitted actions, completion evidence and conditions requiring a person. Frameworks come later. LangChain cannot rescue success criteria that read like a horoscope.

I choose frequent, bounded, reversible work, like checking payroll inputs or preparing an expense claim for review. Before the model sees a prompt, the owner defines a measurable outcome. I map each step, separating fixed rules from acceptable judgment, and give human approval an explicit gate. I name the business owner accepting the result and the operational owner paged when the workflow jams. Only then do I choose tools and authority. If I cannot draw the finish line on a whiteboard, the thing does not belong in production.

I learned this through a mildly humiliating publishing workflow. It correctly published a post, and I congratulated myself like a Labrador opening a door. Then I reconciled referrer analytics with Google Search Console. Roughly 99% of the apparent activity came from bots. The automation had completed its task while feeding garbage into every later decision.

Charles Crouchman captured the production gap:

The demo is the easy part. The real test begins when an AI agent touches a production system, a financial transaction or a customer interaction.

A polished demo proves the happy path has excellent lighting. Show me the missing field, stale policy and customer record duplicated three years ago because Friday lunch involved two negronis.

Business data cannot explain company politics

An $18,000 invoice is data. A procurement rule requiring review above $10,000 is process context: it connects the invoice to current policy, responsible people and the next allowed step.

Teams often adopt agents and platforms independently, blurring ownership and duplicating capabilities. Business data alone cannot tell agents what should happen. They need a governed source for current roles, system state, applicable rules and controls. Task-scoped identity lets the platform compare each proposed action with that job’s authority. New evidence can change risk, narrowing permissions, triggering review or containing the workflow. A completed task remains invalid if its path broke a business constraint.

The machinery starts with a business event. For each step, the agent retrieves only the required live context, including the user’s role and process state. An orchestration layer routes work among specialized agents and existing systems. Every tool call carries task-scoped permission; actions beyond delegated authority await human review. Runtime controls log actions and watch for risk-changing behavior. The platform can preserve safe reads while blocking writes, narrowing access or stopping the agent. Its audit record must separate the human request, model proposal and executed action.

Uploading policy PDFs skips half the chain. Documents conflict, employees change roles and approvals expire. Model Context Protocol is useful plumbing between agents and tools. A connected pipe still needs somebody deciding what may flow through it.

Cream policy-binder pages curl into a short pipe sealed by a vermilion cap on cobalt blue.

A 2026 survey of 200 executives at large and multinational organizations by The Hackett Group and ARIS found that 82% considered process context necessary for deploying agents, but only 21% had comprehensive, real-time visibility into enterprise workflows. Organizations using process-context tools were more than five times as likely to report measurable AI returns. That association does not prove the tools caused those returns.

Nor has independent evidence shown those returns surviving years of deployment. Survey confidence is cheap. I want sustained ROI after maintenance, process changes and ugly exceptions.

Workday offers one governed-access example. The company says outside AI tools access its data through Sana, which applies each customer’s permissions and approval policies before acting. Workday also says every agent action is logged with the information used and its approver. These are vendor claims; I would test them against my workflows before opening champagne. Still, the architecture points correctly: live rules govern execution while the model works.

Before granting autonomy, I trace recent cases, especially weird ones. The official process says “manager approval.” Reality says Claudia approves Europe, Ben handles enterprise accounts, and Marco still gets emailed because nobody updated the spreadsheet in 2022. Benvenuti in azienda.

Approval must travel with the action

Every production agent needs an identity separate from the employee launching it. Otherwise, the log says Luca changed a record when software used my account, and everyone learns the difference during an indecently timed incident call.

Authorization starts when a human sponsor requests a defined task. The agent begins with minimum access and gets temporary permission only for the records and tools currently needed. For consequential actions, approval must bind requester and approver to the exact resource, destination and business purpose. A policy service checks those details outside the model’s control. New evidence can reduce access or require another review. Permission expires with the task, preventing yesterday’s approval from becoming tomorrow’s skeleton key. Logs show who requested the work, what the model proposed and what the system allowed.

Chenggong Shi and Hongmei Mou showed why generic approval tokens are weak evidence. Using the original configuration in the official OpenClaw execution environment, they ran controlled local experiments where an agent obtained a normal token and reused it for a protected operation. Nine reruns reached the approval transition; eight completed the exact target action. This was not a reported production breach. The failure was still painfully clear: possessing a token did not establish authority for that requester, recipient and purpose.

Satya Nadella described the principle on the Microsoft Security Blog:

The most trustworthy agentic system will not be the one with the model we trust most. It will be the one designed that enables us to trust the model least.

Start agents at zero or minimum privilege, granting authority for one task and limited time. Microsoft’s guidance describes this model, making policy dynamic rather than a one-off checkbox. New evidence changes risk; revised risk changes permitted actions. Static permissions assume tomorrow resembles the demo, which is adorable.

Controls remain nowhere near mature. A ServiceNow and Phronesis Partners survey of more than 3,700 VP-level or higher leaders across 20 countries found 90% reported an agentic AI security incident in the previous year, versus 10% who did not.

A separate Salt Security survey of more than 380 security and technology professionals found only 18% could automatically stop an agent behaving unexpectedly. Most had to intervene manually or take the application offline, less kill switch, more frantic crawl behind the server rack.

The uncomfortable gap comes before money moves or data escapes. We lack good evidence showing how often runtime controls prevent irreversible harm, or that permissions survive delegation across multiple vendors, tools and MCP servers. Legal accountability blurs when a customer, platform, model provider and tool vendor touch the same harmful action. Anyone calling this solved sells confidence by the kilo.

The execution trace gets the final vote

I evaluate the entire trace because a correct final screen can conceal skipped approval, exposed data or a duplicate write. Strict success requires both the requested outcome and valid execution throughout the workflow.

BizAgentBench makes this concrete. It contains 50 multi-turn tasks based on a payments company’s procedures, backed by 789 knowledge documents and 601 business tools. Claude Opus 5 achieved 64% strict success, requiring goal attainment plus compliance with the benchmark’s constraints. A constructed benchmark cannot predict reliability inside my company, but it shows what “the task was completed” hides.

The obvious pushback is simple: the business requested an outcome, and the agent delivered it. I accept that for low-risk chores. It fails around payroll, payments and customer records, where an invalid intermediate action creates liability despite a perfect final dashboard. Enhan Zhao and the BizAgentBench co-authors distinguish goal attainment from strict success for exactly this reason. Policy constrains the route, not just the destination.

My pilots start in observation mode against the current human or automated baseline. I replay normal cases plus stale policies, unavailable tools and revoked approvals. The agent advances to read-only access, recommendations, then reversible actions requiring approval. Each promotion depends on trace-level evidence from the company’s workflow. I track completion, human intervention and recovery after failure. When ordinary automation wins, I avoid buying an expensive LLM-shaped panini press.

Chen’s benchmark audit makes the harsher point: published scores may measure evaluator defects as easily as agent capability. His replay cut recorded success from 94% to 66% and changed the agents’ ranking. Give me a boring test built from my exceptions over a heroic score from somebody else’s broken oracle.

How well do benchmarks predict performance under each company’s live policies, messy data and exception handling? We do not know.

By 2028, enterprises will choose agent platforms mainly for execution controls and preserved evidence. Models will become replaceable infrastructure. The winning platform will explain exactly what happened Friday evening, after Claudia went home and Marco’s ancient spreadsheet woke up hungry.

Frequently asked questions

How do you build an AI agent for a business?

Building an AI agent for business starts with a job description covering its trigger, inputs, permitted actions, completion evidence, escalation conditions, and human approval gates. A named business owner accepts results, an operational owner handles failures, and tools or authority are selected only after the workflow and finish line are clear.

Why do AI agents need business process context?

Process context connects live business data to current roles, policies, system state, responsible people, and the next permitted step. Without it, an agent may complete an outcome while violating approval rules or other constraints. Governed context also allows permissions to narrow, reviews to trigger, or workflows to stop when risk changes.

How should companies evaluate AI agents before production?

A business AI agent should be evaluated on its complete execution trace, not only its final output. Strict success requires both the requested outcome and valid execution throughout the workflow. Pilots should replay normal cases and exceptions, track completion, human intervention, and recovery, and advance authority only when company-specific evidence supports promotion.