Enterprise AI teams are giving agents more freedom as their confidence in automated testing collapses.
Half of companies have implemented an AI agent or LLM function that passed internal assessments and still caused a customer-facing failure (one in four more than once), according to the June 2026 VB Pulse survey of 157 qualified business respondents at companies with 100 or more employees.
The sample is self-selected and non-probabilistic, so the findings should be read as directional, not precise.
But companies are not responding by slowing automation: 66% of respondents already allow some production deployment without human review. or are building systems intended to do so in the next 12 months. Only 5% say they fully trust automated assessments that would make those launch decisions.
That mismatch is the evaluation gap: the autonomy limit is increasing faster than the guarantee below it.
It also fits into a broader thesis that will be explored in VB Transformation 2026– Enterprises dispatch agents first, while layers of control around identity, assessment, cost, context, and orchestration come later. Next year will be a modernization cycle, with buyers shifting budget toward systems that make agent deployments governable and reliable.
Why a passing evaluation is not a labor agent
Traditional software testing typically asks whether a defined input produces an expected result. Agent testing is more difficult because the system can choose its own sequence of steps, calling tools, retrieving data, altering state, and responding differently from one execution to the next.
An agent can make several individually plausible decisions and still arrive at the wrong result. You can retrieve the correct account but update the wrong field. You can write a valid refund request but submit it without approval. You can call five tools successfully before a sixth step leaks sensitive information or leaves an incomplete workflow.
The survey shows that companies already recognize this limitation. The most common reason for distrusting automated assessment is poor alignment with real-world results, cited by 29% of respondents. This is followed by bias or inconsistencies at 21%, lack of explainability at 18%, and data leaks or privacy concerns at 17%.
That hierarchy matters. Companies say the score often doesn’t predict what happens when a customer, employee, or business process encounters the agent in production; It’s not that automated scoring is too slow or expensive.
NIST makes a similar comment in its Generative AI profile: Measurements collected in controlled environments may not transfer cleanly to deployment because behavior changes with prompts, users, context, and operating conditions. Their guidance calls for field testing, post-deployment monitoring, and clear processes for failure escalation.
Capacity is not consistency
A single successful execution demonstrates that an agent can complete a task. It does not prove that you will complete the task reliably.
Anthropic’s Guide to Agent Evaluation distinguishes between measuring whether a system succeeds at least once in repeated attempts and whether it succeeds on all occasions. That distinction is essential for operational or customer-facing workflows. A model that occasionally produces an excellent response may still be unacceptable if the same task fails unpredictably on the next attempt.
Therefore, business teams should treat repeatability as a first-class metric. That means running the same scenario multiple times, varying wording and context, testing tool failures, and measuring whether the final business result remains correct even when the path changes.
The evaluation suite also has to evolve. Every production incident should become an ongoing regression test. Customer escalations, failed tool calls, incorrect approvals, and data handling errors should be fed back into the pre-deployment suite rather than remaining as isolated support cases.
Autonomy should be expanded by risk, not ambition
The survey does not imply that every agent action must require a person. Human review cannot encompass millions of low-consequence decisions.
But operation without human intervention must be achieved through demonstrated reliability and limited by the consequences of failure.
Low-risk actions, such as writing internal summaries or categorizing documents, can tolerate greater autonomy. Financial transactions, customer communications, code deployment, access control changes, and data deletion require stricter thresholds, repeated consistency testing, policy checks, rollback mechanisms, and clear human escalation paths.
Risk is also not evenly distributed based on company size. Larger companies (those with 2,500 or more employees) are moving faster toward deploying without human staff, at 70% vs. 64% for smaller companies, and they are also sending more agents who fail a customer, 54% vs. 48%.
That’s the warning to business leaders. Taking the human being out of the loop does not eliminate uncertainty. Without stronger security, uncertainty becomes an automated production decision.
The market will continue to push towards greater autonomy because the economic incentive is real. The best-positioned organizations won’t be those that eliminate people the fastest: they will be the ones that take repeatability and regression testing as seriously as deployment speed.
