Few companies face greater risks when implementing AI than WaymoAlphabet’s self-driving car company that spun out of Google. Their models don’t just generate text or automate administrative tasks: they help vehicles navigate unpredictable streets, respond to human drivers, and make split-second decisions in the physical world.
But the methods Waymo uses to manage those risks—continuous assessment, carefully curated data, human oversight, and clearly defined business outcomes—offer a broader playbook for companies deploying AI agents in almost any industry.
Manasi Joshi, director of systems intelligence and machine learning engineering at Waymo, explained in VB Transformation 2026 how the autonomous vehicle company trains, tests and deploys AI at scale. To date, Waymo has powered more than 220 million fully autonomous vehicles, or "cyclist only," miles, with 17 times fewer serious accident injuries than human drivers over the same distance, according to the company.
To achieve these impressive results, Joshi said Waymo adopted what she called “evaluation-forced development” or “evaluation-centered development,” making evaluation a central part of engineering rather than a final check performed before deployment.
“The stage at which our projects are maturing can be easily determined based on the assessment maturity they show,” Joshi said.
In practice, Waymo assesses a project’s readiness in part by examining the maturity of the evidence surrounding it. That approach has clear implications for companies building customer service agents, coding assistants, financial systems, or other AI applications: If a company can’t reliably measure the performance of a system, it may not be ready to put that system into production.
Evaluations should continue after launch.
Joshi said much of Waymo’s quality work has shifted toward evaluations, including tests conducted during model training, after training, and within open- and closed-loop simulations.
“Eval is not a one-time task to launch a model,” he said.
Instead, Waymo treats evaluation as a continuous process that encompasses driving, simulation, and validation. Its methodology combines data sets, performance metrics and infrastructure capable of operating efficiently at scale.
For businesses, that means testing an agent before launch is insufficient. Teams should continue to evaluate it as underlying models, business processes, user behavior, and incoming data change. Those assessments should also connect to actual business results rather than relying solely on broad industry benchmarks.
Joshi cautioned that measurements of model quality are only as reliable as the evaluation data behind them. Waymo therefore combines its performance claims with information about the properties of the data sets used to test its systems.
Testing the rare and dangerous cases
Waymo’s review hierarchy remains based on one overriding goal: safety.
The company relies on its own driving records, some third-party data, and realistic simulations that expose its systems to scenarios spanning billions of synthetic miles. Task owners choose specialized data and metrics for situations involving vulnerable road users, railroad crossings, construction zones, and other complex environments.
The same principle applies outside of autonomous driving. Companies should test not only routine requests that their agents handle successfully, but also rare situations in which errors could lead to financial, legal, security, or reputational damage.
Joshi emphasized that Waymo does not leave launch decisions entirely in the hands of automated systems. Their production readiness reviews include extensive human oversight, while internal security leaders approve software releases and service area expansions.
“This is not AI-driven, it’s not fully automated and it has no human oversight,” he said. “Human lives are at stake.”
Efficiency cannot come at the expense of reliability
Waymo faces another problem familiar to enterprise AI teams: Demand for compute, storage, memory, and network capacity is growing faster than available resources.
The company seeks efficiency in data extraction and storage, distributed model training, model distillation, simulation and evaluation. It also emphasizes “data efficiency,” selecting the most useful training examples rather than treating a larger volume as inherently better.
Waymo began using transformers in 2017 and subsequently expanded to large language models, vision-language models, and vision-language-action models. Joshi said the company now uses generative multimodal models as part of its core model strategy.
Waymo divides its technology between onboard systems within each vehicle and external infrastructure used for model development, data processing and simulation. That combination forces the company to optimize both real-time inference and the larger systems that support it.
Agents need their own evaluations.
Waymo also uses AI agents internally as productivity tools for engineers. Joshi said the agents help analyze data distribution, evaluate data efficiency and classification issues encountered in vehicle telemetry, training runs and failed evaluation jobs.
The goal is to speed up research work so that engineers can spend more time judging and solving difficult technical problems. But Waymo also tests those agents to ensure they produce reliable and accurate results rather than sending employees down unproductive paths.
For business leaders, the most important lesson from Waymo is that agent AI requires more than choosing a powerful model. Organizations need a clearly defined goal, representative evaluation data, continuous testing, infrastructure that can operate efficiently, and designated human decision makers who remain responsible for implementation.
"Earning trust is extremely important," Joshi said.
