Companies that build and deploy agents have a problem: It’s taking too long for their engineers to discover that an agent made a mistake, and the loop continues to perpetuate itself, especially without a human at every step.
LangSmith, LangChain’s monitoring and testing platform, launched a new capability in public beta that could make that problem more manageable. Lang Smith Engine automates the entire chain by detecting production failures, diagnosing root causes against the live codebase, writing a fix, and preventing regression. It does it in a single automated pass.
LangSmith Engine offers AI engineers a faster path to classification, but it’s launching into a crowded field: Anthropic, OpenAI, and Google are pushing observability and evaluation. on their own platforms.
LangSmith Engine analyzes the failures
LangChain said in a blog post that the typical agent development cycle begins by tracking the agent to understand what it is doing, followed by identifying gaps, making changes to prompts and tools, and creating real data sets. Developers then run experiments and check for regressions before shipping the agent.
The problem is that customers often have problems when the follow-up review does not reveal bad patterns, the repetition of errors becomes difficult to see, and there is no specific tester to detect the same problem when it is repeated in production.
LangSmith Engine works by monitoring production traces for several types of signals, “explicit errors, online evaluator failures, trace anomalies, negative user comments, and unusual behavior, such as the user asking questions that the agent was not designed to address,” according to the blog post.
The engine will then read the live codebase, find the culprit, and write a pull request before proposing a custom tester for that specific failure pattern. The human enters the approval step.
It is built on top of LangSmith’s existing monitoring and evaluation infrastructure and also works with the results of a company’s evaluator.
Unlike observability tools like Weights & Biases, Arize Phoenix, and Honeyhive, LangSmith Engine takes the entire chain automatically (detecting the fault, diagnosing the root cause, writing a fix) and brings the human in only at the approval step.
Model providers that add evaluators to the platform
While LangSmith identified this evaluation cycle as a necessity for many companies, Engine comes at a time when larger vendors are starting to offer observability tools within their platform. This means that companies can choose to use an end-to-end platform instead of adding LangSmith Engine to their existing workflows.
Agents managed by Claude de Anthropic brings together deployment, testing, and agent orchestration in a single suite. The OpenAI frontier offers a similar end-to-end platform for creating, governing and evaluating enterprise agents, although both have faced questions from companies wary of committing to a single provider.
However, practitioners note that not everyone wants to fully bring assessments and observability together on a single platform.
Leigh Coney, founder and principal consultant at Workwise Solutions, told VentureBeat that third-party observability is the default option for many companies.
“One fund I work with runs Claude for analytics and GPT for a separate workflow. If observability resides within each vendor’s tools, you now have two systems that can’t talk to each other. Your compliance team can’t produce a unified audit trail,” he said. “Thus, third-party observability is surviving because multi-model is already the default in enterprises, and someone has to sit between the vendors.”
Jessica Arredondo Murphy, CEO and co-founder of True Fit, said independent platforms like LangSmith have to show companies that they can "answer the long-term question of whether they will become the operational layer between models in terms of quality and reliability.”
“Enterprises are not consolidating on first-party model vendor tools as quickly as model vendors would prefer. What I see is a pragmatic divide: teams will use source tools for quick onboarding and debugging in the early stages, but as soon as they worry about production reliability, governance, and long-term flexibility, they tend to introduce a more neutral layer for observability and evaluation,” he said.
LangSmith Engine is now available in public beta. Teams can connect a trace project and optionally connect its repository, and Engine will begin detecting issues from production traces automatically.
