A team that leads consultations through a coding specialist, a logic specialist, and a generalist model assumes that each will cover the blind spots of the others. A new study Evaluation of 67 frontier models from 21 vendors shows that the assumption is mathematically flawed, and the flaw has a name: the co-failure ceiling.
The assumption works like this: As long as two models do not fail with exactly the same indications, combining them is supposed to create a safety net against failure.
The true limit of orchestration is not how often the models disagree, but the percentage of prompts in which each model in the group gives the wrong answer at a time. By ignoring the co-failure ceiling, companies are building complex and expensive routing infrastructure to chase performance gains that don’t exist. Fortunately, developers can use this same math to create a free trial that determines exactly when multi-model orchestration will really pay off.
The hidden costs of the multi-model strategy
To orchestrate multiple language models, developers typically rely on three architectures. model routers They act like traffic police, sending complex queries to expensive models and simple queries to cheaper models. Waterfalls send all messages to a low-cost model first and only escalate to a premium model if the initial system indicates low trust. Finally, approaches like Mixture of agents (MoA) fuse multiple models by asking them the same question and generating a synthesized answer from their combined results.
These architectures introduce a "shadow price" to inference costs. Every time a development team deploys a router or cascade, they pay a premium in additional system latency, complex infrastructure maintenance, and increased governance risks across multiple API providers.
To justify these operating costs, engineers rely on “pairwise error correlation” to select their group of models. Imagine that a developer has Model A, who writes great Python but fails in SQL, and Model B, who writes great SQL but fails in Python. Because they fail on different types of cues, their pairwise error correlation is low. The developer assumes that by placing a routing layer in front of them, they have created a composite system that rarely fails in coding.
According to the study, combining diverse models based on low correlation can hurt performance if the models are not equally capable: When voting among diverse but unequal models, the weaker ones often join together and outvote the smarter one.
Josef Chen, author of the paper, told VentureBeat that in his experiments, "Naive majority voting in unequal models had a negative mean gain (minus 10 points in our hard combination): the diverse but weaker members outvote the strong one." Practical advice for developers is "only combine models within a band of equivalent quality." If you can’t match the quality, take the base of a single model and spend your budget on the best model available.
The paper provides a bright spot for this approach with respect to MoA architectures. When building sets, teams often use "Auto-MoA," where they query the same premium model multiple times to generate a synthesized response. The researchers found that, with equivalent quality, building a diverse set of models with low pairwise correlation outperforms a high-correlation Self-MoA setup.
However, when teams use that same pairwise correlation metric to predict the absolute accuracy of their overall system, the math breaks down.
"Therefore, teams pay orchestration overheads up front (latency, complexity, multi-vendor operations) assuming a diversity dividend comes later." Chen said. "This is generally not the case, because the best current models agree and, worse, fail on the same questions… the message simply conveys little signal about which model will be correct when the frontier disagrees."
Why Mathematics Fails: The Failure Ceiling
The main finding of the study focuses on a metric called "co-failure rate" — the formal name of the totally wrong scenario described above. No router, voting system, or cascade will be able to achieve accuracy higher than the limit it imposes.
The coding, logic, and generalism set shows low pairwise correlation in routine prompts; They rarely fail together. But the co-failure ceiling represents the obscure and highly complex edge case that pushes the limits of current AI architectures. If a message is so difficult that all three models hallucinate or fail, it doesn’t matter how intelligently the router distributes the task. The entire pool is cleared at once.
The researchers tested their set of 67 models, which included GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro, on the open mathematics benchmark MATH-500. Based on standard pairwise correlation, statistical models predicted that the entire group would disappear simultaneously in only 2.3% of the questions. In reality, the co-failure rate was 5.2%.
Standard correlation metrics underestimated the failure rate by approximately 2.25 times. The culprit is not just an independent difficulty, but a shared point of failure.
"The driver is what we call a common mode atom: a portion of queries where the entire market fails, which no pairwise statistics can detect." Chen said. "Adding a number 20 model to your pool does not buy tail coverage. The queue is shared."
The researchers also found that task format directly triggers joint failure. When they took graduate-level science questions from the GPQA benchmark and changed them from multiple-choice formats to free-response formats, the completely incorrect tail expanded to 12.7%.
However, developers can design around the roof. "The engineering implication is awkward: multi-model setups buy less exactly where teams want it most, in an open generation," Chen said. "Anywhere you can turn generation into verification or constrained selection (structured results, verifiable answers, execution tests), you open the roof again."
Ultimately, the researchers found that this limit limits AI applications in two distinct ways, depending on the domain:
-
Sandboxes (e.g. open math): The co-failure rate is high. The task is too difficult and all models fail simultaneously. No amount of routing can overcome the underlying lack of capacity.
-
Environments linked to achievability (e.g. graduate sciences): The co-failure rate is close to zero, meaning that at least one model in the group usually knows the answer. However, the models disagree so subtly that a routing layer cannot reliably choose the correct answer without an omniscient oracle.
The $0 Pre-Deployment Sanity Check
Before spending engineering hours building a router, teams can calculate their absolute performance limit for free using a mathematical formula called the Clopper-Pearson limit.
The Clopper-Pearson limit works as a worst-case calculator. If you flip a coin ten times and get eight heads, you can’t guarantee that the coin will come up heads 80% of the time forever. The limit takes a small sample of exam questions and generates a mathematically guaranteed limit.
Applied to language models, suppose a team tests a group of five agents on 50 sample queries and finds that they all fail on just two questions. A developer might assume that their multi-agent system will achieve 96% accuracy in production. The Clopper-Pearson formula corrects this optimism. It analyzes the small sample size and provides a mathematical guarantee that the true co-failure rate could be as high as 12%.
To use this in practice, companies must create an available data set. A fintech company, for example, could take 200 complex customer support tickets from the previous quarter and have human agents write perfect resolutions to serve as a benchmark. While this seems like a heavy manual project, mature engineering teams can automate the entire roof calculation.
"The integration is trivial: it is a job of counting the evaluation records that the teams already produce," Chen points out, "so it runs in the same CI stage as the evaluation set and is reactivated whenever the model group or workload changes."
The engineering team then compares their candidate models against these 200 tickets once and records the results. When they want to evaluate multiple model configurations, they can use the co-failure rate measure to predict the maximum accuracy they can get from the system without running additional queries.
An important conclusion from the study is that on tasks where answers can be definitively verified, combining models rarely outperforms using the best model on the market, unless the computer has an exceptionally strong query-level routing signal.
In a business environment, a definitively proven task has an objective, zero-tolerance response. This includes generating an SQL query that should execute without errors, extracting the total for a specific invoice from a 50-page PDF, or formatting a JSON payload that perfectly matches a strict schema. For these tasks, companies are often better off paying a premium for the smartest frontier model rather than combining three cheaper models and hoping that a router chooses the correct output. The study did not test subjective, ungraded tasks, such as writing marketing copy; The authors note that whether these findings hold outside their verifiable benchmarks remains an open question.
Because this mathematical verification is free, business teams can track their own co-failure rates as new models appear.
"It costs nothing to measure, so any team can track its own co-failure rate across generations of models and see if the tail is closing." Chen says. In the end, "The lever buyers hold is failure mode heterogeneity and market churn, not model count."
