The scaffolding layer that developers once needed to ship LLM applications (indexing layers, query engines, retrieval channels, carefully orchestrated agent loops) is collapsing. And according to Jerry Liu, co-founder and CEO of CallIndexthat’s not a problem. That’s the point.
“As a result, there is less need for frameworks that actually help users compose these deterministic workflows in a lightweight and superficial way,” explains Jerry Liu, co-founder and CEO of LlamaIndex, in a new article. VentureBeat Beyond the Pilot Podcast.
The context is becoming the moat
Liu’s LlamaIndex is one of the leading recovery augmented generation (RAG) frameworks that connects private, personalized, and domain-specific data with LLM. But even he acknowledges that these types of frameworks are becoming less relevant.
With each new release, the models demonstrate incremental abilities to reason about “massive amounts” of unstructured data, and they are getting better at it than humans, he notes. They can be trusted to reason broadly, self-correct, and do multi-step planning; The Modern Context Protocol (MCP) and Claude Agent Skills plugins allow models to discover and use tools without requiring integrations for each of them independently.
Agent patterns have consolidated into what Liu calls a "managed agent diagram" — a leveraging layer combined with tools, MCP connectors, and skill plugins, rather than a custom orchestration for each workflow.
Additionally, coding agents excel at writing code, meaning developers don’t need to rely on extensive libraries. In fact, around 95% of LlamaIndex’s code is generated by AI. “Engineers don’t actually write real code,” Liu said. “Everyone is writing in natural language.” This means that the layers between programmers and non-programmers are collapsing, because “the new programming language is essentially English.”
Instead of manually coding or struggling to understand the API and document integration, developers can simply point it at you with Claude Code. “These kinds of things were extremely inefficient or would just ruin the agent three years ago,” Liu said. “It’s much easier for people to build even relatively advanced recovery systems with extremely simple primitives.”
So what is the main differentiator when the stack collapses?
Context, says Liu. Agents must be able to decrypt file formats to extract the correct information. Providing greater precision and more economical analysis becomes key, and LlamaIndex is well positioned here, he maintains, due to its developments with agent document processing through optical character recognition (OCR).
“We’ve really identified that there’s a core set of data that’s been locked up in all of these file format containers,” he said. Ultimately, “it doesn’t really matter if you use OpenAI Codex or Claude Code. What everyone needs is context.”
Maintain modular stacks
There is growing concern about builders like Anthropic locking session data; In light of this, Liu emphasizes the importance of modularity and agnosticism. Builders should not go for any frontier model, nor build too much in a way that overcomplicates stack components.
Recovery has become more “sandboxed,” as he describes it, and companies need to ensure their codebases are free of tech debt and adaptable to changing patterns. They should also recognize that, over time, some parts of the stack will have to be discarded as a matter of course.
“Because with every new model launch, there is always a different model that is more or less the winner,” Liu said. “You have to make sure you have some flexibility to take advantage of it.”
Listen to the podcast to learn more about:
-
LlamaIndex’s beginnings as a ‘toy project’ with initially only 40% accuracy;
-
How SaaS companies can leverage complicated workflows that must be standardized and repeatable for average knowledge workers;
-
Why AI verticals are taking off and why “build versus buy” is still a very valid question in the age of agents.
You can also listen and subscribe Beyond the pilot in Spotify, Apple or wherever you get your podcasts.
