When agent workflows fail, developers often assume that the problem lies in the reasoning capabilities of the underlying model. In reality, the limited information provided by the recovery interface is usually the main limiting factor.
Researchers from multiple universities propose a technique called direct corpus interaction (DCI) that allows agents to completely avoid model embedding and search raw corpuses directly using standard command line tools.
The limits of classic recovery
In classic recovery systems like RAGDocuments are fragmented, converted to vector representations (or embeddings), and indexed offline in a vector database. When an AI system processes a query, a retriever filters the entire database to return a ranking. "top-k" list of document fragments that match the query. All evidence must pass through this scoring mechanism before any further reasoning occurs.
But modern agent applications demand much more. "Dense retrieval is very useful for broad semantic retrieval, but when an agent has to solve a multi-step task, it often needs to search for strings, numbers, versions, error codes, file paths, or sparse combinations of exact clues." the authors of the DCI article said in comments provided to VentureBeat. "These long-tail details are precisely where semantic similarity can be fragile."
Unlike static search, agents must also revise their search plans dynamically after observing partial or localized evidence. Exact lexical restrictions and multi-step hypothesis refinement are difficult to execute with semantic retrievers. Because the retriever compresses access into a single pass, any critical evidence filtered by similarity search cannot be retrieved later, no matter how advanced the agent’s subsequent reasoning capabilities are. As the authors explain, current recovery channels can become a bottleneck because "They decide too early what the agent can see."
Direct corpus interaction
This shortcut addresses a central problem in enterprise environments: data obsolescence. Integrated indexes are always a snapshot of a specific moment in time, and their creation and maintenance require a considerable amount of computation and time.
"In many business environments, data does not constitute a stable collection of documents. It’s the daily financial reports, live logs, tickets, code commits, configuration files, incident schedules, and internal documents that keep changing." the authors said. DCI allows the agent to reason about the current state of the workspace instead of yesterday’s vector index.
The agent operates in a terminal-like environment where its observations are results of raw tools such as file paths, matching text spaces, and surrounding lines. The basic tools provided by DCI are few but very expressive. Agents use commands such as “find” and “glob” to navigate directory structures and locate files. For exact matching, they use “grep” and “rg” to locate specific keywords, regular expression patterns, and exact strings. When local inspection is needed, tools such as head, tail, sed, cat, and lightweight Python scripts allow the agent to take a look at the context surrounding a match or read specific file sections.
The agent can combine these tools through shell channels to execute complex search logic in a single step. An agent can pipe commands to enforce strict lexical constraints, such as searching for one term in a file and pipe the output to search for a second term. You can combine multiple weak tracks into a corpus by finding a specific file type, searching for a keyword like "report," and filtering for a year like "2024." You can also immediately verify a hypothesis by inspecting the exact lines around a keyword match.
DCI delegates semantic interpretation directly to the agent instead of relying on embedding-based similarity search. The agent can formulate hypotheses, test exact lexical patterns, and extract detailed information that a traditional semantic retriever might miss.
The researchers propose two versions of this system. DCI-Agent-Lite is designed as a lightweight, low-cost configuration built on the GPT-5.4 nano model and restricted exclusively to raw terminal interactions such as bash commands and basic file reads. Because reading raw files can quickly fill the memory of a smaller model, this release relies on lightweight runtime context management strategies to sustain long-term exploration.
DCI-Agent-CC is the highest performance version, designed for teams with a larger computing budget. still works Claude Code powered by Claude Sonnet 4.6. Claude Code provides stronger prompts, stronger tool orchestration, and superior integrated context handling, improving agent stability during complex, multi-step searches across heterogeneous data sets.
DCI in action
The researchers tested both versions of DCI on agent search benchmarks such as BrowseComp-Plus, knowledge-intensive quality control with single-hop and multi-hop reasoning, and information retrieval classification on tasks requiring domain-specific reasoning and scientific fact checking.
They tested DCI against three baselines. The first included open weight recovery agents such as Search-R1 and proprietary agents powered by frontier models such as GPT-5 and Claude Sonnet 4.6, combined with standard recuperators. The second baseline included classic sparse retrievers like BM25 and dense retrievers like text-embedding-3-large and OpenAI’s Qwen3-Embedding-8B. The third baseline consisted of high-performance reasoning-oriented reclassifiers such as ReasonRank-32B and Rank-R1.
DCI consistently exceeded baselines, according to researchers. In the complex BrowseComp-Plus benchmark, swapping a traditional Qwen3 semantic retriever for DCI on a Claude Sonnet 4.6 backbone improved accuracy from 69.0% to 80.0% and reduced API cost from $1,440 to $1,016. The return on investment for light agents was also notable. DCI-Agent-Lite with GPT-5.4 nano competed with the OpenAI o3 model using traditional recovery and reducing costs by more than $600.
In multi-hop QC benchmarks, DCI-Agent-CC achieved an average accuracy of 83.0%, improving the strongest open weight recovery baseline by 30.7 points, according to the researchers.
The data shows that DCI has lower overall document recall than dense embedding models, but once it finds a relevant document, it extracts substantially more value from it.
"If an enterprise AI leader were to ask where DCI is most clearly useful, they would point to tasks that require pinpointing evidence in a dynamic workspace: debugging production incidents, searching large code bases, analyzing logs, compliance investigation, audit trails, or root cause analysis of multiple documents." the researchers point out.
In a complex deep investigation task, the agent had to identify a specific soccer match based on 12 intertwined clues, including exact attendance, yellow cards and players’ birth dates. A traditional retriever would fail if it showed short, disconnected fragments. Instead, the DCI agent scanned the archive directory, read specific lines from a report of the 1990 England versus Belgium match to verify the exact number of substitutions, extracted a specific quote from an interview file, and verified the exact birth dates of two players by looking at their Wikipedia text files. By chaining together these simple commands, DCI ensures that no evidence is permanently lost behind a faulty semantic search algorithm.
Limits and practical implementation of DCI.
DCI has a clear operating environment where it scales excellently in search depth, but struggles with search breadth. When the experimental corpus was expanded from 100,000 to 400,000 documents, the system’s accuracy decreased significantly and the average number of tool calls increased. While DCI is powerful once a promising document is found, the cost of locating that useful initial anchor document grows considerably as the size of the candidate space increases.
DCI also has lower recall of broad documents compared to dense embedding models. Trade exhaustive retrieval for high-resolution local precision. If a business workflow strictly requires finding all relevant documents in a massive data set, DCI may not be the right tool.
Giving an agent expressive tools, such as an unrestricted bash shell, increases latency and computational costs due to the large volume of iterative tool calls required to complete a search. It also creates significant security and context management challenges for IT departments.
"Tool calls can return great results; long paths can fill the context window; and direct access to the terminal requires sandboxing, permissions control and careful engineering," the authors said. To manage the context window, the researchers found that moderate truncation and compaction help the agent perform longer searches, while overly aggressive summarization tends to discard useful evidence.
Due to these operational realities, the DCI is not intended to be a mandatory replacement for existing vector infrastructure. Rather, it serves as a complement.
"For orchestration engineers and data architects, our view is that the most practical deployment pattern in the near term is hybrid," the authors said. Semantic retrieval can still provide high-recall candidate discovery when a user’s intent is broad or poorly specified. "DCI can then operate as a precision and verification layer: the agent can search within the retrieved documents, expand them to neighboring files, check exact constraints, and combine weak signals between documents."
The researchers have published the code for DCI under the permissive license of MIT.
"Longer term, DCI changes the way we think about enterprise data. Data will not only need to be stored for humans or indexed for search engines; It will need to be organized so that agents can inspect, compare, track, trace and verify." the authors conclude. "File names, timestamps, stable identifiers, metadata, version history, and machine-readable structure become part of the recovery interface."
