This is the fourth post in a series about my journey building DevIndext and exploring how coding agents can benefit from better context.
At the end of Part 3, I said this closed the first DevIndext series and that I would keep sharing what I learned as the benchmarks broadened. That is not how it went. I have stopped active work on DevIndext.
The bad news: DevIndext wasn’t proving to be a viable option for coding agents. The good news: the software did not fail. Ultimately, I shelved it because I ran out of evidence that it was worth continuing.
The Idea
DevIndext was a .NET code indexing and retrieval engine for AI coding agents. Instead of relying only on Grep, Glob, and Read, an agent could ask for structured information about symbols, inheritance, implementations, references, and relationships in C#. Roslyn provided the underlying analysis and semantic model for this structured information.
The hypothesis was simple:
Better structural context leads to fewer tool calls, lower token use, and more accurate, efficient agents.
I still think that is a reasonable thing to work towards. The work was testing whether it was true.
What Worked
The engineering went well.
- A persistent, Roslyn-backed indexing engine with SQLite storage.
- An MCP server that exposes structured code retrieval to agents.
- Large indexing speedups from prepared statements, batching, caching, and resolution optimizations.
- Compact retrieval results and targeted relationship queries.
- A Claude Code integration that supplies context automatically after a file is read.
- Independent Roslyn-based validation of the index, and controlled agent evaluations.
Indexing became considerably faster, and the structural data it produced was highly accurate. If the question had been “can this be built?”, the answer was yes.
What the Experiments Found
The question that mattered was whether agents did better with it. I ran a series of evaluations to find out.
| Experiment | Finding |
|---|---|
| MCP adoption | Across 25 runs, agents made zero voluntary DevIndext calls. |
| Forced MCP usage | Cost rose about 31%, with no demonstrated improvement in answer quality. |
| Proactive enrichment pilot | Automatic context delivery worked, but no improvement in answers could be attributed to the injected context. |
| Cut-order policy change (C2) | Reprioritizing structural facts increased direct coverage from 26% to 63% within the same 300-token budget. |
| Read-rate evaluation | Only 14.3% of statable item-runs triggered useful enrichment. |
Each result closed off a different route to value.
Agents did not pick the tools. Given DevIndext next to their native tools, agents went to the native tools every time. That was the zero in the adoption test.
Making them use it did not help. When I forced the issue, the cost went up by roughly a third without a measurable improvement in answer quality. Using the tool is not the same as benefiting from it.
Delivering context without being asked didn’t demonstrate an advantage either. The next idea was to stop asking agents to choose, and inject the context for them. A Read hook added a short block of structural information whenever the agent opened a file. The delivery worked. I could not attribute any better answer to it. Even when the information was accurate, the agent often already had it from the searches it had run to find the file.
The hook relied on a behavior agents often skip. In the final read-rate evaluation, only 6 of 42 opportunities to deliver a statable structural fact were actually triggered by the files agents read—14.3%. Most agents answered these questions through Grep without opening the files containing the relevant relationships. The entire final evaluation effort cost about $2.43, which is cheap, and it confirmed the problem clearly.
The C2 result is the one I am proudest of for engineering. Each enrichment brief had a fixed budget of 300 tokens per event. When a file had more relevant structure than fit, the engine had to drop lower-ranked items, and C2 changed the order in which the engine spent its limited token budget: base relationships first, then relationship counts, then individual names. Changing that order more than doubled how much direct structural coverage fit in the same budget. It was a real improvement to something that agents were mostly not reaching.
Why I Stopped
Three things came together.
- Adoption. Agents consistently chose native search when they had a choice.
- No demonstrated incremental value. Where context was delivered, it rarely told the agent anything new.
- A poor fit with agent behavior. The delivery mechanism depended on an action that agents frequently do not take.
There were also known technical limits: incomplete dependency-direction coverage, and difficulty resolving types with the same name. Fixing them was possible. But it would not have answered the larger question, which was whether agents needed this information at all. Better answers to questions nobody was asking would have been a good way to keep busy.
There was no shortage of technical work left to do. I could improve ambiguous type resolution, expand dependency analysis, or experiment with different delivery mechanisms. Each would make DevIndext more capable. None would establish that the additional capability mattered. At some point, continuing to build becomes easier than admitting you haven’t validated the premise.
What the Project Showed, and What It Did Not
DevIndext showed that I could efficiently index a large .NET repository, retrieve accurate structural relationships, and inject them into an agent’s context.
Unfortunately, it did not show that doing so improved coding accuracy, reduced cost, or sped up tasks.
The tests were also limited. The task set used in the later design and read-rate work was exploratory, not a validated benchmark, and I would not claim the results generalize beyond the repositories, tasks, and agents I tried. What I can say is that I looked hard for an advantage and did not find one that justified more investment.
Where It Stands
I kept the enrichment engine, including the cut-order policy change and the devindext enrich command, and preserved the evaluation artifacts. The Claude Code Read hook remains experimental and opt-in, not a default. I deferred further work on the resolver, dependency analysis, additional integrations, and evaluation infrastructure.
The decision was to stop investing, not to keep tuning something whose advantage was still unproven.
The Larger Lesson
The problem DevIndext was aimed at is still real. Agents struggle with long-running context. They lose what they discovered, miss architectural intent, and start each session without what the last one learned.
DevIndext attacked that mostly through indexing and structural retrieval. My experiments suggest that current agents are already reasonably good at that part. They search, read, and piece together structure well enough that handing them a cleaner version of it did not change much.
What they are worse at is remembering. Decisions, constraints, discoveries, and hypotheses that were tried and ruled out are all things an agent works out during an investigation and then loses. I suspect the more promising opportunity is preserving what an agent has learned, not making source code easier to retrieve.
I do not have evidence for that yet, and after this project I am wary of believing something just because it sounds right. That is part of why this post exists. A negative result is still a result, and I would rather publish it than let the project quietly fade.
Conclusion
DevIndext was a technically successful experiment in structured context retrieval. I shelved it because it became clear that controlled testing did not show a meaningful advantage over the native tools coding agents already use, even though the broader problem of agent context remains very much open.
In the end, I learned a lot about Roslyn, SQLite, MCP, and benchmarking along the way. I spent a lot of time making DevIndext faster, more accurate, and easier for agents to use. What I should have established earlier was whether agents actually needed it. That’s the lesson I’ll carry into the next project: validate the advantage before optimizing the implementation.
As always, keep learning and keep building.