This is the third post in a series about my journey building DevIndext and exploring how coding agents can benefit from better context.
Getting DevIndext to work was a milestone. Getting it to the point where I would trust it in an ordinary development workflow is a different kind of work.
That is the phase I am working through now.
The core application is in place. DevIndext can incrementally index a repository, extract C# structure, answer repository-local questions, index Git history, group repositories in a workspace catalog, provide bounded context retrieval, and expose those capabilities to coding agents through a local stdio MCP server. Now I am focused on making all of that measurable, recoverable, documented, and repeatable.
It is easy to overlook this work because it does not always make for a dramatic demo. It is also where a project starts becoming a product.
The Work Behind “Ready to Use”
For DevIndext, release preparation means a few things need to work together.
- A clean checkout should restore, build, and pass the full test suite.
- Release candidates need to be packaged and smoke-tested on Windows and Linux.
- CLI and MCP failures should be actionable when configuration, repository resolution, indexing, or a query goes wrong.
- Incremental indexing and context retrieval need benchmarks, not just my sense that they are fast enough.
- Installation, upgrade, recovery, and configuration guidance need to match the software people actually install.
The application already has CI that restores, builds, tests, publishes, and smoke-tests self-contained Windows and Linux candidates without publishing release artifacts from pull requests. A manually triggered release workflow produces versioned packages, checksums, tags, and GitHub releases.
That does not mean the work is done. It means the release process is becoming something I can test instead of a list of steps I happen to remember.
Keeping Refresh Safe
One piece I recently completed is incremental refresh through MCP. A coding agent needs an index that reflects the repository it is working in, but “refresh” should not be an excuse for a remote tool to do broad or destructive work.
The behavior is intentionally constrained. A repository refresh uses the existing repository configuration and performs incremental indexing. A workspace refresh only resolves repositories from the local workspace catalog, refreshes them one at a time, and continues if one fails so the rest of the results are still useful.
The tool returns change counts and progress instead of a vague success message:
refresh_index(repositoryRoot)
-> added, modified, deleted, unchanged
-> elapsed time and progress
-> actionable error when needed
refresh_workspace_indexes(workspaceName)
-> catalog-registered repositories only
-> one result per repository
Those boundaries protect a few things that matter. A refresh does not quietly broaden configured include and exclude rules. It does not require a destructive rebuild. It has cancellation and time limits. And one failed repository does not erase the successful results for the rest of a workspace.
Measure the Retrieval, Not Just the Build
Context retrieval is still the feature I care about most. If DevIndext is supposed to save exploration steps and reduce unnecessary context, I need to measure that claim.
The project has deterministic synthetic benchmarks for direct relationships, multiple hops, fan-out limits, and cycles. It also has a real-repository benchmark mode that indexes DevIndext itself, checks curated expected symbols, captures the repository revision and graph size, and reports quality, latency, and allocation metrics. The reports are retained with CI, so a result can be traced back to the revision that produced it.
The first real-repository baseline used four scenarios: the context engine itself, the CLI command, the MCP tool, and Git-context indexing. It ran 100 measured iterations after warmup against a pinned DevIndext revision containing 130 indexed documents, 932 symbols, and 1,138 relationships. The context paths landed at roughly 3.6 to 4.4 milliseconds at p50 in that run. More importantly, the benchmark checks precision, recall, and an explanation for every returned item instead of treating a fast response as a useful one.
One result was especially useful because it was not a victory lap. The MCP scenario was deliberately limited to two returned items. Comparing that cap with four expected items made it clear that the benchmark needed to test the bounded contract, not mistake an intentional result limit for lost relevance. That adjustment made the scenario more honest and the tool’s behavior easier to reason about.
I also added an incremental-indexing fixture that measures the transitions a normal working tree actually sees. On its small, repeatable twenty-file fixture, a no-change refresh had a 10 ms p50; a one-file edit, rename, and delete stayed in the roughly 18–22 ms range. Narrowing configuration, which removes nineteen indexed documents, was about 106 ms at p50. Those are baseline measurements, not promises for every repository or machine, but they give future changes something concrete to compare against.
The storage diagnostics have been just as valuable. A 7,500-document test showed an unnecessary FTS rebuild creating about 2.25 MiB of free pages in the SQLite database. Removing that rebuild means a healthy no-change index no longer rewrites the FTS data. A separate 10,000-document structural-replacement test kept free space below half a percent even after every document had been replaced. That is exactly the kind of less-visible behavior I want to catch before it becomes somebody else’s performance problem.
That is encouraging, but I do not see one repository or one fixture as a finish line. I still need stable hosted-runner baselines and a broader set of representative repositories and operational edge cases.
Good benchmarks are not just about getting a smaller number. They help me notice when a change makes retrieval slower, noisier, or less relevant before that regression becomes part of somebody else’s workflow.
What Is Left
The remaining hardening work is practical: broader performance baselines, large-repository and edge-case behavior, diagnostics, failure modes, upgrade and recovery guidance, and continued validation that the index remains derived local data rather than a risky second source of truth.
I also want to keep the scope honest. DevIndext does not need to become a cloud platform, dashboard, or background service to be useful. Its value is a focused, local way for developers and coding agents to retrieve persistent development context.
Conclusion
Building the CLI and MCP server has been a great exercise in architecture, indexing, developer tooling, and AI-assisted workflows. Finishing the release work has been a reminder that reliable software is not only about the most interesting feature. It is also about the path from a fresh machine to a working tool, how it behaves when things go wrong, and whether people can trust its results.
This closes the first DevIndext series. I will keep sharing what I learn as the benchmarks broaden and the release work continues.
As always, keep learning and keep building.