This is the first post in a series about building Ask Eric, an AI-assisted way to explore the writing on this blog.
I have been interested in AI for a while, but what keeps pulling me in is the engineering behind a useful AI feature. It is easy to make a chat bot. I did that a 2023 with an instruction set, an API Key and a UI. However, it is much harder to make one that can show where an answer came from, stay inside a cost boundary, and admit when the information is not there.
That is what Ask Eric is for. It lets you ask questions about the posts on this site and get an answer built from those posts, with links back to the evidence. This first post is the personal one. It covers what I did, in the order I did it, and what I learned. The rest of the series goes into the details.
It Did Not Start With AI
The first piece of work was not AI at all. In August, npm audit reported 151 vulnerable packages in this blog’s build: 6 low, 63 moderate, 68 high, and 14 critical. That number looked alarming, but npm audit --omit=dev reported zero. Everything was in the build tooling, a pile of old Grunt plugins and a TypeScript 2.6 toolchain that I had not touched in years.
I did not run npm audit fix --force and hope. I wrote a plan instead. Upgrade the platform first (Ruby, Jekyll, Node, npm), so stale dependencies disappear on their own. Triage what is left by whether it can actually reach a browser. Replace Grunt with a few plain npm scripts. Plan the move from Bootstrap to Tailwind as its own piece of work. The lesson I took from this is that I need to stay on top of my dependencies. A build-only moderate finding and a reachable browser critical finding are different problems, but the audit plan was useful because it forced me to upgrade everything systematically.
Then the Ask Eric Work
Ask Eric began in early September. Phase 0 finished on September 12 with a parser that could read the published posts and print each one’s title, URL, date, headings, and clean text. All 14 posts parsed. I had written that first version in Python, and the next day I moved it to .NET 10, where I wanted the whole application to live. I did not trust the port until it matched the Python output field by field on all 14 posts. Only then did I delete the old code.
From there the work went in the order the series follows:
- Chunk the posts in a repeatable way, with a stable ID and a path back to the source file and canonical URL.
- Embed the chunks and prove retrieval on its own, before any answer is generated. The first run embedded 157 chunks, and an evaluation set of known questions found an expected post in the top five for all 18 answerable cases.
- Generate grounded answers. The model cites chunk IDs, and the server maps those IDs to the chunks it actually retrieved, so an invented citation is rejected. In the saved evaluation, all 24 answerable questions got an expected citation and both unanswerable questions got an honest “the blog does not say.”
- Wrap it in an ASP.NET Core API, then deploy it on Azure behind a small Function proxy that checks Cloudflare Turnstile and rate limits requests before anything reaches the API.
- Build the Ask page and the per-post dialog, then add the safeguards: a kill switch, budget alerts, and a privacy notice.
The blog itself stays out of that picture. It is a static site, and it does not know the API exists until a visitor asks a question. If the API, the database, or the model provider goes away, the blog keeps working. I turned the AI feature off in production with a single setting to confirm that, and nothing else changed.
Two Migrations in the Middle
Two big changes happened while I was building, and I did not plan either of them when I started.
The first was storage. I began with PostgreSQL for the budget ledger and Qdrant for the vectors. Running two datastores for roughly 157 chunks was more than the project needed, so I replaced both with SQL Server 2025, using its native VECTOR(1536) type and exact cosine search. At this size an approximate index would only add risk, so I left it out. The evaluation scores had to stay the same across the move, and they did.
The second was the blog itself. In early October I migrated the site from Jekyll to Astro so I could stop building in Docker. The rule was that visitors, search engines, and feed readers should see no difference, so Jekyll’s final build became the reference. I froze it (94 pages, 81 sitemap URLs, 10 feed entries) and wrote checks against it. The new sitemap.xml came out byte for byte identical, every page’s SEO tags matched, and the feed kept its original entry IDs so subscribers would not see old posts again. I tagged the last Jekyll commit so I could always go back. Because the chunk IDs include the source path, the move also meant re-embedding the whole corpus, and the evaluations, now on a larger set of 30 questions, still passed: 30 of 30 retrieval hits, 30 of 30 cited answers, and 2 of 2 abstentions.
Using Astro is also a significant improvement for development speed and maintainability, as it allows me to leverage modern web tooling without the overhead of Docker.
What I Learned
- Freeze a baseline before you change anything. The npm cleanup, the Python port, the SQL Server move, and the Astro migration each worked the same way. Capture what correct looks like, change one thing, compare. It is slower for an afternoon and faster for the month.
- Preparing the content matters more than the prompt. Most of the answer quality came from knowing exactly what the published content was and splitting it consistently. The prompt was the smaller part.
- Measure retrieval before you generate anything. If the right passage never reaches the model, no prompt will rescue the answer. A small set of questions with known sources told me more than any demo did.
- Treat money as an engineering constraint. The application reserves budget before every model call, reconciles it with the reported usage afterward, and refuses requests when it cannot reserve. I wrote a concurrency test for it and then weakened the database isolation level to check that the test failed. It did, which is the only reason I believe in it.
- Plans change, and the written plan still earns its keep. I swapped databases, dropped Sass, and replaced a sitemap plugin with a short hand-written endpoint. Each time, the plan gave me something to compare the change against, and I recorded what differed and why.
- Keep the AI feature replaceable and switchable. The proxy, the API, the database, and the static site deploy separately, and there is a switch that turns the feature off without touching the blog.
Where this Series Goes
I am planning to detail the build in the posts that follow. Each one covers a milestone, what I decided, and what I got wrong along the way. I am writing them from my own project notes and test results, so the numbers are the ones I recorded at the time.
My next post goes back to the first milestone: establishing a reliable baseline of the blog’s content before adding retrieval or generation.
As always, keep learning and keep building.