This is an AI system that answers questions about my background while remaining grounded in real portfolio data rather than autocomplete.
A senior solutions architect I was speaking with, at a relatively large company, had their team building a RAG system for a number of their company's websites which were dense in specific information. It technically worked and it ran without real errors, but they were clearly unsatisfied with it as its responses just weren't good enough. It was a short wind-down conversation after a longer chat but it stuck with me — I'd hit that same gap before myself: AI systems that run without errors but still aren't good enough to actually trust or useful enough to beat simple solutions. A RAG was easy enough to get running and apparently deceptively hard to get good; even experienced developers who know their own company's content better than an outsider ever could, still struggled. I wanted to experience the process for myself and see if I could do better, or if it really wasn't worth it.
The pipeline works in five stages, end to end, in a couple seconds per query.
The portfolio content alongside much of my professional information (work history, projects, skills, FAQ answers) are split into chunks on average 150 words about specific topics. This is paired with relevant metadata like the document sources, probable sections, then these chunks are embedded with Qwen3-Embedding-0.6B, and stored in a persistent ChromaDB collection with cosine similarity set explicitly as the distance metric. This chunk-to-embedding process happens once on my device so the raw data stays local and does not happen on every cold server start. Testing with Chroma (which defaults to L2) showed better performance with cosine similarity than L2 for this use case. What does happen on every cold start (since Modal scales this to zero when idle for 10 minutes with our setup) is a reload where the embedder loads back into memory and the prebuilt collection reconnects without being recomputed.
Next we need the query (the user's question) embedded to compare similarities with our data, but first the query must be cleaned and normalized, which for us is transforming some simple queries like "does he" into "does Sherriff Kadiri". This anecdotally helps in avoiding hallucinations where the model believes it is Sherriff Kadiri or that it is answering on behalf of someone other than Sherriff Kadiri. The cleaned query then goes to locally-hosted Qwen3-8B model with a prompt to return JSON array strings. This JSON enforcement is on the schema level rather than left to the model itself, to ensure JSON-parseable responses every time. This prompt takes the query — a question like "has he shipped production apps, and what's his ML background?" — and breaks it down into three distinct sub-questions that will retrieve more specific responses than our single prompt. While this query-split step may feel excessive for a project of this size that will often receive simple questions, I have found it is one of the critical differences in getting responses that feel as though they get at the heart of what the asker wants to know. This lacking depth of output is what I think is critical in making projects like RAGs not feel "good enough".
The original question and every generated variant each run two searches in parallel: a BM25 keyword search over the full corpus, and a dense vector search against Chroma, five results each. This expanded context really ran against the limits of our previous Gemma 2:2B model and was the final reason we swapped to Qwen3-8B, alongside minor benefits like larger prompt contexts. If the best BM25 score across the whole corpus doesn't clear a minimum relevance floor, BM25 results for that query are discarded outright rather than folded in as noise — exact-term search should either be confident or silent, not weakly present.
Every one of the ranked lists for the original question and every variant, BM25 and vector, get merged with reciprocal rank fusion (k = 60). Critically this is not just the original question's two lists but each sub-question too. A chunk that shows up reliably across multiple query variants and multiple retrieval methods outranks one that only scored well once, enhancing the final output. An incidental benefit of this fusion is that since scores accumulate per chunk ID, no chunk can appear twice in the fused output.
The top five fused chunks go to the model under a constrained system prompt. Every response is checked for persona breaks or refusal to answer before it's returned. If either fires it falls back to a fixed response instead of guessing. While more research will be done on handling vagueness, I'm of the belief that when a person or AI model truly doesn't know something, it's best to simply state so.
Due to how much personal data this project requires I decided to avoid sending all my details to a third-party API from the start. This meant some self-hosting was inevitable, and without powerful hardware I planned around a small model from day one: all-mpnet-base-v2 for embeddings and Gemma 2:2B for generation. Some of my master's coursework scanning Twitter posts for emergency response data meant that processes like vectorization were already familiar to me, so I started with this common knowledge rather than try forcing something like TF-IDF to work. The articles Grounding Your LLM and its follow-up, Your Chunks Failed Your RAG in Production, both by Priyansh Bhardwaj, convinced me to spend real time on chunking early and keep coming back to it, which mattered more later than it seemed to at the time.
The first version I got running was about as basic as a RAG pipeline gets — every chunk got embedded once with all-mpnet-base-v2 and put into Chroma. Each question was embedded when asked, compared against every stored chunk embedding with Chroma's default L2, and whatever was returned in the top 3 was sent in a prompt to Gemma 2:2B to answer from. That was the simple pipeline, without rewriting questions, negative chunks, or keyword search.
It worked, in the sense that it produced answers and didn't crash. It did not work in any sense that actually mattered. Its tone would shift mid response depending on the chunks retrieved, there were no guardrails against hallucination, and it could easily be tricked into alternate behaviors. While expected to a degree, it was quite harrowing to see how badly a RAG could work. The first two fixes were a keyword list that caught the model breaking character or admitting it was an AI, and a small set of rules that rewrote shorthand like "does he" or "is he" into something the embedder could actually work with. Neither was sophisticated. Both were necessary before anything else was worth building on top of.
Building a better eval
This is also where I built a basic testing process rather than just eyeballing answers. My first instinct of writing the test questions myself effectively tested what I thought was important, rather than asking realistic questions from the visitor's point of view, which skewed initial results. I used Claude and Gemini to help generate a more realistic, less self-referential question set — things like "full-stack or mainly backend?" or "how does this person handle ambiguity?" — which felt closer to questions someone unfamiliar with the project might ask. The methodology I settled on splits review into roles that never share information they shouldn't: an accuracy reviewer that sees the expected answer, a validity reviewer that only sees the response and checks for tone, and hard-coded checks kept separate from both. I have learned how critical this separation is from working with DataAnnotation to test in development frontier ML models doing specific tasks. The instinct to let one reviewer both know the answer and judge tone is a trap worth resisting deliberately, as following it will result in massively skewed test results.
From there it was real iteration, each change tested against the same 35-question set so I'd know if it actually helped or just felt like it should. Adding BM25 alongside the vector search was the first thing that moved the number in a way I could point to.
A pure embedding search treats something like "85% validation accuracy" as just more token soup, since cosine similarity was never built to preserve exact numbers or proper nouns — keyword search catches precisely what that method is bad at. Chunk size tuning at this stage was not particularly useful, and though managing the chunks became considerably important later, I avoided spending too much time on it here, seeing little improvement.
I also tested going bigger before I tested going deeper. Larger models passed my tests more easily than the small ones, but the answers still felt shallow, which was the first real sign that passing my own tests and actually being good were different problems. I went back to a smaller model deliberately, and kept working on retrieval instead of leaning on model size to paper over whatever was actually broken.
The metadata experiment is the one I'd call an honest mistake. I added a relevance-ranking field to weight which projects surfaced more, then built a three-way fusion of vector score, BM25 score, and that metadata relevance. My goal in doing so was to bias certain projects as more important overall than others. It was obvious, looking at the results, that I was trying to inject depth into a system still fundamentally shallow underneath, and my 35-question test wasn't sensitive enough to catch this. I pulled almost all of it out and focused the metadata down to a handful of likely-related questions per chunk, a short section summary, and a sources field. Less metadata, used more carefully, kept a level of response consistency that the original metadata inclusion created without further confusing the smaller models. With that pullback, plus deduplication so a single topic couldn't dominate a result set, I moved to a new 70-question test focused on specifically testing factual recall, identity correctness, synthesis, soft-skill questions, boundary cases, roleplay, effort, and honesty. By this point I had moved from an average of 46 out of 70 to an average between 55–58 out of 70, with quite some hyperparameter tuning.
What was still failing at that point was that the model could sometimes be talked into roleplay, handled soft-skill questions inconsistently, and would occasionally fabricate information — for instance, treating something I was actively learning as something I'd already shipped if a question was ambiguous enough. Spending some time on system prompts showed that small rule sets helped, but a long, fully-specified prompt covering every edge case added latency for barely more depth. Multi-query splitting worked well and pushed the average into the low 60s, but the system still felt like it was hitting a ceiling. It was accurate, well-guarded, and not satisfying to actually talk to. A much bigger model probably would have papered over it, but as I was trying to maximize speed and minimize costs, I wanted this to be the last resort.
It was accurate, well-guarded, and not satisfying to actually talk to.
This is roughly when the leftover deduplication process from the metadata phase stopped making sense. Once that field was dropped, deduplication lost its job, and properly implemented reciprocal rank fusion turned out to cover some of the same ground anyway, as it accumulates score per chunk ID and can't surface the same chunk twice. With its original purpose gone and RRF already doing that work incidentally, keeping a separate step stopped being worth it.
What actually broke the ceiling was borrowing an idea from early image-generation models: negative prompting. I went back through my chunks and explicitly added what I hadn't done, not just what I had — "he has not run a production Kubernetes cluster," not just silence on the topic. That single change gave the model room to answer with actual confidence instead of vague hedging, and it's directly why "does he know Rust" gets a nuanced answer today instead of a confident lie or an unhelpfully vague non-answer. At this stage, further narrowing down each chunk to more specific information and limiting the summary size to around 150 words did start to show improvements.
That single change gave the model room to answer with actual confidence instead of vague hedging.
Deployment
As data control was a critical part of this project and I had resolved to make this project cost as little as possible, I needed hosted server providers with strong limits on pay-as-you-go pricing to avoid an expensive bill if a public endpoint I built ever got hit hard or abused. I found Modal through a LinkedIn post, and it fit for a specific reason beyond the easy Python integration: the free monthly credit comfortably covers the traffic I actually expect. Of course I then had to deal with the cold-start problem, where it takes quite some time for the service to spin up and actually be usable after spinning down when not in use.
I started with the obvious playbook: tuning how long a container stays alive between requests, and pinging it the moment someone loaded the page so the warmup was already running before they'd typed a question. Both helped, a little. What actually closed most of the gap was more structural. Forcing the data to load and CUDA to compile during image build instead of at request time, and baking Ollama's model loading into the image the same way, accounted for 80 to 90 seconds of the improvement on its own, though it required quickly coming to understand Modal's server implementations. With the page-load warmup running ahead of the actual question, and the container staying warm for ten minutes after, a burst of visitors mostly shares the cost of one cold start instead of each paying it individually. With this I went from an initial 115-second startup time to a technical 26 seconds that could average anywhere from 10 to 30 seconds based on warmup pings.
The output guardrail checks tone, not truth. It catches persona breaks or refusals — it doesn't check whether a confident answer is actually grounded in the retrieved context, claim by claim. A concrete instance of the same class of gap: dropping the old deduplication cap closed one problem but opened a narrower one. Nothing currently stops five fused chunks from being near-duplicates of the same narrow point, if a question is broad but doesn't cleanly decompose the way multi-query splitting expects. A groundedness check or reranking pass addresses the first; a lightweight diversity check on the final top five would address the second.
The interface decision, paying off.
My Rust library, hdvlibrary, isn't listed anywhere on this site or my resume — too small to earn one of a handful of homepage project slots. Ask the assistant directly, and it's a different story.
The same assistant that answers questions on the homepage — ask it something, or compare its answers to the logged exchanges below.
A few real exchanges, unedited, including ones that show the system's actual boundaries, not just its best answers: