A sequel to the architecture guide. Part 1 explained how the search engine was built. This one explains how we found out whether it was any good, tried four ideas to improve it, and shipped the two that actually worked.
If you read Part 1, you know the setup: I'd been collecting technical books for years — mostly Humble Bundle hauls — and had over 100 of them sitting on a hard drive, mostly unread. So I built a semantic search engine over the whole library, exposed as an MCP server Claude could query. Ask a question, get back real passages from Neal Ford or Vaughn Vernon, instead of an AI's vague recollection of them.
By the end of Part 1, it worked. But "it worked" and "it's good" are different claims, and I'd only ever checked the first one. This post is the story of taking the second one seriously.
The whole thing follows from one sentence, taken literally:
You cannot improve what you cannot measure.
The question that started this
When you type a question and passages come back, they look relevant — the human brain is generous that way. But three honest questions have no answer from eyeballing:
- If I change something, did it get better or worse?
- Better or worse for which kinds of questions?
- Is a change even worth its cost — slower, more memory, more complexity?
Answering any of these means turning a vague feeling ("seems good") into a number you can compare before and after a change. That number is an evaluation metric, and the machinery that produces it is an evaluation harness. Building a trustworthy one turned out to be most of the work.
The metric that lied to me
I already had a metric going in: Recall@5. Write down test questions, note in advance which book should answer each one, run the search, and check whether the right book shows up anywhere in the top 5 results. It's simple, reasonable, and it told me the system was at 92%.
That sounded great. It was also nearly useless.
Recall@5 is a yes/no question — was the right book somewhere in the top 5? It doesn't care if it was result #1 or result #5. It doesn't care if the other four results were junk. And once a metric is sitting at 92%, almost nothing you do moves it — there's no room left. A ruler that reads "about 92" no matter what you change can't tell you whether a change helped or hurt. This had already bitten the project once: an earlier tweak had been abandoned purely because the metric couldn't distinguish "helped" from "hurt."
The lesson generalizes well past search: a metric near its ceiling is blind. If your test always passes, it has stopped testing anything.
Building a real ruler
Before improving the engine, I needed a better ruler — one with three parts: a set of test questions with known answers, metrics that reward good ordering (not just presence), and a way to slice results so I could see where something helped.
Ground truth. I wrote 50 test questions in the voice of myself, an enterprise architect — and tagged each one along two axes:
- Subject: which topic it's about (event-driven architecture, domain-driven design, API governance, etc.)
- Complexity, on a four-level scale:
| Level | Type | Example | What it needs |
|---|---|---|---|
| L1 | Factoid | "What is a dead letter queue?" | One passage |
| L2 | Explanatory | "How does the circuit breaker work?" | One topic, explained |
| L3 | Comparative | "Choreography vs. orchestration?" | Weighing two things |
| L4 | Synthesis | "How do EDA and DDD fit together?" | Spans many passages |
I also flagged a special group: lexical-mismatch questions, where my words don't match the book's words — asking about an "event broker" when the book says "message broker." That group becomes the hero of the story later.
Two details separated a rigorous answer key from a sloppy one, and both bit me before I fixed them:
- Graded relevance. Not every "right" book is equally right — I scored a 3 for the primary source and 1–2 for acceptable secondary ones, so the metric could reward putting the primary source on top rather than treating any correct-ish hit as equally good.
- The duplicate-editions trap. My library has some books twice, under two IDs (first and second editions). If the answer key names one edition and the search returns the other, a naive scorer records a miss — punishing the engine for being right. Fixed by mapping both IDs to one canonical work before scoring. This turned out to be a recurring theme: the bugs are usually in the ruler, not the thing being measured.
Rank-aware metrics. The fix for Recall@5's blindness is measuring not just whether the right answer showed up, but where:
- success@1 / @5 / @10 — same idea as Recall@5, but at different cutoffs. success@1 ("was the very first result right?") is demanding and has plenty of room to move.
- MRR (Mean Reciprocal Rank) — for each question, score the reciprocal of the rank of the first correct result (rank 1 → 1.0, rank 2 → 0.5, rank 3 → 0.33...), then average. Rewards correct answers sitting near the top.
- nDCG — grades the whole top-10 ordering using the graded relevance scores, so putting the primary source first beats putting a secondary source first. Normalized 0–1, where 1.0 is a perfect ranking.
- pass@5 — the strict one. Only counts a hit if a top-5 result is from the right book and roughly the right page. Catches the gap between "found the right book" and "found the right paragraph."
Why so many numbers? Because they fail differently, and a change can help one while quietly hurting another.
Slicing. Because every question is tagged, the harness reports each metric per complexity level and for the lexical-mismatch group specifically. This is the difference between "the change helped overall" and "the change helped easy questions but hurt the hard ones" — an average can hide a disaster in a subgroup.
The baseline, once all this was built:
| success@1 | success@5 | MRR | nDCG | pass@5 | latency |
|---|---|---|---|---|---|
| 0.60 | 0.92 | 0.73 | 0.71 | 0.69 | 25 ms/query |
Notice what the richer metrics reveal that Recall@5 (≈ success@5 = 0.92) couldn't: the right answer was findable 92% of the time, but it was the #1 result only 60% of the time. That 32-point gap was the room to improve — room that had been hiding in plain sight.
Four ideas, tested like experiments
The original engine does dense retrieval: turn the query into a meaning-vector, return the passages whose vectors are nearest. Good at meaning, with known blind spots. I tried four fixes.
1. Reranking. The dense retriever is a bi-encoder — it embeds the query and each passage separately, then compares vectors. Fast (every passage vector is precomputed), but a little myopic, since it never looks at query and passage side by side. A reranker is a cross-encoder: it reads the query and one passage together and scores relevance directly — far more precise, but it has to run once per candidate, so it's slow. The standard fix: use the fast retriever to pull a shortlist (say 30 candidates), then let the slow, precise reranker re-order just those.
Think of it as a librarian who pulls 30 plausible books off the shelf from memory, followed by a specialist who actually reads the first page of each and puts the best on top.
2. Hybrid search. Dense retrieval is blind to exact words — if someone searches a rare term and the passage uses that literal term, dense search can dilute it into "general meaning" and miss it. The classic fix is BM25, a decades-old keyword algorithm that scores passages on word overlap. Hybrid search runs both and merges the ranked lists using Reciprocal Rank Fusion (RRF): throw away the incompatible scores, keep only rank positions, and sum 1 / (60 + rank) across both lists. A passage that ranks decently in both beats one that ranks #1 in only one.
3. Query transformation. Sometimes the question itself is the problem. This approach uses a model to rewrite the query into several cleaner variations, searches each, and fuses the results with RRF — the hope being that several phrasings catch what one phrasing would miss.
4. The query-quality gate. The most ambitious idea: before searching at all, a model triages the query into PASS (search as-is), CLEAN (rewrite messy input, then search), or REJECT (refuse gibberish or off-topic questions outright). Appealing in theory. Whether it works is exactly the kind of thing a ruler is for.
What actually happened
Every idea was run as a controlled experiment: one change at a time, against the same 50 questions and the same metrics, with results checked at both the average and the slice level.
| technique | success@1 | success@5 | MRR | nDCG | pass@5 | lexical MRR | latency |
|---|---|---|---|---|---|---|---|
| baseline (dense) | 0.60 | 0.92 | 0.73 | 0.71 | 0.69 | 0.43 | 25 ms |
| reranker | 0.74 | 0.90 | 0.82 | 0.77 | 0.79 | 0.60 | 425 ms |
| query transformation | 0.52 | 0.90 | 0.66 | 0.66 | 0.60 | 0.47 | 57 ms |
| hybrid (BM25 + dense) | 0.68 | 0.96 | 0.81 | 0.79 | 0.81 | 0.80 | 123 ms |
| hybrid → reranker | 0.78 | 0.96 | 0.86 | 0.82 | 0.83 | 0.67 | 485 ms |
The reranker worked, at a price. It pulled the right answer into the #1 spot far more often (0.60 → 0.74), but it can only re-order what dense retrieval already found — it can't improve raw recall — and it's roughly 16× slower. The slice view also caught something an average would've hidden: it actively hurt the hardest synthesis-level questions, because the reranker model is trained for short, focused questions and gets confused by broad, multi-part ones. A real win, but not a free one.
Query transformation backfired. This is the kind of result you'd never catch by eyeballing — it made things worse across the board (MRR 0.73 → 0.66). The cause was instructive: the rewrites came from a small local model that didn't know the domain's vocabulary. Asked about a "golden path," it produced generic phrasing like "team best practices" instead of the book's actual term, "paved road." Because RRF weights every list equally, three mediocre rewrites could out-vote the one good original question.
Hybrid search was the star. It was the only technique to genuinely improve recall (0.92 → 0.96) rather than just reshuffle existing results, because BM25 surfaces passages dense retrieval never found in the first place. And it did exactly what it was built for on exactly the group it targeted: on lexical-mismatch questions, MRR jumped from 0.43 to 0.80. The "golden path" question that stumped query transformation? BM25 matched the literal word and nailed it.
Hybrid → reranker, stacked, was the best of all. Hybrid finds a rich, complete candidate set; the reranker orders it well. Together they won on six of eight metrics, pushing success@1 to 0.78. One subtlety the slices revealed: stacking the reranker on top slightly undid hybrid's lexical-mismatch win, because the reranker reintroduces a meaning-based judgment that can demote a keyword-matched hit. Best overall, but the components genuinely interact — nothing here composes for free.
The gate, and the value of being wrong
The query-quality gate deserves its own note, because measuring it taught me the most of anything in this project.
Its first score was 28% — which looked like the model had simply failed. It hadn't. My prompt contained a literal { character from a JSON example, and the code filling in the prompt template interpreted it as a placeholder and crashed silently every time, falling back to doing nothing. The tool was broken, not the model. One-line fix, and the score jumped to 61%.
Lesson: when a result looks too bad, suspect your measurement apparatus first.
But even at 61%, the gate failed the test that actually mattered. A gate's first duty is don't harm good questions — and it did: wrongly rejecting 6 of 50 perfectly good questions, and needlessly rewriting 24 more. End to end, the gated system was worse than having no gate at all.
That's a negative result, and it turned out to be genuinely valuable, for two reasons. First, it stopped a plausible-sounding regression from shipping. Second, it revealed a pattern across all four experiments: every attempt to have a small local model rewrite the input (query transformation, the gate) made things worse, while every attempt to add a complementary signal (BM25) or re-order existing results (the reranker) helped. On this hardware, with models this size: augment retrieval, don't let a weak model tamper with a good question.
Shipping the winner
Winning an experiment isn't the same as being in production — the experiments all ran inside the evaluation harness, while the live server still ran plain dense search. The last step was moving hybrid → reranker into the server itself:
The honest trade-offs: a query now takes about half a second instead of 25 milliseconds, and the server uses more memory. For an interactive personal tool, sub-second is completely fine — and I only know the cost precisely because the harness measures latency alongside quality, not as an afterthought.
One more small lesson worth keeping: BM25 needs an index built over all ~50,000 passages, which takes several seconds and doesn't change between restarts since the books rarely change. So I serialize it — pickle the built index to disk and load it back on later startups, cutting startup time from ~11s to ~6s. Three details made that safe: a fingerprint hash of the book catalog to detect staleness and trigger a rebuild automatically; a fail-safe so a load failure just triggers a rebuild instead of crashing; and the rule that you only ever unpickle a file you created yourself, since loading a pickle can execute arbitrary code.
Why this matters more than the numbers
If four things are worth keeping from this project, it's these:
Measurement is the product. The most valuable thing I built isn't the faster search — it's the ruler. Any future idea can now be judged the same way, in minutes, with evidence. Teams that can measure improve; teams that can't, guess.
A metric near its ceiling is blind. Recall@5 at 0.92 felt like success and told me nothing. The real gains were hiding in metrics that still had room to move.
Averages lie; slices tell the truth. Every real finding here came from slicing — the reranker helping easy questions while hurting hard ones, hybrid fixing exactly the vocabulary-mismatch group it targeted. Always ask: for whom did this help, and whom did it hurt?
Negative results are results. Two of four ideas made things worse, and proving that with numbers was as valuable as shipping the two that worked. It stopped a plausible-sounding regression from reaching production, and it surfaced a real, transferable principle in the process.
The search is meaningfully better now — the right answer lands in the #1 spot 78% of the time, up from 60%, and questions phrased in the "wrong" words finally work. But the durable win isn't the number. It's that I can prove it, and prove the next thing too.
This is Part 2 of a series on building a personal semantic search engine over a technical book library, exposed as an MCP server. Part 1 covers the initial build: chunking, embedding, indexing, and retrieval.
Comments
Post a Comment