Context
A document Q&A system with one hard constraint: nothing leaves the machine. No cloud APIs, no third-party data exposure. Real municipal zoning ordinances, searched and summarized entirely on local hardware, right-sized to run a 1B-param model on CPU.
Architecture
Ingestion: PDFs load page-by-page (pypdf), and each chunk gets a
Source: <file> | Page: <n> header prepended into the chunk text itself, so
provenance survives chunking, embedding, retrieval, and generation no matter what
the pipeline does downstream. Chunks are 800 characters with 150 overlap, embedded
with nomic-embed-text via Ollama into a persistent ChromaDB (HNSW) index.
Query: a FastAPI endpoint runs hybrid retrieval: a vector leg (similarity
search over 2k candidates, distance-normalized) fused 50/50 with a keyword leg,
collapsing to the best chunk per source document, top-k of 3. Generation runs on
llama3.2:1b, streamed token-by-token to a React UI over a streaming fetch
(ReadableStream + TextDecoder), so first tokens appear while the model is
still generating.
Current scale: 3,796 chunks across 9 sources in a 50MB index, and ~700 lines of Python.
The hard problem: retrieval you can audit
The interesting engineering here wasn't making retrieval work. It was measuring whether it actually works, and being honest about the answer.
I built a 50-query benchmark over the zoning corpus. Key finding: what it measures is source-level attribution (does the right document appear in top-k?), which is not the same as passage-level relevance. The keyword leg, benchmarked in isolation, hits 34% @k=3 and 50% @k=5. That number isn't impressive. It's diagnostic, and it pointed straight at three concrete design flaws:
- Tie collapse. Substring-count keyword scoring gives hundreds of chunks an identical score for a common query, so ranking silently falls back to insertion order. This is the strongest concrete argument for real BM25/IDF weighting.
- Ranking granularity vs. corpus shape. Keeping only the best chunk per source means top-3 can never return three passages from the same ordinance, the wrong trade for a corpus that is effectively one huge document.
- O(corpus) keyword scans. The keyword leg pulls every chunk into Python on each request. Fine at 3,796 chunks, a wall at 100k.
Decisions & trade-offs
- Provenance in-band, not in metadata. Stamping source/page into the text itself is crude and costs tokens, but it's the one place citation info cannot be lost, no matter how retrieval or prompting changes.
- Right-sizing over horsepower. A 1B model with CPU fallback keeps the air-gap promise real on ordinary hardware, and the retrieval quality work matters more than model size.
- Honest metrics over impressive ones. Publishing the diagnostic baseline and its failure modes beats quoting a number I can't reproduce.
Results & next steps
A working single-user prototype with verified provenance on every answer. Next, in order: BM25/IDF to fix tie collapse, passage-level ground truth so the benchmark measures relevance rather than attribution, and first-token latency measurement for the streaming path.