Context
A document Q&A system with one hard constraint: nothing leaves the machine. No cloud APIs, no third-party data exposure. Somewhere that real municipal zoning ordinances can be searched and summarized entirely on local hardware, right-sized to run a 1B-param model on CPU.
Architecture
Ingestion: PDFs load page-by-page (pypdf), and each chunk gets a
Source: <file> | Page: <n> header prepended into the chunk text itself, so
provenance survives chunking, embedding, retrieval, and generation no matter what
the pipeline does downstream. Chunks are 800 characters with 150 overlap, embedded
with nomic-embed-text via Ollama into a persistent ChromaDB (HNSW) index.
Query: a FastAPI endpoint runs hybrid retrieval: a vector leg (similarity
search over 2k candidates, distance-normalized) fused 50/50 with a keyword leg,
collapsing to the best chunk per source document, top-k of 3. Generation runs on
llama3.2:1b, streamed token-by-token to a React UI over a streaming fetch
(ReadableStream + TextDecoder), so first tokens appear while the model is
still generating.
Current scale: 3,796 chunks across 9 sources in a 50MB index, and ~700 lines of Python.
The hard problem: retrieval you can audit
Most of the engineering went into measuring whether retrieval actually works.
I built a 50-query benchmark over the zoning corpus, though it measures source-level attribution (does the right document appear in top-k?) rather than passage-level relevance, and those are different things. The keyword leg, benchmarked in isolation, hits 34% @k=3 and 50% @k=5. I keep that number on the page because it pointed straight at three design flaws:
- Tie collapse. Substring-count keyword scoring gives hundreds of chunks an identical score for a common query, so ranking silently falls back to insertion order. This is the strongest concrete argument for real BM25/IDF weighting.
- Ranking granularity vs. corpus shape. Keeping only the best chunk per source means top-3 can never return three passages from the same ordinance, the wrong trade for a corpus that is effectively one huge document.
- O(corpus) keyword scans. The keyword leg pulls every chunk into Python on each request. That's fine at 3,796 chunks, but impossible to manage at 100k.
Decisions & trade-offs
- Provenance in-band, not in metadata. Stamping source/page into the text itself is crude and costs tokens, but it's the one place citation info cannot be lost, no matter how retrieval or prompting changes.
- A 1B model on purpose. Running small with CPU fallback keeps the air-gap promise real on ordinary hardware, and the retrieval quality work matters more than model size.
Results & next steps
It works as a single-user prototype, and every answer carries verified provenance. Future updates include implementing BM25/IDF to fix tie collapse, passage-level ground truth so the benchmark measures relevance rather than attribution, and first-token latency measurement for the streaming path.