Context
Municipal zoning codes, searched and answered on one machine. No cloud API for embeddings or generation, so the whole pipeline runs on local models on ordinary hardware. The corpus is five zoning PDFs, 1,070 pages, 4,854 chunks in a 90MB index.
The constraint that shaped everything: a confidently wrong answer about a setback rule is worse than no answer. So most of the work is not generation. It is knowing whether retrieval found the right page, and refusing to answer when it did not.
Architecture
Ingestion: PDFs are read page by page with PyMuPDF, falling back to OCR when a page has no text layer. Pages split at section headings before chunking, so a chunk rarely straddles two rules. Chunks cap at 800 characters with 150 overlap and the median lands at 603. Table rows carry their column labels into the text, because a borderless table flattened by reading order turns a setback table into nonsense. Each chunk's id is a hash of source, page, and position, so re-ingesting a corrected PDF updates only what changed instead of duplicating the corpus. Source and page travel in chunk metadata and become the numbered citations in the prompt.
Retrieval: a vector leg (nomic-embed-text over a persistent ChromaDB HNSW index) fused with a BM25 leg, weighted 0.25 to 0.75 because BM25 earns it on this corpus. Each leg pulls twice k candidates. Fusion ranks individual chunks rather than collapsing to one per document, top 3, cut to 1 when BM25 has a clear leader. Search can be scoped to selected PDFs, and a near-tie between two cities triggers a clarifying question instead of a guess.
Answering: llama3.2:3b drafts from the retrieved passages, then every claim is checked against the passage it cites. Unsupported sentences get repaired or dropped, and an answer with nothing to stand on abstains. The API streams stage status as server-sent events, then sends the checked answer.
The hard problem: retrieval you can audit
Most of the engineering went into measuring whether retrieval actually works.
The first benchmark was 50 queries and it was close to worthless. Every question carried the same source label, including questions about leases and mortgages, so it mostly measured itself. The replacement is 28 questions, each labeled with a file and a physical page, and each label checked against an excerpt that has to appear on that page before the question counts.
That harness settled three flaws the earlier build had:
- Tie collapse. Substring-count keyword scoring gave hundreds of chunks an identical score, so ranking silently fell back to insertion order. BM25 replaced it. Same harness, same questions, same index: keyword page hit@3 went from 57.1% to 82.1%.
- Ranking granularity. Keeping only the best chunk per source meant the top 3 could never return three passages from one ordinance, the wrong trade for a corpus that is effectively one huge document. Fusion now ranks chunks.
- O(corpus) keyword scans. Half fixed. The BM25 index is cached and rebuilt only when ingestion publishes a new version, instead of pulling every chunk out of the database per request. But a request still runs up to four scoring passes over all 4,854 chunks in memory. At this size that is 18ms. At 100k chunks it is a redesign.
With 28 questions each one is worth 3.6 points, and 8 of them were used while developing, so the number is a signal rather than a grade. A 42-item holdout exists for exactly that reason and has not been run yet.
Decisions and trade-offs
- Sealed by default, and tested that way. The default deployment runs every service on an internal Docker network behind a localhost gateway that drops its default route at startup. With it running, the backend could not resolve or reach an outside host, and a full ingest of all 1,070 pages plus a cited answer through the UI still worked. Vector-store telemetry is off in code and guarded by a test, and OCR models ship in the image instead of downloading on first use. It is isolation by configuration, not a hardware air gap: the gateway still sits on the bridge Docker needs to publish localhost ports, and no packet capture has been run.
- Provenance in metadata, and citations built from it. An earlier version prepended a source and page header into the chunk text itself, which only ever landed on the first chunk of a page. Metadata is what actually carries provenance through chunking, retrieval, and generation.
- Streaming traded for verification. The answer used to arrive token by token. It cannot now, because checking a claim means holding the whole draft first. First visible text lands in a median of 1.7 seconds on a laptop GPU, and that is a fair price for not showing a sentence that turns out to be unsupported.
- A 3B model on purpose. Generation quality matters less here than retrieval quality and the check that follows it.
Results and next steps
On the 28-question benchmark, the shipped hybrid retrieval puts the right page in the top 3 for 89.3% of questions and the right document for 100%, against 64.3% page hit@3 for the vector leg alone. 145 automated tests cover ingestion, retrieval, answering, the network configuration, and the frontend.
The honest gaps: the holdout has never been run, the citation judge agrees with human labels on only 1 of 5 partial-support cases, a query that names its own city can still trigger the clarifying question, and 3 of 5 latency probes needed the fallback draft. Next is passage-level labels, so the benchmark scores the chunk it retrieved and not just the page.