Most RAG demos answer confidently no matter what you ask. That's the easy part — retrieve some chunks, hand them to an LLM, let it generate something plausible. The hard part is getting the system to know when it doesn't have enough evidence, and to say so instead of quietly making something up. That's the problem Document Navigator is built around.
It's a locally-run assistant that answers questions over a PDF corpus, cites every claim as [filename.pdf:page], and — this is the part I actually care about — refuses to answer when the retrieved evidence is too weak to support a claim. No API keys, no cloud calls: FAISS for the vector index, MiniLM for embeddings, Ollama running qwen2.5:7b for generation.
The pipeline
Ingestion loads PDFs page-by-page, splits them into overlapping chunks with source and page metadata attached, embeds them with MiniLM, and persists the index to disk. At query time, retrieval runs top-k similarity search and converts FAISS's L2² distances into interpretable cosine scores — normalizing the embeddings up front means FAISS's inner product mathematically equals cosine similarity, so I get clean [0, 1] scores without a separate normalization pass.
Every retrieval gets logged as a structured trace before generation even runs, so I have a paper trail for any answer the system produces without needing to re-embed anything to debug it later.
The part that actually matters: the refusal gate
Before the LLM is ever called, the top-1 similarity score gets checked against a threshold. Below 0.25, the query is refused outright — no generation, no chance of the model quietly answering from its own training data instead of the corpus. Above that line, the retrieved chunks get assembled into a structured context block and handed to the model with separated system and human messages, treating the retrieved content as untrusted data rather than instructions. That single design choice does double duty: it's a cost optimization (skip the LLM call entirely when there's nothing to answer from) and a safety measure (it's also where prompt-injection attempts from inside the corpus get neutralized, since the LLM never confuses retrieved text for a system instruction).
Did it actually work? The eval harness says mostly yes
I didn't want to just eyeball a few outputs and call it done, so I built a 20-question evaluation set: 15 questions scoring answer quality, 5 adversarial questions designed to trigger the refusal path.
| Metric | Score |
|---|---|
| precision@1 (answer cohort) | 0.867 |
| precision@3 / precision@5 | 1.000 |
| citation accuracy | 0.800 |
| refusal correctness (safety cohort) | 1.000 |
| false refusal rate (answer cohort) | 0.000 |
| mean latency | 4.7s / query |
The number I was most nervous about was refusal correctness, since a system that refuses too eagerly is nearly as useless as one that hallucinates. All 5 adversarial rows — 3 out-of-scope questions, 2 prompt-injection attempts — triggered the pre-LLM refusal path correctly, and zero legitimate questions got falsely refused.
What I'd tighten next
- The 20-question eval set is small enough that these numbers are directional, not statistically robust.
- Adversarial robustness is only proven at the similarity-gating layer — all 5 safety rows got caught before reaching the LLM, so the second line of defense (separated system/human messages) hasn't actually been tested under real injection pressure yet.
- The app also supports session-scoped uploads — visitors can query their own PDFs without touching the evaluated corpus — but that upload path doesn't have adversarial test cases yet either.
None of that is a reason not to ship it — it's the difference between a demo and a system with a documented boundary. That boundary is exactly what I'd want to know as the person deciding whether to trust this on real documents.