RAG Book Assistant — grounded answers from a 400-page book

the problem

Ask an LLM about a specific book and it will answer — confidently, fluently, and often wrongly. The model has fuzzy memories of the text at best. The assistant needed to answer questions about a book using only the book, and show its sources.

architecture

              ingestion (once)                     query time
 ┌──────────┐   ┌──────────┐   ┌─────────┐    ┌──────────────┐
 │  book     │──▶│ structure │──▶│ vector   │◀───│ standalone    │
 │  (text)   │   │ chunking  │   │ store    │    │ query rewrite │
 └──────────┘   └──────────┘   └────┬────┘    └──────▲───────┘
                                     │ top-20          │
                                     ▼                 │ user question
                               ┌──────────┐      ┌────┴─────┐
                               │ reranker  │─top-5▶ LLM with  │
                               └──────────┘      │ citations │
                                                 └──────────┘

the three things that actually mattered

1. Structure-aware chunking. Fixed-size chunks cut sentences and chapters mid-thought, and retrieval quality was terrible. The fix: split on chapters → sections → paragraphs first, fall back to size limits only inside a paragraph, and overlap adjacent chunks ~15-20%. Every chunk carries its chapter title as metadata, so answers can cite where they came from.

2. Query rewriting before retrieval. Real questions are conversational: "what happened to him after that?" embeds to nothing useful. A cheap LLM pass rewrites the question into a standalone query using chat history, and retrieval quality jumps.

3. Retrieve wide, rerank narrow. Pull 20 candidates by vector similarity, rerank, keep 5. The reranking step costs milliseconds and fixes most "right book, wrong page" misses.

the honesty constraint

The most important line in the prompt: if the retrieved passages don't contain the answer, say so. Without it the model fills gaps with plausible fiction — and an assistant that invents plot points is worse than no assistant. With it, "I don't know, that's not covered in this book" became a feature users trusted.

what i'd do differently

• Eval set first. I tuned chunking by feel; 30 question/answer pairs with known source passages would have made every change measurable.
• Hybrid retrieval — BM25 + vectors. Names and rare terms are exactly where pure vector search is weakest, and books are full of them.

↗ view the code