What I Learned Building a RAG Book Assistant

RAG demos are easy. RAG that answers correctly about a 400-page book is a different sport entirely.

I built a book assistant — ask questions about a book, get grounded answers with the relevant passages — and most of what I learned had nothing to do with the LLM.

Chunking is the whole game

My first version split the book into fixed 1000-character chunks. Retrieval was terrible. Chapters got cut mid-sentence, context evaporated, and the model confidently answered from fragments.

What worked better:

• Split on structure first — chapters, then sections, then paragraphs. Fall back to size limits only inside a paragraph.
• Overlap generously. A 15–20% overlap between adjacent chunks rescued a surprising number of "the answer was right at the boundary" failures.
• Keep metadata. Storing the chapter title with every chunk meant the model could say where an answer came from, which users trust far more than a bare answer.

Retrieval quality > model quality

Swapping a small model for a bigger one barely moved answer quality. Improving retrieval moved it a lot. Two cheap wins:

1. Rewrite the user's question before embedding it. "What happened to him after that?" retrieves nothing; the rewritten standalone question retrieves the right chapter.
2. Retrieve more, then rerank. Pull 20 candidates, keep the best 5. The reranking step costs milliseconds and fixes most "right book, wrong page" errors.

Say "I don't know"

The most important prompt instruction I added: if the retrieved passages don't contain the answer, say so. Without it, the model fills gaps with plausible fiction — and a book assistant that invents plot points is worse than no assistant at all.

RAG isn't magic. It's plumbing. But well-laid plumbing is what makes the magic trick land.