RAG Interview Questions
Ten questions on production RAG: architecture, chunking, hybrid search, reranking, evaluation, isolation and when not to use RAG.
Q1Explain a production RAG architecture.show answer
In plain words: Documents are split and stored with details like owner and permissions. At question time you search, filter by permission, pick the best pieces and answer with sources.
Full answer · Ingestion: documents arrive asynchronously through a queue, are normalized, chunked in a structure-aware way with metadata (tenant, ACL, source, section, timestamps), embedded and indexed; jobs are idempotent and observable. Query: authenticate, optionally rewrite the query, run hybrid retrieval with tenant/ACL filters applied before ranking, rerank the top candidates, build a bounded context, generate an answer constrained to the sources with citations and a refusal path, then log traces. Evaluate retrieval and generation separately, and handle updates and deletions.
Q2How do you choose chunk size?show answer
In plain words: Split by headings and paragraphs. Test a few sizes on real questions. Pick the size that finds the right text most often.
Full answer · There's no universal size, so start from the content: split on structure (headings, paragraphs, functions), keep tables and code intact, add a small overlap and carry the heading path as context. Then tune empirically — compare candidate strategies on a labelled question set using recall@k and answer groundedness, and check cost. Too small separates context; too large reduces precision and wastes tokens. Parent–child chunking (retrieve small, return the surrounding section) often helps.
Q3What is hybrid retrieval?show answer
In plain words: It searches by meaning and by exact words together, then merges the results. It catches things either method alone would miss.
Full answer · Combining keyword search (strong on exact identifiers, names, error codes) with vector search (strong on meaning), then merging the rankings — commonly with reciprocal rank fusion, which works on rank positions rather than incomparable scores. It improves recall on queries that need both exact and semantic matching, and is a common production default.
Q4When would you use reranking?show answer
In plain words: When the right result is found but not at the top. A second, smarter pass reorders them so only the best few reach the AI.
Full answer · When first-stage retrieval has decent recall but poor precision at the top: retrieve ~30–50 candidates cheaply, score them with a stronger cross-encoder or model, and pass only the best 5–10 to the generator. It improves answer quality and reduces context cost at the price of added latency and compute, so measure the gain on your eval set and skip it if the baseline already meets targets.
Q5How do you evaluate retrieval separately from generation?show answer
In plain words: Test the search with questions that have known answers. Then test whether the answer used the found text correctly. This shows which part to fix.
Full answer · Build a labelled set of questions with the passages that contain the answers, and measure retrieval alone with recall@k, precision@k, MRR or nDCG. Then evaluate generation given the retrieved context — groundedness (every claim supported), completeness and correct refusal when evidence is missing. Separating them tells you whether to fix chunking, ranking or prompting, and lets you add a regression case whenever a retrieval bug is found.
Q6How do you enforce tenant isolation in vector search?show answer
In plain words: Put the customer filter inside the search itself, before ranking. Then test it by trying to read another customer's data.
Full answer · Carry tenant and user identity from the request into retrieval and apply metadata filters (tenant, ACL, document type, region) as part of the query so ranking only ever sees permitted chunks; never filter after retrieval or in the UI. Verify the same isolation in tool calls, SQL queries, caches, logs and async jobs, test it with adversarial cross-tenant queries, and consider separate indexes or encryption keys for sensitive tenants.
Q7A RAG answer is wrong. How do you debug it?show answer
In plain words: Check each step: was the document stored, was it found, was it ranked high and did the AI use it? The first failing step is the one to fix.
Full answer · Trace the request: was the relevant document ingested and chunked sensibly, did retrieval return it (recall), was it ranked high enough to reach the context, did the context contain conflicting or stale text, and did the model use it faithfully? Each answer points to a different fix — ingestion, chunking, ranking, freshness or prompt. Add the failing case to the eval set once fixed.
Q8How do you handle documents that change or are deleted?show answer
In plain words: Keep the original files separate from the search copies, track versions and remove the old copies. Test that deleted text no longer appears.
Full answer · Keep immutable source documents separate from derived chunks and embeddings, store a content hash and version, re-embed only what changed, and remove or tombstone vectors when sources are deleted, including from caches and backups according to policy. Verify with tests that deleted content can no longer be retrieved, and expose freshness metadata so answers can cite the date of the source.
Q9When is RAG the wrong approach?show answer
In plain words: When the question is about exact numbers or records, use a database. Use RAG for text knowledge, not for everything.
Full answer · For structured, transactional questions ('what were sales last month?') use SQL or an API; for relationship queries consider a graph; when the knowledge is small and stable, put it in the prompt; and when you need a behavior or style change rather than knowledge, consider fine-tuning. Choose retrieval by data type and correctness requirements before choosing a vector database.
Q10How do you reduce hallucinations in a RAG system?show answer
In plain words: Make the search better, tell the AI to answer only from the sources and cite them, and check the claims. It reduces mistakes but never removes them.
Full answer · Improve retrieval quality first; instruct the model to answer only from the supplied sources with citations and to say it doesn't have enough information otherwise; verify that each cited passage supports its claim; keep context small and relevant; use deterministic checks for numbers and entities; and evaluate groundedness continuously. RAG reduces hallucination but doesn't eliminate it, since irrelevant or contradictory retrieval can still mislead the model.