RAG & Retrieval

Answer

Start by proving where the failure happens — retrieval, context assembly, or generation. Correct document in the top-k does not mean the model actually used it.

Step 1: Inspect what the LLM actually received

Log the final prompt: system instructions, retrieved chunks (with scores and IDs), and the user question. Common issues:

• The right doc is retrieved but ranked low and truncated out of context
• The answer lives in a different chunk than the one surfaced
• Metadata or headers were stripped, removing critical context
• Multiple conflicting chunks confuse the model

Step 2: Isolate retrieval vs generation

Run a retrieval-only check: can a human (or a smaller model) answer correctly from the retrieved chunks alone? If yes, the bug is in generation or prompting. If no, retrieval or chunking is still broken even if the "right document" appears somewhere in results.

Step 3: Check generation settings

• Temperature too high for factual Q&A
• Weak or missing citation instructions ("answer only from context; say I don't know if unsupported")
• Context window overflow silently dropping early chunks
• Model ignoring retrieved text because system prompt prioritizes general knowledge

Step 4: Validate chunk boundaries

The correct document may be retrieved while the specific sentence with the answer was split across chunks or buried in noise (navigation, boilerplate, legal disclaimers).

Step 5: Add tracing per request

Track: query → rewritten query → top-k IDs → chunks in prompt → model output → user feedback. One broken link in that chain explains most "right doc, wrong answer" cases.

Strong interview framing: "I'd verify the model saw the evidence, then test whether a human could answer from that evidence, then tune prompting and chunking before blaming the LLM."

Answer

"Don't hallucinate" is vague. Models need constraints, evidence, and escape hatches.

Ground answers in retrieved context

• Require citations to chunk IDs or source URLs in every response
• Instruct: "If the context does not support an answer, respond with: I don't have enough information in the provided documents."
• Post-process: reject answers with no citation or citations that don't match retrieved chunks

Reduce generation freedom

• Lower temperature (0–0.3) for factual tasks
• Use structured output (JSON with answer, sources, confidence) for easier validation
• Prefer models with strong instruction following for enterprise Q&A

Improve retrieval quality

Hallucinations often start when retrieval returns weak or empty context — the model fills gaps from parametric memory. Fix recall and relevance first.

Add verification layers

• Self-check prompt: "List claims in your answer and quote the supporting sentence from context"
• Secondary model or rules engine to flag unsupported claims
• For high-stakes domains: human review or deterministic checks (dates, numbers against a database)

Measure and iterate

Build an eval set with:

• Answerable questions (should cite context)
• Unanswerable questions (should refuse)

Track hallucination rate separately from fluency. Prompt tweaks only matter when measured against these buckets.

Interview soundbite: Grounding, citations, refusal behavior, and automated verification — not louder disclaimers.

Answer

High recall / low precision means you find something relevant but bury it in noise. The fix is narrowing the candidate set and re-ordering it.

1. Add a reranker

Retrieve top 50–100 with bi-encoder (vector search), then rerank with a cross-encoder (Cohere Rerank, bge-reranker, etc.) and keep top 5–10. This is the highest-leverage precision upgrade for most teams.

2. Tighten the first-stage retrieval

• Raise similarity threshold — discard low-score matches
• Reduce top-k before reranking to control cost, then rerank a fixed pool
• Filter by metadata (product, date, department, doc type) before vector search

3. Improve embeddings and chunk quality

• Embed cleaned text (remove headers, footers, duplicate boilerplate)
• Use domain-fine-tuned embeddings if generic models confuse similar policies
• Smaller, semantically coherent chunks improve embedding signal

4. Query transformation

• HyDE, query expansion, or LLM rewrite to align user language with document language
• Decompose multi-part questions into sub-queries

5. Negative mining from production

Log queries where users re-ask or thumbs-down. Add those failure pairs to eval sets and tune thresholds / filters.

6. Hybrid retrieval

Combine BM25 + vectors (see next question) when users use exact product names, error codes, or policy numbers.

Metric shift: Optimize MRR@k and precision@k, not recall alone. Interviewers want trade-off awareness, not infinite recall.

Answer

Use hybrid search when users and documents share exact terminology that embeddings alone mishandle.

Choose hybrid when:

• Queries contain SKUs, error codes, legal citations, API names, person names, or acronyms
• Documents use precise vocabulary that must match literally ("Section 409A", "CVE-2024-1234")
• Users search with keywords, not natural-language paraphrases
• Your corpus mixes short factual snippets (great for BM25) with long conceptual prose (great for vectors)
• Pure vector search returns semantically related but wrong docs (same topic, different entity)

Pure vector search is enough when:

• Questions are conceptual ("How does our refund policy work?")
• Wording varies widely between query and document
• Corpus is small and reranking already delivers strong precision

How to combine them

Common patterns:

• Parallel retrieval: run BM25 and vector, merge with RRF (Reciprocal Rank Fusion)
• Weighted score fusion with tuned weights per collection
• Cascade: BM25 filter → vector rerank within subset (or vice versa)

Operational note

Hybrid adds indexing complexity (sparse + dense indexes) and tuning surface. Ship hybrid when eval data shows keyword misses that vectors cannot fix — not by default.

Interview line: "Hybrid when exact tokens matter; vectors when meaning matters; rerank either way."

Answer

There is no universal chunk size — you optimize for retrieval hit rate and answer completeness on a representative eval set.

Start with document structure

• Structured docs (policies, FAQs): chunk by heading / section; size follows semantics (200–800 tokens typical)
• Narrative docs: sliding windows with overlap
• Code / tables: preserve logical blocks; naive token splits break syntax and tables

Use overlap to prevent boundary loss

Overlap 10–20% of chunk size so sentences spanning boundaries appear whole in at least one chunk. Increase overlap when eval shows correct answers split across adjacent chunks.

Run a chunking sweep

Hold embedding model and retrieval pipeline fixed. Test matrix e.g.:

Measure on 100–500 real questions:

• Recall@k — is the answer in retrieved set?
• Answer correctness end-to-end
• Latency / cost — larger chunks mean fewer chunks but more tokens per prompt

Watch failure modes

• Too small: fragments lose context ("it" has no antecedent)
• Too large: embeddings average away specificity; wrong section dominates
• Too little overlap: boundary facts vanish
• Too much overlap: redundant storage, repeated content in prompts

Production shortcut

Start 512 tokens / 64 overlap, evaluate, then adjust per content type. Enterprise KBs often end up with multiple chunking strategies by document class.

Interview framing: Chunking is an empirical retrieval problem, not a constant you guess once.

Answer

Full rebuilds do not scale. Treat the index as a versioned projection of source documents with incremental maintenance.

1. Stable document IDs

Each source page/file has a canonical ID. Chunks inherit doc_id + chunk_index. Updates target IDs — delete old vectors for that doc, insert new ones.

2. Change detection pipeline

• Webhooks or scheduled crawls emit create / update / delete events
• Compare content hash or last_modified — skip unchanged files
• Parse diffs at section level when possible to avoid re-embedding unchanged chunks

3. Incremental vector upserts

Vector DBs (Pinecone, Weaviate, pgvector, Qdrant) support delete-by-filter on doc_id then upsert new embeddings. Workflow:

1. Delete vectors where doc_id = X
2. Re-chunk and embed only changed doc
3. Upsert new vectors with fresh metadata (version, indexed_at)

4. Tombstone deletes

When docs are removed, delete by doc_id immediately. Avoid orphan vectors polluting retrieval.

5. Background reconciliation

Nightly job compares source catalog vs index manifest — fixes missed events. Full rebuild remains a disaster-recovery option, not daily ops.

6. Staging and blue-green index

For large corpora: build new index version in background, swap alias when validation passes — zero-downtime refresh without per-query inconsistency.

7. Metadata for freshness

Store source_updated_at on chunks. Optionally boost recent docs in reranking for time-sensitive domains.

Interview summary: Event-driven incremental upserts keyed on document ID, plus periodic reconciliation — not nightly full re-embed.

Answer

Shipped RAG is not proven RAG. Evaluation must be offline (before release) and online (in production).

Offline eval foundation

Build a dataset of real user questions with:

• Gold answers or acceptance criteria (human-written)
• Gold source documents or chunk IDs
• Unanswerable cases (should refuse)

Core metrics

Faithfulness matters most

A fluent wrong answer fails. Use:

• LLM-as-judge with rubric: "Is every claim supported by retrieved context?"
• Citation match: linked chunk actually contains the claim
• Abstention rate on unanswerable questions (should not invent)

Compare against baselines

• No retrieval — same model, no context
• Previous pipeline version
• Human expert answers on a sample

RAG must beat baselines on correctness, not just user delight at speed.

Online monitoring

• Thumbs up/down, re-query rate, escalation to human
• Sample production traces for weekly human review
• Track drift when docs or models change

Regression gates in CI

Block deploy if offline eval drops on:

• Faithfulness
• Recall@5
• Refusal accuracy

Interview close: Retrieval metrics alone lie — measure grounded correctness end-to-end against baselines, then monitor in production.

Answer

Treat every incorrect answer as a replayable incident with a full trace ID.

Step 1: Pull the request trace

Reconstruct the exact session:

• User query (and conversation history)
• Model and prompt version
• Rewritten / expanded query
• Retrieved chunk IDs, scores, and full text injected
• Tool calls and API responses
• Final model output and latency per stage

Without this, debugging is guesswork.

Step 2: Classify failure mode

Step 3: Reproduce offline

Replay the same inputs in staging. Toggle one variable at a time:

• Swap retrieval off → model-only baseline
• Inject gold chunks manually → isolate generation
• Re-run retrieval only → inspect rank and chunk boundaries

Step 4: Check recent changes

Deploy of new model, prompt edit, index rebuild, or doc deletion often correlates with spike in reports.

Step 5: Close the loop

• Add case to regression eval set
• Fix root cause (re-chunk, re-embed, prompt, reranker)
• Verify fix on trace + neighbors (similar queries)

User communication

Acknowledge error, log ticket ID, avoid exposing internal trace details — use internal tools for RCA.

Interview line: Full trace replay, classify retrieval vs generation vs staleness, reproduce with ablations, add to regression set.

Answer

Use controlled ablation — change one layer while holding others fixed.

Retrieval problem signals

• Gold document exists in corpus but not in top-k
• Right doc retrieved but wrong chunk (answer split across boundaries)
• Query-document vocabulary mismatch (jargon, acronyms)
• Fixing with better embeddings, hybrid search, or query expansion fixes answer

Test: Human answers correctly from top-k chunks → if human fails, retrieval. If human succeeds and model fails → not retrieval.

Prompt problem signals

• Retrieved context contains the answer verbatim but model ignores it
• Model answers from general knowledge despite contrary context
• Changing system instructions (cite only context, lower temperature) fixes behavior
• Inconsistent format or policy violations while facts are present

Test: Same chunks + different prompt → output changes dramatically.

Model problem signals

• Prompt and context are correct; multiple models on same prompt disagree
• Systematic reasoning errors (math, multi-hop logic) even with perfect context
• Weak instruction following across tasks — not isolated to one query type
• Upgrading model tier fixes without retrieval/prompt changes

Test: Swap model only; retrieval and prompt frozen.

Decision flowchart

Gold doc in top-k?
  No  → Retrieval
  Yes → Human can answer from chunks?
          No  → Retrieval (chunking/ranking)
          Yes → Change prompt fixes it?
                  Yes → Prompt
                  No  → Model (or task needs tools)

Log ablation results per incident — patterns emerge quickly in production.

Interview close: Ablation beats intuition — prove whether evidence was present, whether instructions were followed, then swap model last.

Answer

Don't take top-20 by vector score and truncate — compress, rerank, and diversify before the LLM sees anything.

Step 1: Rerank aggressively

• Cross-encoder or LLM reranker on top-20 → pick best 8 by query-chunk relevance, not bi-encoder score alone
• Bi-encoder recall is broad; reranker precision wins context slots

Step 2: Deduplicate and merge

• Near-duplicate chunks from same doc → merge into one passage
• Keep highest-scoring sentence windows; drop boilerplate headers/footers

Step 3: Maximize information density

• Prefer chunks that contain entities, numbers, dates, definitions matching the query
• Drop navigational fluff and legal preambles (classify chunk type if needed)

Step 4: Diversity when multi-doc

• MMR (maximal marginal relevance) — avoid 8 chunks saying the same thing from one doc
• Ensure coverage of distinct sub-questions in multi-intent queries

Step 5: Summarize overflow

• Map-reduce: summarize groups of low-rank chunks with cheap model → one synthesis chunk competes for slot 8
• Or extract bullet facts per chunk via small model, feed fact list not raw text

Step 6: Dynamic budget

• Simple factual query → 3–4 chunks may suffice; save tokens for answer
• Complex comparison → allocate more slots, shorten system prompt elsewhere

Measure

Offline eval: answer quality vs using all 20 (oracle) vs naive top-8 truncate. Good selection should be within few points of oracle at fraction of tokens.

Interview line: Rerank to 8, dedupe, diversify sources, summarize the long tail — never silent truncate by vector order alone.

Answer

Isolation must be enforced in storage, index, and query paths — not by hoping the LLM respects instructions.

Data plane isolation

Option A — Separate index per tenant (strongest) - Dedicated collection/namespace in vector DB per tenant_id - Query always includes tenant_id filter at API level - S3 prefixes or separate buckets per tenant for raw docs

Option B — Shared index with mandatory metadata filter - Every vector tagged tenant_id; DB rejects queries without filter - Risk: misconfiguration leaks cross-tenant — needs integration tests and audit

Option C — Dedicated infra for enterprise tier - Separate VPC/cluster for regulated customers

Ingestion

• Upload pipeline tags tenant at ingest; no shared embedding batch that mixes tenants without scoping
• Encryption at rest per tenant keys (KMS) where required

Application layer

• Auth token binds tenant_id — server derives scope, never trust client-supplied tenant in body alone
• Retrieval service API: search(query, tenant_id) — filter enforced in one code path

Embedding and models

• Shared embedding model is OK; vector namespaces are not
• Optional: tenant-specific fine-tuned embedders for enterprise only

Testing isolation

• Automated tests: Tenant A query must never return Tenant B doc IDs
• Chaos: attempt IDOR on doc/chunk IDs in API

Ops

• Per-tenant quotas, audit logs, delete/export for GDPR
• Reindex jobs scoped to single tenant

Interview close: Namespace-per-tenant indexes, tenant-bound auth at retrieval API, encryption and quota per customer, and automated cross-tenant leak tests.

Answer

Citations and explanations require structured outputs, chunk-addressable retrieval, and post-generation verification.

Retrieval design

• Chunk with stable IDs and metadata: doc_id, page, section, url
• Return chunks to orchestrator with IDs — model cites chunk_id, not free-form URLs

Prompt contract

Require structured response:

{
  "answer": "...",
  "citations": [{"chunk_id": "c_42", "quote": "exact supporting sentence"}],
  "reasoning": "Brief chain: user asked X → sources A and B state Y → therefore Z"
}

Instruct: every factual claim must map to a citation; if unsupported, abstain.

Post-processing validation

• Citation checker: each chunk_id was in retrieved set
• Quote verifier: fuzzy match quote against chunk text (catch fabricated citations)
• Reject or regenerate if validation fails

UI layer

• Render citations as clickable links to source passage
• Show confidence or "partially supported" when only subset of claims cite

Explanation depth

• User-facing: short rationale (2–3 sentences) — not chain-of-thought leakage
• Internal trace: full retrieval scores, prompt version for auditors

Eval

• Citation precision/recall on labeled set
• Human review: does explanation match cited evidence?

Limits

Models can cite wrong chunk confidently — verification code is mandatory for high-stakes domains.

Interview summary: Stable chunk IDs, structured answer+citation+reasoning schema, server-side citation validation, and UI that links claims to evidence.

Answer

Conflicts are normal when docs differ in version, jurisdiction, or authority. The system must detect, rank authority, and surface uncertainty — not pick randomly.

Step 1: Detect conflict

• Retrieval returns chunks with contradictory claims (dates, prices, policies)
• NLI model or LLM pass: "Do chunk A and B contradict on entity X?"
• Flag conflict set before synthesis

Step 2: Authority hierarchy

Encode metadata at ingest:

• doc_type: policy > FAQ > forum post
• effective_date, version, region
• source_system: HRIS beats scraped wiki

Retriever or orchestrator prefers higher-authority, newer docs when scores are close.

Step 3: Prompt behavior

Instruct model to:

• State both positions when conflict unresolved
• Cite each side with source and date
• Recommend verification path ("Contact HR for your region")
• Never silently merge incompatible facts

Step 4: User context

• If user has region=EU, filter or boost EU policy chunks
• Personalize only from authorized customer-specific docs

Step 5: Operational fix

• Conflicts often mean stale docs in index — route to content owners
• Dashboard: top conflicting doc pairs for KB cleanup

Eval

• Gold cases with known outdated + current policy — expect explicit conflict handling, not wrong merge

Interview close: Metadata-driven authority, explicit conflict detection, answer presents both sides with dates, abstain or escalate when hierarchy unclear.