Interview Question
A model gives confident answers that are not supported by retrieved documents. How would you troubleshoot it?
What the interviewer is testing
- Layered diagnosis
- Evidence-based evaluation
- Fallback design
Real-World Sample Answer
I would approach “A model gives confident answers that are not supported by retrieved documents. How would you troubleshoot it?” by clarifying the requirements first, then using these considerations: in a real interview, i would not jump straight to a technology choice. for “a model gives confident answers that are not supported by retrieved documents. how would you troubleshoot it?”, i would first separate retrieval quality from prompt/context construction. then i would check whether retrieved evidence actually supports the answer and evaluate model behavior on a representative set. i would also define refusal or fallback behavior. finally, i would instrument retrieval, context, and generation separately. i would make the assumptions explicit and explain what evidence or production signals would make me revisit the decision.. I would state my assumptions and defend the trade-offs rather than presenting the choice as universally correct.
What a Strong Answer Should Cover
In a real interview, I would not jump straight to a technology choice. For “A model gives confident answers that are not supported by retrieved documents. How would you troubleshoot it?”, I would first separate retrieval quality from prompt/context construction. Then I would check whether retrieved evidence actually supports the answer and evaluate model behavior on a representative set. I would also define refusal or fallback behavior. Finally, I would instrument retrieval, context, and generation separately. I would make the assumptions explicit and explain what evidence or production signals would make me revisit the decision.
Common Mistakes
- Blaming the model immediately
- Using only anecdotal examples
- No measurable grounding target
Interviewer Follow-ups
- What assumptions would you clarify before committing to the design?
- What changes if the scale, reliability target, security requirement, or budget changes?
- What is the biggest failure mode in your proposed approach?