LLM Fundamentals An interviewer asks you to explain, in one breath, why a model can sound completely confident while being completely wrong.
How does an LLM actually generate its next word, and why does that explain hallucination?
Show model answer
An LLM doesn't "know" facts — at each step it computes a probability distribution over the next token given everything before it, and samples from that distribution. Confidence in the output is really just a high probability score for that token sequence, which is a property of what's statistically likely to follow, not a check against ground truth. Nothing in that process consults a database of true facts unless you've explicitly wired one in (RAG, tool calls). So a model can produce a fluent, high-probability sentence that's also factually wrong — the mechanism that makes it sound confident and the mechanism that would make it correct are unrelated. That's why "the model seemed so sure" is not evidence.
Freshers: naming next-token prediction correctly is usually enough to pass this. Experienced candidates are expected to follow it with what they'd actually do about it (grounding, verification) — stopping at the mechanism reads as bookish.
Prompt Engineering Your prompt works fine in your notebook. In production it occasionally returns a string that isn't valid JSON, and the parser downstream throws.
How do you structure a prompt so a model reliably returns valid, parseable output?
Show model answer
Three layers, not one: (1) constrain at the model level first if the API supports it — structured output, JSON mode, or function-calling schemas beat prompt instructions alone, because the sampler itself is constrained. (2) If that's unavailable, make the format contract explicit and narrow in the prompt — a schema, a short example, an instruction to return nothing else. (3) Never trust either of those alone in production: parse defensively, catch failures, and retry once with a corrective follow-up prompt that includes the parse error. The mistake is treating "good instructions" as sufficient — reliability comes from combining a constrained decode path with a defensive parse path, not from wording the prompt more forcefully.
Freshers often stop at "I'd tell it to return JSON." Experienced candidates are expected to mention structured-output/function-calling APIs and a retry path, not just prompt wording.
Prompt Engineering You tweak a prompt, it looks better on the handful of examples you tried by hand, and you ship it.
How do you know a prompt change is actually better, not just better on the examples you eyeballed?
Show model answer
Eyeballing a few hand-picked examples is confirmation bias — you'll always find cases that look better. The real answer is a small eval set (20-50 real or realistic input/expected-output pairs) scored the same automated way every time: exact match for structured output, similarity or grounding score for open text. Run both the old and new prompt against it and compare numbers, not impressions. The set isn't a one-time gate — re-run it on every future change so a "small tweak" three weeks from now doesn't silently regress something that used to work.
Freshers: naming the eval-set idea at all is a strong signal. Experienced candidates should also name the regression-testing angle — re-running it on future changes, not just this one ship.
RAG & Embeddings Your retrieval logs show the correct chunk was pulled back. The chatbot still answered wrong.
Your RAG system retrieved the right context, but the answer's still wrong — walk me through why.
Show model answer
Split it into two separate failure classes before debugging further: retrieval failure (the right chunk never made it into context — check top-k, chunking, embedding similarity scores) versus generation failure (the right chunk was there, and the model ignored or misread it anyway — check prompt structure, whether context is buried below other noise, or whether the model is leaning on its own parametric knowledge instead of the provided context). Since retrieval already worked here, this is a generation failure: look at how the context is presented in the prompt, and consider explicitly instructing the model to answer only from the provided context and say so when it can't.
Experienced candidates are expected to separate these two failure classes cleanly and reason from the evidence given (retrieval already confirmed working), not restate a generic RAG explainer.
RAG & Embeddings You can't have a human re-read every answer your RAG system produces before it ships to a user.
How would you automatically catch a RAG system giving a confident, ungrounded answer?
Show model answer
Compute a grounding score between the generated answer and the retrieved context — cosine similarity between their embeddings is a cheap, no-API-key way to start (a small local embedding model, no external calls). Log every response's score, flag anything under a threshold for review, and treat that threshold like any other production metric — tune it against real examples, not a guess. It doesn't tell you which failure class caused a low score, but it tells you something needs a human look, automatically, at production volume.
Freshers: describing the idea of a similarity score is enough. Experienced candidates are expected to mention logging it at production scale and treating the threshold as tunable, not fixed.
Context & Memory A user tells your agent their account number in turn 2. By turn 12, it's asking for it again.
Your agent loses track of something the user said several turns ago — how do you fix that without just cramming more tokens into the prompt?
Show model answer
Growing the context window isn't a fix, it's a tax — cost, latency, and the model's own tendency to lose things in the middle of a long context all get worse. The real fix is separating what needs to persist from what's disposable: extract durable facts (account number, stated preference) into a small structured memory object updated each turn, and only inject that summary plus the last few turns into the prompt — not the full transcript. That keeps the prompt small and puts the facts that matter somewhere the model reliably sees them, rather than buried in turn 4 of 20.
Freshers commonly answer "increase the context window" — naming why that's insufficient (lost-in-the-middle, cost) is the actual differentiator interviewers listen for.
Agents & Tool Use Someone on your team wants to split a single well-working prompt into three "specialist" agents that talk to each other.
When does a multi-agent setup actually earn its complexity over a single well-designed prompt?
Show model answer
Multi-agent adds real cost — more latency (multiple round trips), more failure surface (agents can miscommunicate or loop), more to debug. It earns that cost when a task genuinely needs different tools, different context windows, or different degrees of autonomy per sub-task — e.g., a researcher agent that browses versus a writer agent that shouldn't have browsing access at all. It doesn't earn the cost when the real problem is a single prompt trying to do too much in one pass; that's usually better solved by breaking the prompt into sequential steps inside one agent, not spinning up separate agents. The test: could this be a function call inside one loop instead of a message between two agents? If yes, it should be.
This question rewards candidates who push back on unnecessary complexity — naming when multi-agent is overkill is as important as knowing when to use it.
Evals & Guardrails A dependency update or a "small" prompt tweak quietly makes your AI feature worse. Nobody notices until a user complains.
How do you catch a regression in an AI feature before your users do?
Show model answer
The same discipline as any other production system: a regression suite that runs automatically, not a human spot-check. Maintain a fixed eval set of real or representative cases with expected outcomes, score every relevant change against it (prompt edits, model version bumps, retrieval config changes), and gate deploys on the score not dropping below a baseline. Pair that with production monitoring — sampling a slice of real traffic and scoring it the same way — since an eval set can't cover everything real users will do. The goal is the same feedback loop CI gives you for code: catch it in minutes, not from a support ticket.
Freshers: describing the eval-set idea covers the basics. Experienced candidates should connect it to a deploy gate and production sampling, not just a one-time test.
Evals & Guardrails A system flags something as high-risk. The person who has to act on that flag asks "why" — and "the model said so" isn't an answer they can use.
How would you make an AI system's flagged decisions explainable to the person who has to act on them?
Show model answer
Explainability has to match the audience, not just the model — a feature-importance plot means nothing to someone who needs to decide in 30 seconds. Practically: surface the specific input signals or retrieved evidence that drove the score in plain language, not the model's internals; keep a threshold-and-reason-code system so a flag always comes with an actionable reason attached; and log both the flag and the eventual human decision so you can measure whether the flagged reasons actually correlate with real outcomes over time. Explainability that isn't actionable by the reviewer is decoration, not a guardrail.
Experienced candidates are expected to talk about actionability for the specific reviewer, not generic interpretability techniques.