Every AI engineering interview I’ve seen opens with some version of the same question: explain, in plain terms, how a large language model turns your input into its output.
It sounds like a warm-up. Candidates often treat it that way — a quick “it predicts the next word” and move on to the parts they’d rather talk about, like RAG pipelines or agent frameworks. That’s a mistake, and interviewers know it’s a mistake, because this one question is a remarkably efficient filter. It tells them in ninety seconds whether a candidate has an accurate mental model of the system they’re about to be trusted to build with, or a folk-explanation borrowed from a blog post skim.
Why this question filters so well
The honest answer requires naming five distinct mechanisms in sequence, and most candidates stall after the first two. Tokens and embeddings are the easy part — they show up in every introductory explainer, so most people can recite them. Attention is where things get thinner: candidates know the word, fewer can say what it actually does. And the hallucination question — why does a model state something false with total confidence — is where the gap between “read about it” and “understands it” becomes obvious.
That gap matters more than it might seem. An engineer who thinks hallucination is a data-coverage problem will reach for the wrong fix in production — more context, bigger models, better retrieval — when the actual cause is structural: there is no truth-checking step anywhere in how the model generates text. It predicts what’s statistically plausible. Confident tone is a stylistic pattern learned from training data, not a signal that the underlying claim was verified.
The part that actually separates strong candidates
Here’s the detail that consistently distinguishes a strong answer from an average one: explaining why attention makes prompt structure matter, not just defining attention as a term.
Attention is the mechanism that scores which parts of the context are relevant to each token the model is about to generate. That’s the textbook definition. The useful version of that fact, the one that shows real hands-on experience, is this: when your actual instruction is buried under paragraphs of boilerplate, that boilerplate isn’t neutral. It competes for attention against the instruction you actually care about, and the output gets measurably worse as a result. Candidates who’ve hit this in practice — a prompt that started working better after they moved the key instruction to the top or the end — explain it from experience. Candidates who’ve only read about attention explain it as trivia.
The context window is not memory, and conflating the two costs interviews
A second common stumble: treating the context window as if it were the model’s memory. It isn’t. It’s a fixed-size buffer that resets completely on every single API call. Nothing persists between requests unless an application explicitly resends it. The reason a chatbot appears to remember earlier turns in a conversation has nothing to do with the model — it’s the application layer replaying prior messages back into the request every time. Say “the model remembers” in an interview and you’ve signaled a gap that will surface again the moment the conversation moves to production memory management, multi-turn cost, or RAG.
What a complete answer actually sounds like
A candidate who covers all five — tokens, embeddings, attention, the generation loop, and the hallucination root cause — in under two minutes, without prompting, is telling the interviewer something specific: this person has built with these systems, not just read a summary of how they work. That’s the actual signal being tested. The mechanism itself is genuinely learnable in an afternoon. What takes longer is internalizing it well enough to explain it fluently under the mild pressure of an interview, which is a different skill than understanding it in the abstract.
If tokens, attention, and hallucination are the parts of this you’d want to firm up before your next interview, session one of the AI/ML engineering course covers this exact question with a full scored model answer and the do’s and don’ts interviewers are actually listening for.