KnowML
Interview Mastery
Every interview question is testing the same underlying skill, wearing a different costume. This isn't a 28th topic to memorize. It's the framework, the drills, and the question bank you run on top of everything else on this site.
"Why this model," "design a RAG system," "why did your loss go to NaN." Different words, same test. The interviewer usually isn't checking whether you remember a fact. They're checking whether you can reconstruct a design decision from first principles: what problem it solves, what the alternatives were, and what you gave up to get here.
Nobody cares that you memorized a layer count. Everybody cares whether you can explain why anyone would pick that number over another, and what breaks if you don't. Learn the shape of the answer and it transfers to a question you have never seen before, which is the situation you will be in for at least half of a real interview.
01The six-question framework
Representation → Objective → Architecture → Training → Inference → Deployment. Same six questions, every model, every time.
This is the order every page on this site is written in (see the home page's mental model), and it is also the order a well-run technical interview typically follows, whether or not the interviewer says so out loud.
Treat it as a silent checklist, one you run the moment you hear "why did you use X" or "why not Y instead." Fill in an honest sentence for all six and you have a real answer. Get stuck on one? That box is precisely the gap you need to go revise. It is not a cue to change the subject.
Applying it live
Say the question is: "Why would you use a Transformer decoder instead of an LSTM for a customer-support chatbot?" Here's what running the checklist out loud sounds like:
- Representation. Both consume a sequence of tokens, but they hold history differently. An LSTM compresses everything before the current step into one fixed-size hidden state. A Transformer keeps every past token's key/value pair explicitly, and lets the current step attend back to any of them directly. State that difference first. It is the root cause of everything that follows.
- Objective. Both are typically trained with the same next-token cross-entropy loss, so this box is a wash here. Say so out loud: knowing when a box does not differentiate two options signals understanding just as much as knowing when it does.
- Architecture. The real answer lives here. An LSTM's hidden state has to carry turn 1 through every step up to turn 40 sequentially, degrading with distance. A Transformer gives turn 1 and turn 40 a direct, one-hop path via self-attention (see 08). This direct path matters when a user references something they said much earlier in a long conversation.
- Training. LSTM training is inherently sequential, because step 40 needs step 39. It cannot parallelize across the time axis on a GPU. A Transformer processes the whole sequence in one parallel matmul. This is why Transformer-based chatbots could be trained on the data volumes they were: an LSTM at that scale would take too long to be practical.
- Inference. The tradeoff flips here, and saying so is what separates a memorized answer from an understood one. LSTM inference is O(1) memory and compute per generated token. A Transformer needs a growing KV cache, and attention cost that scales with context length. That is why serving tricks like grouped-query attention and paged KV caches exist (see 08).
- Deployment. Given that tradeoff, the decision depends on constraints the question has not specified: expected conversation length, latency SLA, GPU budget. That is your cue to ask a clarifying question instead of guessing, and asking it is part of the answer.
Notice the answer never says "Transformers are just better." It says what each architecture does differently, why that difference matters for this specific task, and what it costs. That is mechanism plus cost with no verdict, and it is what makes an answer sound understood instead of memorized.
02System-design answers have a shape
"Design a system that does X" is not a request to reproduce an architecture diagram from memory. It is a request to watch how you reason under uncertainty. Every strong answer follows the same shape, whether X is a recommender, a RAG pipeline, or a fraud model.
- Clarify. Requirements, constraints and scale, before proposing anything. Volume, latency budget, freshness needs, who the users are.
- Baseline. The simplest thing that could plausibly work. This proves you know how to ship something before you know how to gold-plate it.
- Bottleneck. Name the failure the baseline hits. Be specific instead of vague about "scaling issues."
- Iterate. Upgrades in order, each one justified by the exact bottleneck it fixes.
- Evaluate. State the metric you would track and why it fits this problem. Don't default to accuracy.
- Monitor. Name the failure modes that only show up in production, and what you would alert on.
Skipping Clarify is the most common way strong candidates give weak system-design answers. They solve a problem the interviewer didn't ask about.
Two minutes spent pinning down volume, latency and freshness changes which baseline is even reasonable. A batch job that reruns overnight and a service answering interactively are not the same problem, and the question rarely says which one you are in.
Model answer: "Design a RAG system for internal company docs"
- Clarify. How many documents, and how often do they change? Does the answer need a citation back to the source doc? Single-turn Q&A or multi-turn conversation? Do different users have different document access? These answers change the entire design, so ask before proposing anything.
- Baseline. Chunk documents into roughly paragraph-sized pieces. Embed each chunk with an off-the-shelf model and store the vectors in an HNSW-style approximate-nearest-neighbor index. Embed the incoming query the same way, retrieve the top-k most similar chunks, and put them into the LLM's prompt as context. See 11 for how this pipeline composes with the model itself.
- Bottleneck. This baseline fails in three predictable ways. It misses exact matches like a policy number or a table value, because semantic embeddings blur precise tokens. It misses multi-hop questions needing facts from several chunks, because top-k retrieval treats each chunk independently. And it ignores per-user permissions entirely, since the index has no notion of who may see what.
- Iterate. Add hybrid retrieval: combine BM25 keyword search with dense embedding search and merge the ranked lists, catching exact matches dense retrieval misses. Add a cross-encoder reranker over the top candidates, since jointly scoring query and chunk is far more accurate than embedding similarity alone, just too slow to run over the whole corpus. Chunk more carefully, keeping tables and code blocks intact rather than splitting mid-structure. Attach access-control metadata and filter at query time. Each fix answers a specific bottleneck named above, and is not a feature dumped in because it sounds sophisticated.
- Evaluate. Two different things need two different metrics. Retrieval quality: recall@k and MRR against a labeled set of question-to-correct-chunk pairs. End-to-end answer quality: whether the final answer is grounded in what was retrieved, checked with a human rubric or an LLM-as-judge, plus plain latency P50/P95.
- Monitor. Two failure modes matter most in production. Hallucination when retrieval returns nothing relevant: the system needs an explicit "I couldn't find this in the docs" path so the LLM doesn't guess. Staleness when documents change but the index is not re-embedded. Track re-indexing lag as a first-class metric.
03Debugging questions have a shape too
"Your loss went to NaN, what do you check?" is not a trivia question. It checks whether you have a mental fault tree instead of a shrug. The shape is consistent:
- Name the symptom precisely.
- List the causes that produce that exact symptom, and skip every other possible ML bug.
- Say what you would inspect first to distinguish between them.
Here is that fault tree for five failure modes that come up constantly.
| Symptom | Likely causes | What you'd check first |
|---|---|---|
| Loss plateaus early, stays flat | LR too low (barely moving) or too high (bouncing around a floor); dead ReLUs / vanishing gradients; bad initialization; a layer accidentally frozen; gradient clipping set too aggressively | Plot loss against a short LR range test; log per-layer gradient norms — a norm near zero points at dead units or a frozen layer, not the data |
| Loss goes to NaN | LR spike causing an exploding update; fp16 mixed precision without loss scaling; a log(0) or divide-by-zero in the loss (e.g. a class with zero predicted probability); a corrupted batch (inf/NaN pixel or feature value, an out-of-range label index) | Check the last good batch for inf/NaN before it hits the model; check gradient norm right before the spike; if on fp16, switch to bf16 or add loss scaling; add gradient clipping |
| Train loss keeps falling, val loss rises (overfitting) | Model has the capacity to memorize a training set that's too small or too repetitive relative to that capacity; no or weak regularization; no augmentation; too many epochs with no early stopping | Plot train vs. val curves side by side, not just final numbers; check dataset size and duplication rate; check whether dropout/weight decay/augmentation are enabled and not silently no-ops |
| Train loss won't go down at all | Gradients aren't flowing (a detached tensor, wrong parameters passed to the optimizer, a frozen backbone by accident); loss function mismatched to label format (e.g. expects class indices, got one-hot); LR far too low; a data/label pipeline bug (inputs and labels shuffled independently) | The standard sanity check: try to overfit a single batch of 8–16 examples. If loss won't go near zero on that, the bug is in the pipeline or the graph, not the dataset scale — go check gradients are nonzero and a few (input, label) pairs by hand before touching hyperparameters |
| Model collapses to one output | Severe class imbalance and "always predict the majority class" is a real local optimum for the loss; for GANs, discriminator overpowering the generator (mode collapse); LR spike early in training wiping out useful weights; a loss the model can satisfy with a trivial constant answer (e.g. predicting the dataset mean under MSE with weak signal in the features) | Check the distribution of predictions, not just the loss number; compute the "always predict majority" baseline as a sanity floor — if your model matches it, that's your answer; for GANs, plot generator vs. discriminator loss over time to see who's winning |
Before debugging anything else, try to overfit a tiny slice of the data (a handful of examples) to near-zero loss. If the model cannot do that, you don't have an optimization or generalization problem yet. You have a pipeline bug, and no amount of hyperparameter tuning will fix it.
This single move is the fastest way to split "something is broken" into "the code is broken" versus "the learning problem is hard."
04The 60-second explanation drill
"Explain X in 60 seconds" tests compression. Can you find the single sentence that matters, and then stop? The shape is four beats, out loud, and no more: one-line definition → the problem it solved → the one key mechanism → one tradeoff. Here it is applied to the Transformer. The full page is 08; this is the compressed version you would say.
- One-line definition. The Transformer is a neural network architecture that processes an entire sequence in parallel using self-attention instead of recurrence.
- The problem it solved. RNNs process tokens one at a time, so training cannot parallelize across the sequence. Information from early tokens has to survive a long sequential chain of updates to reach later ones, which made long-range dependencies hard to learn.
- The one key mechanism. Self-attention. Every token computes a weighted average of every other token's representation, with weights learned by comparing queries against keys. Any two tokens, however far apart, get a direct one-hop path to influence each other.
- One tradeoff. That direct path costs $O(n^2)$ compute and memory in sequence length. Which is exactly why long-context serving needed a second wave of engineering (FlashAttention, grouped-query attention, paged KV caches) to become practical.
That's the whole drill. Notice it never drifts into multi-head attention, positional encodings, or layer-norm placement, which are 08's job and not this 60 seconds'. Saying less, correctly and in the right order, usually reads as more senior than saying everything you know.
05Compare-two-architectures drill
The shape: what's shared → what's different → why the difference exists → when each wins. The trap is jumping straight to "different" without stating "shared" first. Most of the value in a comparison answer is showing you know which parts are the same idea in different clothes. Model answer: GANs versus diffusion models, both mainstream approaches to image generation.
- What's shared. Both are deep generative models that turn some form of noise into a realistic sample. Both train end-to-end with gradient descent. Both can be conditioned on extra input, like a class label or a text prompt, to control what gets generated.
- What's different. A GAN generates a full sample in a single forward pass through a generator, trained by pitting it against a discriminator in an adversarial minimax game. A diffusion model starts from pure noise and runs many small denoising steps. Each step trains with a comparatively simple regression-style objective: predict the noise that was added.
- Why the difference exists. The adversarial game is what makes GANs capable of one-shot generation. That same minimax setup is notoriously unstable to train and prone to mode collapse, where the generator finds a handful of outputs that reliably fool the discriminator and stops exploring the rest of the distribution.
- Diffusion sidesteps the adversarial game entirely. It reframes generation as reversing a fixed, known noising process, which turns training into a well-behaved regression problem. It is far more stable, and empirically better at covering the full diversity of the data. The direct cost is needing many sequential denoising steps to produce one sample instead of one pass.
- When each wins. GANs win when single-pass, low-latency generation matters more than training stability or sample diversity, and you are willing to invest in the tuning tricks adversarial training needs. Diffusion wins when stability and output diversity matter more than raw speed. That is why most state-of-the-art text-to-image systems moved to diffusion, or a distilled few-step variant, and away from pure GANs. They trade some inference latency for consistently higher-quality, more diverse output, and a training process that does not require balancing two competing networks.
06Paper-reading drill
"Walk me through a paper you've read recently" is one of the highest-signal questions in an ML interview. Summarizing an abstract is easy, and most candidates stop there. Four things demonstrate you understood the paper, in order:
- The core contribution.
- The baseline it is beating.
- The key ablation proving the contribution itself matters, beyond the whole system.
- The stated limitation.
Applied to Attention Is All You Need (already the subject of 08, so you can check this against a page you know is accurate):
- Core contribution. The claim isn't "they built a good translation model", since plenty of systems already did. It is narrower and more interesting: recurrence can be removed entirely and replaced with attention alone, while matching or beating those systems. Nobody had shown that before.
- Baseline. The best contemporary recurrent and convolutional sequence-to-sequence systems for machine translation: RNN-with-attention encoder-decoders, and convolutional seq2seq approaches. The paper has to beat those on the same benchmark to justify the architectural claim, and showing that attention "works" in isolation would not be enough.
- Key ablation. The paper doesn't just report one final BLEU score. It varies the number of attention heads and the per-head dimension while holding total model size roughly fixed, and shows quality degrades if you collapse to too few heads or shrink each head too far. That ablation proves the specific design choice, splitting attention into multiple smaller heads, is doing real work. Not just "attention in general" being good. That distinction is what an interviewer is listening for.
- Stated limitation. The paper is explicit that full self-attention is quadratic in sequence length, and flags restricting attention to a local neighborhood as future work for handling very long sequences. This is the constraint that later drove FlashAttention, sparse and windowed attention, and the KV-cache engineering covered in 08 and 23.
Reciting the abstract and the headline result isn't the same as reading the paper. If you cannot name one ablation from the results section, you have not shown that you read past the introduction, and that is exactly what this question is designed to catch.
07Interview questions
Eight real prompts spanning the categories interviewers draw from. Each model answer applies one of the frameworks above out loud, on purpose. Read these as demonstrations of the shape and don't memorize them as scripts.
Model selectionGradient-boosted trees or a deep neural net for tabular customer-churn prediction — which do you pick, and why?
Run representation → objective → training → deployment. Representation: tabular data with a mix of categorical and numeric features, moderate row count — no inherent spatial or sequential structure for a neural net to exploit. Objective: both can optimize the same log-loss / AUC target, so that box is a wash. Training: gradient-boosted trees are dramatically more sample-efficient and robust on tabular data out of the box — they handle missing values and mixed feature types natively, need far less tuning to hit a strong baseline, and give you feature importances for free, which matters if the business needs an explanation for a churn score. A neural net can match or beat trees here, but usually only with more data, embedding layers for high-cardinality categoricals, and meaningfully more tuning effort. Deployment: trees are cheaper to serve and easier to audit. Verdict: trees, unless there's a specific reason to want representation learning — e.g. fusing tabular features with text or sequence data in one model, which is where neural approaches start to win on tabular problems.
Architecture derivationWhy does a ResNet need a residual/skip connection — what actually breaks without it?
Very deep plain (non-residual) networks were observed to get worse training error, not just worse test error, as depth increased past a point — that rules out overfitting as the explanation and points at optimization itself: the extra layers were making the function harder to fit, not easier. A residual connection reframes each block as learning a delta from the identity function rather than the full transformation from scratch — in the worst case, if the extra layers aren't helpful, the block can drive its own contribution toward zero and default to passing the input through unchanged. The practical effect is a direct additive path for gradients back toward the input, so depth stops being an optimization liability — the same reasoning that makes the residual stream inside a Transformer block trainable at 96+ layers (see 08).
DataYour fraud-detection model is getting 99% validation accuracy (illustrative number) — suspiciously good. Walk through how you'd check whether something's wrong.
First, check whether 99% is even meaningful: fraud datasets are usually severely imbalanced, so compute the "always predict not-fraud" baseline accuracy — if it's also near 99%, the metric itself is the problem, and you should be looking at precision/recall or PR-AUC instead. Second, check for leakage: is any feature computed using information that wouldn't exist yet at prediction time — e.g. a field only populated after a transaction was already confirmed fraudulent? Third, check the split: fraud data is inherently time-ordered, so a random train/val split lets future information leak backward into training; it needs a temporal split instead. Fourth, check for near-duplicate rows straddling the train/val boundary. Only after ruling all of that out would "the model is genuinely this good" become a plausible conclusion.
System designDesign a real-time voice agent — what's the first thing you clarify, and what's your baseline?
Clarify first: what's the end-to-end latency budget (time to first audio response), does the system need to handle interruptions/barge-in mid-response, and what language(s) and acoustic conditions (noisy environments, accents) need coverage. Baseline: a cascaded pipeline — streaming ASR converts speech to text incrementally, a streaming LLM starts generating a response as soon as it has enough transcript, and streaming TTS starts speaking the first sentence before the LLM has finished generating the rest, so latency stages overlap instead of stacking fully in sequence. The bottleneck this baseline hits: cascaded latency still stacks (ASR finalization + LLM first-token + TTS first-audio) and errors compound — a mistranscription poisons everything downstream. Iterate by feeding partial ASR hypotheses to the LLM earlier, running voice-activity detection concurrently for barge-in, and considering a smaller/faster model specifically for latency-critical turns. Evaluate on end-to-end latency percentiles, word error rate, and task success rate, not just WER in isolation. Monitor for false-interruption triggers from background noise and WER degradation drift in noisy real-world audio versus the clean audio it was likely evaluated on.
DebuggingTraining loss was decreasing normally for 2,000 steps, then spiked to NaN and stayed there. What do you check, in order?
This is the NaN row of the debugging table, applied: first check the last good batch for inf/NaN values in inputs or labels before they even hit the model — a single corrupted example is a common and easy-to-miss cause. Next, check the gradient norm in the steps right before the spike; a sudden explosion there points at an optimization instability rather than a data issue. Check precision — fp16 without loss scaling overflows far more easily than bf16, and this exact symptom (fine for a while, then NaN) is its classic signature. Check whether the LR schedule did something unusual right at that step, like a warmup restart. Fix candidates, cheapest first: add gradient clipping and switch to bf16 if on fp16, then resume from the last good checkpoint rather than restarting from scratch, and watch whether the same step number reproduces the failure — if it does, it's very likely a specific bad example, not randomness.
Latency & costYour RAG chatbot has 4-second P95 latency (illustrative) and the product needs it under 1.5 seconds. Where do you look, and what do you trade off?
Break the budget down by pipeline stage before touching anything — query embedding, vector search, reranking, LLM generation, and network hops each contribute, and generation almost always dominates. Levers, each an explicit quality-for-latency trade you'd justify with measured numbers, not guesses: a smaller or distilled model for the generation step; streaming tokens to the client so perceived latency drops even if total generation time doesn't change much; caching embeddings and retrieval results for repeated or similar queries; reducing top-k or skipping the reranker on latency-sensitive paths at some cost to retrieval recall; quantizing the serving model; and speculative decoding to cut generation latency without changing output quality. The interview signal here isn't naming all of these — it's stating that every one of them is a tradeoff you'd measure, not a free win.
Distributed trainingYou're fine-tuning a 70B-parameter model and it doesn't fit on one GPU. What are your options, and how do you choose?
Data parallelism alone doesn't help — a single replica already can't fit, so replicating it across GPUs just fails on every replica. The real constraint is memory: parameters, gradients, optimizer states, and activations all have to live somewhere. Options: tensor parallelism splits individual matrix multiplies across GPUs, which needs fast interconnect since every layer communicates; pipeline parallelism splits layers across GPUs and streams micro-batches through them like a pipeline, communicating less often but introducing idle "bubble" time between stages; and sharded data parallelism (ZeRO/FSDP-style) shards the optimizer states, gradients, and parameters themselves across GPUs while every GPU still processes different data, which is the least invasive to adopt. In practice you'd start with sharded data parallelism, add tensor parallelism within a node where the interconnect is fast (e.g. NVLink) if a single shard still doesn't fit, and add pipeline parallelism across nodes if you need to scale further. The interviewer is checking whether you understand that memory — not raw GPU count — is the actual constraint.
Production incidentA recommendation model's click-through rate silently dropped 15% (illustrative) over two weeks, with no errors logged. How do you investigate?
No errors logged means this is a drift or pipeline problem, not a crash — treat it like the debugging shape, but for infrastructure instead of a training run. Check for upstream data changes first: did a feature start silently returning nulls or defaults after a schema change, did a join break quietly. Check for training/serving skew — is the online feature computation actually identical to what the model was trained on, or did one side get updated and not the other. Check for real distribution shift in the input population — a new user segment, a seasonal effect, a competitor change altering behavior. Check the serving infrastructure itself — a stale model version redeployed, a caching layer serving old results, a misconfigured A/B traffic split. Then correlate the exact start of the drop against every deploy and config change in that window before concluding the model itself degraded — the large majority of silent production regressions turn out to be pipeline or infrastructure issues, not the model quietly getting worse on its own.
08Where to practice
This page is the framework. The depth to apply it to lives on the specific topic pages. Go to whichever row below matches where you're weakest.
| If you're weak on… | Go to |
|---|---|
| LLM system design — RAG, fine-tuning, serving, evaluation | 10 · LLM Architecture & Training, 11 · RAG, Agents & Reasoning |
| Vision system design — detection, segmentation, retrieval | 05 · CNNs & Vision, 06 · Vision Foundation Models |
| Latency / cost / memory tradeoffs, distributed training | 23 · Efficient AI & Systems |
| Evaluation, reliability, safety | 25 · Evaluation & Reliability & Safety |
| Speech / voice system design | 13 · Speech & Audio |
| Multimodal system design | 14 · Multimodal AI |
| Recommendation / search design | 16 · Recommenders & Ranking & Search |
| RL / robotics design | 15 · Reinforcement Learning, 21 · Robotics & Embodied AI |
●Now write it yourself
Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.
Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.