Hiring for LLM Engineering: What the Interview Actually Needs to Test

Most engineering teams that hire for LLM roles run roughly the same interview: two rounds of LeetCode, a system design question, maybe a quiz on transformer internals. They're assessing for the wrong things — and they know it. The candidates who ace those screens often struggle to ship working AI features, while the ones who stumble on binary search can build an eval suite from scratch and debug a hallucinating pipeline in an afternoon.

The skills that predict success in LLM engineering have almost no overlap with what traditional ML or software interviews test. Hiring managers who haven't updated their process are generating false negatives at a high rate — rejecting engineers who would succeed — while false positives walk in with solid LeetCode scores and no intuition for when a model is confidently wrong.

Why the Standard Screen Fails Here

Traditional ML interviews optimize for math recall and algorithmic coding. Can you derive the attention mechanism from scratch? Do you know when to use precision vs. recall? Can you implement a binary tree traversal in fifteen minutes?

Those skills matter for researchers building models. They are nearly irrelevant for engineers deploying them.

LLM engineering is primarily a discipline of selection, evaluation, and integration. You are not training a model; you are configuring one, measuring its failure modes, and deciding whether those failures are acceptable in production at a given cost. The core competency is judgment under uncertainty — specifically, the ability to reason about probabilistic outputs, design tests for vague specifications, and develop an instinct for where models break that doesn't require exhaustive enumeration.

A candidate who can explain multi-head attention in detail but has never written an eval harness, never debugged a prompt that degrades on edge cases, and has no instinct for cost estimation is not ready for this work. Yet that candidate clears most hiring bars with ease.

The Three Skills That Actually Predict Success

After stripping away the credential noise, three capabilities distinguish engineers who ship working AI products from those who prototype endlessly and never reach production confidence.

Eval design instinct. Can the candidate take a vague, business-language specification — "the model should summarize support tickets accurately" — and translate it into a concrete, measurable eval suite? This requires understanding what "accurate" means operationally, knowing which failure modes to cover (hallucinated resolutions, wrong ticket category, incomplete extraction), and deciding how many examples are sufficient to get signal before you have thousands of labeled pairs. Engineers without this instinct ship AI features that have never been tested against any systematic criteria. They rely on vibes.

Failure mode intuition. LLMs fail in ways that are structurally different from deterministic software. They are confidently wrong. They perform well on the cases you tested and poorly on the cases adjacent to them. They degrade on input distribution shifts in non-obvious directions. A strong LLM engineer has developed a mental catalog of these failure patterns — prompt injection, context window boundary effects, instruction-following collapse on multi-step tasks, overconfidence on knowledge-cutoff-adjacent queries — and will probe for them without being told to. This intuition comes from having built systems that failed in production, not from having read about failure modes in a paper.

Prompt debugging skill. Given a prompt and a set of outputs that don't meet the quality bar, can the candidate diagnose the cause and produce a targeted fix? This is harder than it sounds. Bad debugging looks like random perturbation — changing words, adding instructions, hoping something improves. Good debugging looks like hypothesizing a root cause, designing a minimal test to confirm it, and making a surgical change. The difference is immediately visible when you watch someone work.

What the Interview Should Actually Test

The most revealing interview format is a ninety-minute practical session built around three exercises. No quiz questions. No theory recall. Just tasks that proxy the actual job.

Exercise 1: Build a small eval from a vague spec. Give the candidate a two-sentence description of a feature — for example, "an LLM that classifies customer feedback as a bug report, feature request, or general question" — and a dozen unlabeled examples. Ask them to design an eval: what cases would they create, how would they measure success, how many examples would they need before trusting the metric, and how would they handle edge cases that don't fit neatly into the three categories. There is no correct answer. What you're looking for is whether they ask clarifying questions, whether they think about the false positive and false negative costs asymmetrically (misclassifying a bug report as a feature request might be very different from the reverse), and whether they understand the difference between coverage and precision in their test set.

Exercise 2: Debug a hallucinating prompt. Provide a prompt, a context document, and five sample outputs — some correct, some hallucinated. Ask them to identify which outputs are problematic, hypothesize why, and propose a prompt modification. Watch how they reason. Do they read the source document carefully before forming an opinion? Do they notice when the model cites information that isn't in the context? Do they propose specific, testable changes, or do they rewrite the entire prompt because they're uncomfortable with targeted diagnosis? The debugging process is more informative than the final answer.

Exercise 3: Estimate the cost of a described pipeline. Sketch a feature: "Given 50,000 customer support emails per day, we want to extract the root cause, assign a priority level, and suggest a response template using an LLM." Ask them to estimate the daily token cost, identify the biggest cost drivers, and propose two optimizations. This tests whether they have internalized that production AI is an economics problem, not just an accuracy problem. The best candidates immediately start reasoning about input length, caching opportunities, model tier selection (do all 50,000 emails need the most capable model?), and the option to run cheaper classification first and escalate only ambiguous cases. Candidates without this instinct produce architectures that are technically correct and economically unshippable.

The Red Flags That Standard Interviews Don't Surface

Several failure modes are invisible until someone is actually on the team — unless you design the interview to surface them.

Treating models as magic. Engineers who have only used LLMs through convenience wrappers often have no mental model of what's happening when a prompt fails. They expect the model to "just work" on new inputs. When it doesn't, their debugging strategy is to try different phrasing until something sticks. This produces prompts that are brittle in ways nobody understands.

No evaluation infrastructure experience. Ask directly: describe the last eval harness you built. What did you measure? How did you handle subjectivity? What surprised you? Engineers who have shipped AI features in production always have stories here. The stories are usually about being wrong — about discovering that a metric they trusted didn't correlate with what users actually cared about, or that aggregate accuracy hid a catastrophic failure mode on a specific input pattern. Candidates who can't tell these stories haven't shipped.

Confusing ML math proficiency with LLM engineering proficiency. A candidate who spent three years fine-tuning BERT classifiers has transferable skills, but they are not the same skills. The unit of work in LLM engineering is the prompt, the eval, and the deployment configuration — not the training run. Some strong traditional ML engineers make this transition quickly; others find the probabilistic, specification-driven nature of the work disorienting. The interview should probe whether the candidate has actually operated in this mode, not just studied it.

Overconfidence in benchmarks. Candidates who cite benchmark scores as primary evidence for model selection decisions are signaling that they haven't encountered production distribution shift. Benchmarks measure performance on curated datasets that may not represent your inputs. Strong candidates treat benchmark results as a prior and then describe how they'd validate on their own data before committing.

What You Should Actually Be Hiring For

The useful reframe is this: you are not hiring someone to understand LLMs; you are hiring someone to build reliable products that use LLMs despite their unreliability. That framing changes what signals matter.

Portfolio evidence beats credentials. There are no meaningful certifications in this space. Look at what candidates actually built. Ask them to walk you through a project where the first approach failed. The quality of the failure analysis tells you more than the description of the successful outcome.

Opinionated model selection signals real experience. Strong LLM engineers have views on when a smaller, cheaper model is sufficient and when it isn't. They've developed heuristics for which model families fail in which ways. They have opinions on the trade-offs between hosted APIs and self-hosted models that aren't lifted from blog posts. Candidates who answer "it depends on the use case" without substantiating what it depends on and why are revealing that they haven't made enough of these decisions under real constraints.

Data literacy is underweighted. Much of senior LLM engineering work is examining examples, spotting distributional patterns, and diagnosing why the model behaves differently on a class of inputs. This requires careful attention to data — the ability to spot inconsistencies in labeled examples, to notice when your eval set has structural biases, to be appropriately skeptical when metrics look too good. Engineers who skip this and jump to model changes first will repeatedly fix the wrong thing.

Adjusting the Bar for What's Genuinely Scarce

A practical note on calibration: the candidate pool for LLM engineering has inflated with people who have watched tutorials and run notebooks but have never shipped a system that failed in production and had to be diagnosed and fixed. The bar for "I have LLM experience" is too low to be useful.

The practical exercises above will empty a pipeline that fills with tutorial-completers. That's the point. You want the interview to be hard in exactly the ways that matter — not because of arbitrary difficulty, but because the exercises proxy the actual work. An engineer who can design a meaningful eval from a vague spec, debug a hallucinating prompt systematically, and think in terms of pipeline economics is prepared for this job. An engineer who can explain backpropagation but has never written an eval harness is not.

Running this screen will probably mean interviewing more candidates for fewer offers. That's fine. LLM engineering is a role where one engineer who ships working systems is worth several who prototype fluently. Optimize the hiring bar for the thing that's actually hard.

The three exercises described here work for both mid-level and senior hiring. For senior roles, add a fourth: give them a failing system — real or constructed — and ask them to propose and prioritize a debugging plan. Seniority shows in how they triage, not in how much they know.