The ML Engineer You Hired Isn't the AI Engineer You Need

A VP of Engineering decides the company needs to "do AI." There is already a machine learning team — three people who built the recommendation model, tuned the fraud classifier, and maintain a feature store. The obvious move is to point them at the new LLM initiative. They know the math. They've shipped models. How different could it be?

Six months later the prototype demos beautifully and dies in production. Nobody can say why the agent occasionally books the wrong meeting, the cost per request is four times the estimate, and there is no way to tell whether last week's prompt change made things better or worse. The ML team is frustrated because none of the tools they're good at — gradient descent, data pipelines, hyperparameter sweeps — apply to a model they can't retrain and can't see inside.

This is the most common org mistake in AI right now, and it comes from a reasonable-sounding assumption: that the craft of building models and the craft of building on models are the same job with a different label. They are not. They overlap less than "frontend engineer" and "backend engineer" do.

Two Crafts That Share a Vocabulary and Almost Nothing Else

The cleanest way to see the split is by what each person actually touches all day.

A machine learning engineer works on a model. The dataset fits in a table — maybe millions of rows, but bounded and yours. Training takes minutes to hours. The levers are features, architecture, loss functions, and weights. The hard skills are statistics, linear algebra, and the discipline of not fooling yourself with a leaky validation set. When something is wrong, the fix lives inside the model: more data, a better feature, a different objective. The model is the deliverable.

An AI engineer works on a system wrapped around a model they did not train and cannot retrain. The "model" arrived pre-trained on trillions of tokens and shows up as an API endpoint that returns different output for identical input. There are no weights to tune. The levers are context, tools, retrieval, orchestration, and guardrails. When something is wrong, the fix almost never lives inside the model — it lives in what you fed it, how you framed the task, which tools you exposed, and what you did with the output. The system is the deliverable; the model is a vendor dependency.

Put those side by side and the divergence is obvious. One job is about producing a function from data. The other is about engineering reliable behavior out of a black box you rent. Statistics is load-bearing for the first and merely useful for the second. Distributed-systems instinct, latency budgeting, and failure-mode paranoia are load-bearing for the second and nearly irrelevant to the first.

The market has already noticed, even if your job ladder hasn't. AI engineer postings have been growing far faster than ML engineer postings — the demand curve is bending toward people who integrate foundation models, not people who train new ones from scratch. That doesn't make ML engineering obsolete; it means the two roles are separating into distinct hiring funnels, and treating them as interchangeable leaves you short on whichever one you actually needed.

The Role Nobody Put in the Ladder

Here's the structural problem underneath the staffing mistake: most engineering ladders still don't have a rung for this. There's a Software Engineer track and maybe an ML Engineer / Data Scientist track. "AI Engineer" gets jammed into one of those two, and the jamming causes damage in both directions.

Slot it under ML, and you screen for the wrong things. The interview asks the candidate to derive backpropagation or explain bias-variance tradeoff — questions that say nothing about whether they can keep an agent from looping forever or design an eval that catches a regression before a customer does. You filter for model-training depth and end up with someone strong at a craft your product barely uses.

Slot it under generic Software Engineering, and you under-level the work. The hiring manager treats "calls an API" as junior glue code, misses that production LLM systems are a genuine specialty, and staffs the role with someone who has never had to reason about nondeterminism, token economics, or what happens when the model confidently returns garbage. The system ships, then quietly degrades, because nobody owns the part that makes it reliable.

The reason the rung is missing is that the role is new and its core competency — making probabilistic systems behave well enough for production — doesn't map onto either existing track's promotion criteria. Until a company writes that competency down, it can't level it, can't interview for it, and can't tell a strong AI engineer from an enthusiastic one. The missing ladder rung is not an HR formality. It's why the mis-hire keeps happening.

Eval Design Is the Tell

If you can screen for only one thing, screen for evaluation literacy. It is the single highest-signal indicator that someone has actually shipped LLM systems under a performance bar rather than watched tutorials about them.

The reason it's such a clean signal is that you cannot fake having designed real evals. Anyone can say "we tested it." A practitioner can tell you what metric they defined and why, how they assembled the test set, why it had to be refreshed as the data drifted, and — the part that only comes from production scars — which specific failure mode the eval was built to catch. They've sat through a "the new prompt feels better" argument and know it's unwinnable without numbers, so they built the numbers.

This maps directly onto a discipline that has a name now: eval-driven development, the LLM-shaped descendant of test-driven development. You decide what success means and how you'll measure it before you build, because with a nondeterministic component you otherwise have no ground truth and no way to know whether a change helped. Notice that this skill has nothing to do with training models. It's a measurement and systems-design skill. An ML engineer might have it, but their ML background didn't give it to them — and many strong ML engineers don't have it, because their world had clean train/test splits and accuracy numbers handed to them by the framework.

A useful interview move: ask the candidate to walk through an eval they actually built. Push on the dataset rationale and the failure modes it caught. Someone who has done it lights up and gets specific. Someone who hasn't retreats to generalities about "accuracy" with no methodology underneath. The gap is unmistakable in about five minutes.

What Else Actually Separates Lab From Production

Eval design is the headline, but a handful of other competencies reliably distinguish people who've shipped from people who've prototyped. None of them are model-training skills.

• Cost as a first-class number. A production AI engineer can model the per-task profit-and-loss of a feature — tokens in, tokens out, cache hit rate, which calls route to the expensive model and which to the cheap one. Someone who has never thought about inference cost will build something that works in the demo and is uneconomic at scale.
• Context engineering over prompt cleverness. The mature version of "prompt engineering" isn't magic incantations; it's deciding what information the model sees, in what order, retrieved how, and trimmed to fit. The best practitioners version, review, and test system prompts the way they review code, because an unreviewed prompt is unreviewed production logic.
• Failure-mode intuition for agents. Multi-step agents fail in ways single calls don't: loops, compounding errors, tool misuse, silent wrong-but-plausible output. Knowing the failure taxonomy — and designing recovery, timeouts, and human checkpoints around it — is the mid-to-senior dividing line.
• Least-privilege tool design. The senior-level tell. An agent with broad tool access and no kill switch is a liability waiting to act. Engineers who think about excessive agency, scoped permissions, and circuit breakers are designing for the day the model does something dumb with real consequences.
• Frontier-model fluency. The field moves weekly. Strong candidates read primary documentation and know the current model lineup and its tradeoffs. A quick fabrication check — ask them to name the current model generation and rough pricing — separates people who track the field from people reciting stale secondhand summaries.

Run down that list and notice what's absent: no loss functions, no optimizers, no data labeling pipelines, no distributed training. The skills that make an AI engineer effective are a different stack than the skills that make an ML engineer effective. Overlap exists at the edges — both benefit from statistical intuition and clean engineering — but the centers of mass are far apart.

What To Actually Do About It

The fix is not to fire your ML team and hire a new crew. It's to stop conflating two roles and staff each one deliberately.

Start by naming the role. Write down what an AI engineer is responsible for — reliable LLM systems, evals, cost, agent safety — as competencies distinct from model development. The moment you can articulate it, you can level it, interview for it, and tell the two career tracks apart. Some of your strongest ML engineers will want to cross over, and some will be great at it; let that be a deliberate transition with its own ramp, not an assumption that the skills transfer for free.

When you interview, lead with evals and cost, not with model theory. Ask for an eval they built and a feature whose unit economics they had to defend. Probe agent failure modes for senior roles and tool-permission design for staff roles. These questions surface production experience that derivation-of-backprop questions completely miss.

And when you plan the org, treat the two as parallel ladders that occasionally hand work to each other, not as one team with a new mandate. The ML team still owns the models you actually train. The AI engineering function owns everything built on models you don't. Conflate them and you'll keep shipping prototypes that demo beautifully and die in production — wondering why the people who built your best model can't seem to make the thing on top of it reliable. They can't because that was never their craft. It's a different one, and now it needs its own name on the door.