12 posts tagged with "hiring"

A VP of Engineering decides the company needs to "do AI." There is already a machine learning team — three people who built the recommendation model, tuned the fraud classifier, and maintain a feature store. The obvious move is to point them at the new LLM initiative. They know the math. They've shipped models. How different could it be?

Six months later the prototype demos beautifully and dies in production. Nobody can say why the agent occasionally books the wrong meeting, the cost per request is four times the estimate, and there is no way to tell whether last week's prompt change made things better or worse. The ML team is frustrated because none of the tools they're good at — gradient descent, data pipelines, hyperparameter sweeps — apply to a model they can't retrain and can't see inside.

This is the most common org mistake in AI right now, and it comes from a reasonable-sounding assumption: that the craft of building models and the craft of building on models are the same job with a different label. They are not. They overlap less than "frontend engineer" and "backend engineer" do.

The candidate hit the wall on the system-design question, paused for two seconds, and said: "I'd just prompt it." Your most senior interviewer wrote strong hire — this is exactly how good engineers work in 2026. Your second-most-senior interviewer wrote no hire — handing the problem to a chatbot is not engineering. Same five words. Same forty-minute window. A forty-point gap on the same scorecard.

The candidate didn't fail your loop. Your loop failed to have an opinion. And the worst part of the debrief is not the disagreement — it's the way each interviewer is so confident their read is the correct one that the meeting devolves into a referendum on AI itself rather than on whether this human can ship.

You open a requisition for an "eval engineer." A week later your recruiter asks the obvious question: what level is this, and what does a good resume look like? You don't have an answer. The title didn't exist two years ago. There is no leveling rubric, no canonical interview loop, no pool of people with the words "eval engineer" already on their LinkedIn. You are hiring for a job the industry has not agreed exists.

This is the quiet bottleneck in shipping AI systems. The model is available. The infrastructure is rentable. What you cannot buy off the shelf is the person whose actual job is keeping a prompt-driven system honest — and your hiring machinery, built for roles with decades of precedent, has no slot for them.

The instinct is to wait. Wait for the title to standardize, for the bootcamps to mint candidates, for someone else to write the leveling guide you can copy. That instinct is wrong. The work exists now whether or not the title does, and the teams staffing it now are the ones learning what "good" looks like before their competitors even open the req.

A two-hour take-home and a 45-minute algorithm round were never the point. They were proxies. The take-home stood in for "can this person ship a feature," and the whiteboard round stood in for "can this person decompose a problem under pressure." For two decades those proxies held up well enough that most teams stopped questioning them. They were cheap to administer, easy to grade, and roughly correlated with the thing you actually cared about.

Coding agents broke the correlation without breaking the format. The interview still runs. It still produces a score. The score still feels like signal. But the gap between what the interview measures and what the job requires has widened to the point where a green result certifies almost nothing — and most hiring pipelines have not noticed, because nothing visibly failed.

This is the quiet kind of invalidation. Not a process that collapsed, but a process that kept running after its assumptions stopped being true.

The coding interview was built to isolate a single variable. Put a person in a room, give them a problem, take away their references, and watch whether they can turn the problem into working code by themselves. Everything about the format — the whiteboard, the blank editor, the prohibition on looking things up — exists to strip away collaborators and tools so you measure one isolated skill: can this person, alone, write correct code under pressure.

That skill is no longer the one the job exercises. Day-to-day engineering in 2026 is a collaboration between an engineer and an agent. The engineer decides what to build, the agent drafts most of the code, and the engineer's real work is reviewing, correcting, and deciding when the agent is confidently wrong. The interview measures solo code production. The job rewards directing a tireless, fast, occasionally hallucinating collaborator. The proxy and the target have come apart, and most hiring pipelines haven't noticed.

This is not a complaint about cheating, though cheating is the symptom everyone fixates on. It's a measurement problem. When you can no longer observe the variable your test was designed to isolate, the test stops producing signal — and a test that produces no signal while everyone still trusts it is worse than no test at all.

The constraint on your AI roadmap isn't GPU capacity, model availability, or prompt-engineering taste. It's the calendar of one or two engineers who actually know how to build an eval that catches a regression. Every PM with a feature is in their queue. Every model upgrade is in their queue. Every cohort drift, every prompt revision, every "is this judge still calibrated" question lands in the same inbox. And the engineer in question said "no, this isn't ready" three times this quarter, got overruled twice, watched the regression compound in production, and is now updating their LinkedIn.

This is the eval bottleneck, and most orgs don't see it until it bites. Through 2025 the visible scaling story was AI engineers — hire AI engineers, ship AI features, iterate on prompts, swap models. By Q1 2026 the throughput problem moved one layer down. The team that doubled its AI headcount discovered that adding more feature engineers didn't make features ship faster, because every feature still needed an eval, and the eval engineer was the same person.

A team I worked with last quarter rejected three candidates in a row from their AI engineer pipeline. All three failed the coding screen — the kind of problem where you implement a sliding-window deduplicator under a 35-minute timer. The team then hired the candidate who passed it. Four months later that engineer was the one who shipped the feature where the eval scored 92% in CI and the support queue lit up the day after launch. The eval was measuring exact-match against a curated test set. Production users phrased their queries differently. Nobody on the hiring panel had asked the candidate how they would have caught that gap.

That's the shape of the bug. The interview pipeline was screening for the skill that mattered least to the job and was blind to the skill that mattered most. The team did not have a "judgment" round. They had a coding round, a system-design round, and a behavioral round, and they were running the same loop they had run in 2021 — the one calibrated for engineers who were going to write deterministic code against stable libraries.

A team I know spent six months running their standard senior-engineer loop with an "AI round" bolted on. They interviewed seventy candidates. They hired three. None of the three shipped an agent that survived a production weekend. The team blamed the talent market. The talent market was fine. The loop was the problem.

The standard engineering interview was calibrated for a stack where correctness is verifiable, performance is measurable on a benchmark, and a good engineer is someone who can decompose a problem into deterministic components and reason about edge cases against a known specification. That stack still exists, and those skills still matter, but the cluster of skills that predicts shipping LLM products is largely orthogonal to it. Your loop is asking the right questions about the wrong job.

This is a structural problem, not a calibration nudge. Adding a forty-five-minute "AI round" to a loop calibrated for deterministic systems doesn't surface AI builders — it surfaces the intersection of classical-systems-strong and LLM-fluent candidates, which is a vanishingly small set, and produces six months of failed loops while everyone wonders where all the AI engineers went.

The three best AI engineers I have hired this year would all fail each other's interviews. The one who writes prompts that survive a model upgrade has never written a useful eval case in her life. The one who designs eval sets that catch the failures that matter writes prompts that other engineers refuse to extend. The one who designs guardrails that fail closed without choking the happy path has opinions about the other two that I cannot print here.

The job ladder calls all three of them "AI engineer." The calibration committee compares their promo packets as if they had been doing the same job. They have not.

The signal is gone. In a recent audit of 19,368 technical interviews, 38.5% of candidates were flagged for AI-assisted cheating, with technical roles hitting 48% and junior candidates cheating at nearly double the rate of senior ones. More damning: 61% of detected cheaters scored above the passing threshold. Without the detection layer, they would have advanced. The interview, as an instrument, is no longer measuring what it was designed to measure.

This is not a moral panic about kids these days. It is a mechanical failure of the instrument. The technical interview was calibrated for a world in which a candidate, under time pressure, in an unfamiliar environment, had to produce correct code from memory and first principles. That constraint — the thing that made the signal legible — has been dissolved by a free-tier chat window running on a second device. Every company that still runs a LeetCode-style screen is now paying to sort candidates on a test the test-taker can trivially outsource.

Most engineering teams that hire for LLM roles run roughly the same interview: two rounds of LeetCode, a system design question, maybe a quiz on transformer internals. They're assessing for the wrong things — and they know it. The candidates who ace those screens often struggle to ship working AI features, while the ones who stumble on binary search can build an eval suite from scratch and debug a hallucinating pipeline in an afternoon.

The skills that predict success in LLM engineering have almost no overlap with what traditional ML or software interviews test. Hiring managers who haven't updated their process are generating false negatives at a high rate — rejecting engineers who would succeed — while false positives walk in with solid LeetCode scores and no intuition for when a model is confidently wrong.

Most teams hiring AI engineers today are running an interview process optimized for a job that doesn't exist. They're screening for LeetCode fluency, quizzing candidates on transformer internals, and rewarding anyone who can confidently sketch a distributed system on a whiteboard. Then those same candidates join the team, struggle to debug a hallucinating retrieval pipeline, and ship a model integration that works beautifully in staging and silently degrades in production.

This isn't a talent problem. It's a measurement problem. The skills that predict success in AI engineering are largely invisible to traditional interview loops—and the skills interviews do measure correlate poorly with what the job actually requires.