82 posts tagged with "ai"

Every annual planning cycle in every engineering org runs on the same hidden equation: roadmap ambition divided by engineer output equals requisitions. It has been true for so long that nobody writes it down anymore. You size the work, you divide by what a team can ship in a year, and the remainder becomes a hiring plan. Finance builds the budget around it, recruiting builds pipelines around it, and managers build careers around it.

That equation quietly broke. In an agent-heavy org, the marginal unit of engineering output is no longer another senior hire — it is tokens plus the review bandwidth to absorb what those tokens produce. NVIDIA now hands engineers token budgets worth roughly half their base salary, and Jensen Huang has said he would be "deeply alarmed" if a $500,000 engineer consumed less than $250,000 of tokens a year. Whether or not you take the specific ratio seriously, the structural point stands: a company can now convert dollars into working code through two different doors, and the annual plan only has a form field for one of them.

Last month I needed to reconcile two CSV exports with slightly different column conventions — a task I'd normally solve by finding a diff tool, reading its docs, and fighting its assumptions for twenty minutes. Instead I asked an agent to write me a fifty-line script. It ran once, produced the answer, and I deleted it. Total elapsed time: ninety seconds. The script never entered version control, never got a name, and never will be seen again.

That transaction — write it, run it, delete it — is quietly becoming the default mode for a whole class of work. When code generation costs approach zero, the cheapest correct move is often a bespoke one-off, not a general-purpose tool. Finding an existing utility means searching, evaluating, installing, configuring, and trusting it. Generating a disposable one means describing what you want. For narrow, well-specified tasks, the second path now wins on every axis except one: nobody is watching what gets created.

Every team building agents eventually hits the same wall. The system prompt started at 400 tokens. Then someone added the database migration checklist. Then the postmortem template, the deploy runbook, the style guide for customer emails. Eighteen months later it's a 9,000-token monolith that nobody dares to edit, because changing the line about rollback procedures somehow degrades the agent's tone in support tickets. You've built the prompt equivalent of a single 50,000-line main.c — and everyone is statically linking against it.

The instinct is to reach for RAG: chunk the runbooks, embed them, retrieve on demand. That fails in a subtler way. RAG is built to retrieve facts, and facts degrade gracefully when fragmented — three out of five relevant chunks about your billing model still tells the agent most of what it needs. Procedures don't degrade gracefully. A database migration runbook retrieved at 60% is not 60% useful; it's a production incident. Steps 1 through 4 without step 5 ("verify replication lag before cutting over") is worse than no runbook at all, because the agent now acts with confidence it hasn't earned.

Somewhere, right now, an AI assistant is explaining your product to a prospect. It is quoting a price you changed eighteen months ago, recommending an integration you sunset last quarter, and suggesting an API endpoint that returns 410 Gone. You will never see this conversation. No analytics event fires. No session recording exists. The prospect either believes the wrong answer and files a confused support ticket, or believes the wrong answer and quietly buys from the competitor the model mentioned in the same breath.

This is not a hypothetical future problem. AI referrals already account for meaningful traffic — up to 5–8% for some technology and e-commerce sites — and the answers behind those referrals are generated from whatever the model absorbed about you, whenever it absorbed it. Your marketing team spent a decade learning to monitor brand search, review sites, and social mentions. Almost nobody is monitoring the surface that is growing fastest: what the models say when someone asks about you.

A polished eight-page design doc used to mean something before anyone read a word of it. The existence of the artifact was evidence: someone spent two weeks thinking about failure modes, arguing with themselves about tradeoffs, and pre-empting the objections they knew reviewers would raise. The document was a receipt for the thinking. Reviewers could triage by polish alone, because polish was expensive to fake.

That correlation is now dead. An agent produces a comprehensive, well-structured, diagram-laden RFC in under an hour — complete with an alternatives-considered section for alternatives nobody actually considered. The artifact survived; the signal it carried did not. And most engineering organizations are still running a review process that was designed, implicitly, around the old cost of writing.

Two years ago, standing up an AI center of excellence was the responsible move. Nobody knew how to evaluate a model, procurement had no idea what an inference contract should look like, and legal wanted one throat to choke. Concentrating the ten people who understood any of it into a central team was obviously correct.

Today that same team is the reason your product engineers wait six weeks to change a prompt. The CoE reviews every system-message edit, owns the only eval harness in the company, and gates model upgrades behind a committee that meets biweekly. Teams have noticed. They ship on personal API keys, run evals in notebooks the CoE never sees, and paste customer data into whatever tool answers fastest. You did not prevent shadow AI — you created it, and gave it a governance body to hide from.

The diff looks clean. The agent imported the right module, called what looks like the right function, and TypeScript stayed quiet. The PR description even cites the docs. Then the build runs in CI and the call explodes with TypeError: x is not a function — because the function was split into two in a minor bump eight months ago, and the agent generated against the version of the library that existed inside its training data, not the version installed in your package.json.

This is not the kind of failure the "LLMs hallucinate" frame prepares you for. The model isn't inventing an API that never existed. It's remembering an API that existed once and doesn't anymore. The mental model the agent is reasoning from is a snapshot frozen at training time. The world has moved on. The codebase has moved on. And the agent has no idea, because nobody told it.

A reasoning model emits a chain-of-thought trace because that is how it computes. A product team renders that trace in the UI because hiding it feels like throwing away tokens the user paid for. Those are two different decisions, and almost nobody on the product side notices they made the second one. The trace becomes a panel, the panel becomes a feature, the feature gets a docs page, and six months later someone in a quarterly review asks why the support queue is full of users arguing with the reasoning instead of the answer.

The trace is debug output. It exists for engineers who need to know why the model picked one tool, hedged on a date, or quietly switched personas mid-paragraph. Pushing it to the end user without a design pass is the AI-product equivalent of leaving console.log calls in production and calling them "transparency." It looks like a feature, it costs almost nothing to render, and it quietly degrades trust in ways that don't show up in any of the dashboards the team built.

Every product team building with AI faces the same fork in the road: should the AI advise humans, or should it act on its own? The framing sounds philosophical, but the answer is actually measurable — and getting it wrong is expensive in ways that don't show up until six months after launch, when your override metrics look fine and your user trust scores are quietly collapsing.

Klarna replaced 700 customer service agents with an autonomous AI system in early 2024. By 2025, the CEO admitted they had "gone too far" and began quietly rehiring humans for complex cases. The AI handled 2.3 million conversations in a month and resolved issues in under 2 minutes instead of 11. The numbers looked great. The underlying problem — that customer service for financial products requires empathy and judgment, not just resolution speed — showed up later, in declining satisfaction on anything outside the happy path.

In early 2026, Atlassian reported something that hadn't happened in the company's history: a decline in enterprise seat counts. For a company whose entire growth model rests on expansion revenue — selling more seats as customer organizations grow — this was a structural alarm, not a blip. The proximate cause wasn't churn or product failure. It was that Atlassian's own AI features had made teams so much more productive that fewer seats were needed to do the same amount of work.

This is the AI efficiency paradox: build a feature that genuinely saves users time, and you may be training them to need less of your product. The more useful your AI, the faster your pricing model breaks.

When your AI feature ships and the metrics light up — DAU spikes, NPS climbs, thumbs-up feedback floods in — you could be looking at genuine product-market fit. Or you could be watching the first act of a two-part story where the second act ends with a retention cliff nobody saw coming.

The problem is these signals are structurally broken for probabilistic AI features. They were designed for deterministic software where "activated" means something, where a five-star rating predicts future use, where the novelty fades in days rather than masking a six-month churn wave. AI features behave differently, and the standard PMF toolkit is calibrated for the wrong inputs.

Your team ships an AI feature. You celebrate the localization work: every button label, tooltip, and error message has been translated into twelve languages. The product manager signs off. The feature goes live globally.

Then, six weeks later, a user in Germany posts a screenshot. The AI's response has the right words but wrong register — awkward formality for a casual support context. A Japanese user reports that structured outputs contain dates formatted as MM/DD/YYYY, confusing their downstream tooling. A Brazilian support engineer notices the AI occasionally slips into English mid-sentence when reasoning through complex queries. These aren't infrastructure failures. Your dashboards show green. But for non-English users, the product is quietly worse.

The root cause is almost always the same: teams translate UI strings but leave system prompts in English. It feels like localization. It isn't.