75 posts tagged with "ai"

A team of twelve engineers adopts AI coding tools enthusiastically. Six months later, each engineer is merging nearly twice as many pull requests. The engineering manager celebrates. Then the on-call rotation starts paging. Debugging sessions last twice as long. Nobody can explain why a particular module was structured the way it was. The engineer who wrote it replies honestly: "I don't know — the AI generated most of it and it seemed fine."

This scenario is playing out at companies everywhere. The individual productivity story is real: developers finish tasks faster, write more tests, and clear backlogs more efficiently. The team-level story is more complicated, and most organizations aren't ready for it.

GPT-4 reproduced exact passages from books in 43% of test prompts when asked to continue a given excerpt. In one 2025 study, researchers extracted nearly an entire book near-verbatim from a production LLM — no jailbreaking required, just a persistent prefix-feeding loop. If your product generates content using a language model, the copyright exposure is not a future risk. It is happening in your users' sessions today, and you probably have no instrumentation to catch it.

This is not primarily a legal article. It's an engineering article about a legal problem that engineering decisions either create or contain. Lawyers will tell you what constitutes infringement. This framework tells you where your system leaks, how to measure it, and what actually reduces risk versus what only looks like it does.

A clean PDF, a capable LLM, and thirty lines of code. The demo works. You extract the invoice total, the contract dates, the patient diagnosis. Stakeholders are impressed. Then you push to production, and within a week the pipeline is silently returning wrong data on 15% of documents — and nobody knows.

This is the document AI trap. The failure mode isn't a crash or an exception; it's a pipeline that reports success while producing garbage. Building production document extraction is a fundamentally different problem from building a demo, and most teams don't realize this until they've already shipped.

You shipped the AI feature. You put it in the product. You wrote the help doc. And still, six months later, your most sophisticated enterprise users are copy-pasting text into ChatGPT to do the same thing your feature already does natively. This is not a training problem. It is a discoverability problem, and it is one of the most consistent sources of wasted AI investment in enterprise software today.

The pattern is well-documented: 49% of workers report they never use AI in their role, and 74% of companies struggle to scale value from AI deployments. But the interesting failure mode is not the late-adopters who explicitly resist. It is the engaged users who open your product every day, never knowing that the AI capability they would have paid for is sitting one click away from where their cursor already is.

Most enterprise security teams have a reasonably well-developed model for insider threats: a disgruntled employee downloads files to a USB drive, emails a spreadsheet to a personal account, or walks out with credentials. The detection playbook is known — DLP rules, egress monitoring, UEBA baselines. What those playbooks don't account for is the scenario where you handed every one of your employees a tool that can plan, execute, and cover multi-stage operations at machine speed. That's what deploying AI coding assistants and RAG-based document agents actually does.

The problem isn't that these tools are insecure in isolation. It's that they dramatically amplify what a compromised or malicious insider can accomplish in a single session. The average cost of an insider incident has reached $17.4 million per organization annually, and 83% of organizations experienced at least one insider attack in the past year. AI tools don't introduce a new threat category — they multiply the capability of every threat category that already exists.

Slack discovered that teams exchanging 2,000 messages converted to paid at a 93% rate. The insight sounds obvious in retrospect — engaged teams stay — but what's less obvious is the engineering consequence: Slack built their entire onboarding flow around getting teams to that message count, not around feature tours or capability explanations. They taught users about Slack by using Slack.

AI features have the same problem, but harder. There's no equivalent of "send your first message" because the capability surface is invisible. A user staring at a blank prompt box has no intuition about what's possible. This is the magic moment problem: your product has a transformative capability, but users can't imagine it until they've seen it, and they won't see it unless you engineer the path.

The data makes this urgent. In 2024, 17% of companies abandoned most of their AI initiatives. In 2025, that number jumped to 42% — a 147% increase in a single year. The technology improved; the onboarding didn't.

Cursor hit $1 billion in revenue in 2025 and lost$150 million doing it. Every dollar customers paid went straight to LLM API providers, with nothing left for engineering, support, or infrastructure overhead. This wasn't a scaling problem—it was a unit economics problem that was invisible until it was catastrophic.

Most engineering teams building AI features make the same mistake: they treat inference cost as a minor line item, ship a flat-rate subscription, and assume the economics will work out later. They don't. Variable inference costs don't behave like any other COGS in software, and the pricing architectures that work for traditional SaaS will bleed you dry the moment your heaviest users find your most expensive feature.

There is a company charging £50 per month per user. Their AI feature consumes £30 in API fees. That leaves £20 to cover hosting, support, and profit — before accounting for a single refund or churned seat. They built a product users love, grew to thousands of subscribers, and unknowingly constructed a business where more customers means more losses.

This is not a cautionary tale about a bad idea. It is a cautionary tale about a pricing architecture imported from a world where the marginal cost of serving the next user was effectively zero. That world no longer fully applies when your product calls a language model.

Traditional SaaS gross margins run 70–90%. AI-forward companies are reporting 50–60% — and the gap is mostly explained by one line item: inference. When tokens are 20–40% of your cost of goods sold, the standard SaaS playbook inverts.

You'll find out through a screenshot. A customer will post it, a journalist will quote it, or someone on your team will Slack you a link at 11pm. Your AI system said something confidently wrong — wrong enough that it's funny, or wrong enough that it could hurt someone — and now it's public.

Most engineering teams spend months hardening their AI pipelines against this moment, then discover they never planned for what happens after it arrives. They know how to iterate on evals and tune prompts. They don't know who should post the response tweet, what that response should say, or how to tell the difference between a one-off unlucky sample and a latent failure mode that's been running in production for weeks.

This is the playbook for that moment.

An agent with 95% per-step reliability sounds impressive. At 10 steps, you have a 60% chance of success. At 20 steps, it's down to 36%. At 50 steps, you're looking at a coin flip—and that's with a generous 95% estimate. Field data suggests real-world agents fail closer to 20% per action, which means a 100-step task succeeds roughly 0.00002% of the time. This isn't a model quality problem or a prompt engineering problem. It's a compounding math problem, and most teams building agents haven't internalized it yet.

This is the delegation cliff: the point at which adding one more step to an agent's task doesn't linearly increase the chance of failure—it multiplies it.

A leaked internal Google memo put it plainly: "We aren't positioned to win this arms race and neither is OpenAI." The author's argument was that fine-tuning a model with LoRA costs roughly $100, that open-source communities could replicate closed-model capabilities within months, and that "we have no moat." This was a Google researcher writing about Google. If that's true inside the world's best-resourced AI lab, what does it mean for your product team betting on a data advantage?

The honest answer is that most AI features are not moats. They are rented capabilities with a UI. But some genuinely compound — and the difference is not about how much data you have. It's about the specific mechanical conditions under which data actually creates defensibility.

Your model achieved 91% accuracy on the held-out test set. Latency is under 200ms at p95. You've cut the error rate by 40% compared to the previous rule-based system. By every technical measure, the project is a success. Six months later, leadership cancels it.

This is not a hypothetical. Eighty percent of AI projects fail to deliver intended business value, and the majority of those failures are not caused by model performance. They are caused by the gap between what engineers measure and what decision-makers understand. The technical team speaks a language that executives cannot evaluate — and in the absence of comprehensible signal, leadership defaults to skepticism.

The metrics translation problem is not a communication soft skill. It is an engineering discipline that most teams treat as optional until the funding review.