778 posts tagged with "llm"

There is a pattern that keeps appearing in production AI deployments, and it runs directly counter to user intuition. When a model says "I'm not sure," users tend to double-check. When a model answers confidently, they tend to trust it. The problem is that frontier LLMs are systematically most confident in exactly the domains where they are most likely to be wrong.

This isn't a fringe failure mode. Models asked to generate 99% confidence intervals on estimation tasks only cover the truth approximately 65% of the time. Expected Calibration Error (ECE) values across major production models range from 0.108 to 0.726 — substantial miscalibration, and measurably worse in high-stakes vertical domains like medicine, law, and finance. The dangerous part isn't the inaccuracy itself; it's the inversion: the same models that show reasonable calibration on general knowledge tasks become confidently, systematically wrong on the tasks where being wrong has real consequences.

GPT-4 reproduced exact passages from books in 43% of test prompts when asked to continue a given excerpt. In one 2025 study, researchers extracted nearly an entire book near-verbatim from a production LLM — no jailbreaking required, just a persistent prefix-feeding loop. If your product generates content using a language model, the copyright exposure is not a future risk. It is happening in your users' sessions today, and you probably have no instrumentation to catch it.

This is not primarily a legal article. It's an engineering article about a legal problem that engineering decisions either create or contain. Lawyers will tell you what constitutes infringement. This framework tells you where your system leaks, how to measure it, and what actually reduces risk versus what only looks like it does.

There is a quiet failure mode baked into almost every globally deployed AI product. An engineer localizes the UI strings, runs the model outputs through a translation API, has a native speaker spot-check a handful of responses, and ships. The product is technically multilingual. It is not culturally competent. Users in Tokyo, Riyadh, and Chengdu receive outputs that are grammatically correct and culturally wrong — responses that signal disrespect, confusion, or distrust in ways the team will never see in aggregate metrics.

The research is unambiguous: every major LLM tested reflects the worldview of English-speaking, Protestant European societies. Studies testing models against representative data from 107 countries found not a single model that aligned with how people in Africa, Latin America, or the Middle East build trust, show respect, or resolve conflict. Translation patches the surface. The underlying calibration remains Western.

Most engineers building multi-step agent pipelines discover the same problem about two weeks after their first production incident: they set a 5-second timeout on their API gateway, their agent pipeline has four hops, and the system behaves as though there is no timeout at all. The agent at hop three doesn't know the upstream caller gave up three seconds ago. It keeps running, keeps calling tools, keeps generating tokens—and the user is already gone.

This isn't a configuration mistake. It's a structural problem. Latency constraints don't propagate across agent boundaries by default, and none of the major orchestration frameworks make deadline propagation easy. The result is a class of failures that looks like latency problems but is actually a context propagation problem.

Your AI coding agent just generated a pull request. The code looks right. It compiles. Tests pass. You merge it. Two days later, your CI pipeline in staging starts throwing AttributeError: module 'openai' has no attribute 'ChatCompletion'. The agent used an API pattern that was deprecated a year ago and removed in the latest major version.

This is the deprecated API trap, and it bites teams far more often than the conference talks about AI code quality suggest. An empirical study evaluating seven frontier LLMs across 145 API mappings found that most models exhibit API Usage Plausibility (AUP) below 30% across popular Python libraries. When explicitly given deprecated context, all tested models demonstrated 70–90% deprecated usage rates. The problem is structural, not a quirk of a particular model or library.

A clean PDF, a capable LLM, and thirty lines of code. The demo works. You extract the invoice total, the contract dates, the patient diagnosis. Stakeholders are impressed. Then you push to production, and within a week the pipeline is silently returning wrong data on 15% of documents — and nobody knows.

This is the document AI trap. The failure mode isn't a crash or an exception; it's a pipeline that reports success while producing garbage. Building production document extraction is a fundamentally different problem from building a demo, and most teams don't realize this until they've already shipped.

Most teams make the cloud-vs-edge decision by gut instinct: cloud is easier, so they default to cloud. Then a HIPAA audit hits, or the latency SLO slips by 400ms, or the monthly invoice arrives. Only then do they ask whether some of that inference should have been local all along.

The answer is almost never "all cloud" or "all edge." The teams running production AI at scale have settled on a tiered architecture: an on-device or on-premise model handles the majority of requests, and a cloud frontier model catches what the smaller model can't. Getting that routing right is an engineering decision, not an intuition.

This is the decision framework for making it rigorously.

Most AI teams treat a passing eval suite as a signal that their system is working. It isn't—not by itself. A suite that reliably scores 87% is doing exactly one thing: telling you the system performs well on the 87% of cases your suite happens to cover. If that suite was hand-curated six months ago, built from the examples the team thought of, and never updated against live traffic, it's measuring the wrong thing with increasing confidence.

This is the eval coverage problem. It's not about whether your evaluators are accurate—it's about whether the distribution of queries in your test set matches the distribution of queries your users are actually sending. When those two distributions diverge, you get a result that's far worse than a failing eval: a passing eval sitting on top of a silently degrading product.

Your model was trained on data from last year. It was evaluated internally two months ago. It shipped a month after that. By the time a user hits a failure and you learn about it, you're already six months behind the world your model needs to operate in. This gap is not a deployment problem — it's a feedback loop problem. And most teams aren't measuring it, let alone closing it.

The instinct when a model underperforms is to blame the model architecture or the training data. But the deeper issue is usually the latency of your feedback system. How long does it take from the moment a user experiences a failure to the moment that failure influences your model? Most teams, if they're honest, have no idea. Industry analysis suggests that models left without targeted updates for six months or more see error rates climb 35% on new distributions. The cause isn't decay in the model — it's the world moving while the model stays still.

There's a pattern that shows up in almost every production AI system: the team starts with a focused system prompt, ships the feature, and then iterates. A new edge case surfaces, so they add a rule. Another ticket comes in, another rule. Six months later the system prompt has grown to 2,000 tokens and covers 20 distinct behavioral requirements. The AI still sounds coherent on most requests. But subtle compliance failures have been creeping in for weeks — formatting ignored here, a tone requirement skipped there, an escalation rule quietly bypassed. Nobody flagged it because no individual failure was dramatic enough to page anyone.

This isn't a model quality problem. It's a fundamental architectural characteristic of how transformer-based language models process instructions, and there's a substantial body of empirical research that makes the failure modes predictable. Understanding it changes how you should write system prompts.

A common assumption in AI product development goes something like this: if a model can handle a hard task, it can definitely handle an easier one nearby. This assumption is wrong, and it's responsible for a category of production failures that no amount of benchmark reading prepares you for.

The research term for the underlying phenomenon is the "jagged frontier" — AI's capability boundary isn't a smooth line that hard tasks sit outside of and easy tasks sit inside. It's a ragged, unpredictable shape. AI systems can write production-grade database query optimizers and still miscalculate whether two line segments on a diagram intersect. They can pass PhD-level science exams and fail children's riddle questions that involve spatial relationships. They can synthesize 50-page documents and then confidently hallucinate a summary of a paragraph they just read.

A team ships a RAG pipeline for internal documentation. Retrieval looks solid — the right passages come back. But in production, users keep getting stale answers. They dig into the logs and find the model is returning facts from its training data, not from the documents it was handed. The retrieval worked. The model just didn't use it.

This is the knowledge contamination problem: the model's parametric memory — the knowledge baked into its weights during training — overrides the retrieved context. It's quiet, it's confident, and it's one of the most common failure modes in production RAG systems.