23 posts tagged with "ci-cd"

A human engineer who sees a test fail on a change that couldn't possibly have caused it does something an agent cannot: they shrug. They hit rerun, mutter about the CI gods, and move on. That shrug encodes years of accumulated context — this test has been flaky since March, that service's staging environment falls over on Mondays, nobody trusts the WebSocket suite. A coding agent has none of this. It sees a red X and takes it as ground truth, because everything in its training and its prompt tells it that a failing test means the code is wrong.

What happens next is the expensive part. The agent doesn't shrug — it acts. It "fixes" code that was never broken. It reverts a correct change because the suite went red after applying it. It burns through its token budget chasing a phantom, adding retries and sleeps and defensive checks to code paths that were fine, until the flaky test happens to pass and the agent concludes its last mutation was the cure. Nondeterminism in your test substrate was always a tax on human attention. For agent loops, it's something worse: corrupted training signal, injected directly into the decision-making of a system that acts on it at machine speed.

Somewhere in your codebase there is a config file that says something like model: small-and-cheap under the test profile and model: frontier under production. It felt responsible when someone added it — why burn frontier-model tokens on CI runs that fire twenty times a day? But that one line quietly repealed a rule your team has followed for fifteen years without thinking about it: the environment you test in should behave like the environment you ship to.

The twelve-factor methodology called this dev/prod parity, and we got so good at it that we stopped noticing it. Docker gave us bit-identical runtimes. Infrastructure-as-code gave us identical topology. Then we put a language model in the middle of the request path and reintroduced the exact gap we spent a decade closing — except this time the divergent component isn't a database version. It's the part of the system that makes the decisions.

Your coding agents just made writing code the cheapest part of shipping software. They did not make landing it any cheaper. Teams with high AI adoption merge nearly twice as many pull requests as they did before — and their delivery metrics barely move, because every one of those PRs still has to squeeze through the same review pipeline, the same CI fleet, and the same merge queue that was sized for human typing speed. The constraint didn't disappear. It moved downstream, to the narrowest pipe in the system: the serialized path between "approved" and "on main."

This is a classic theory-of-constraints story, and most engineering organizations are living through it right now without naming it. When one developer can direct five or ten agents in parallel worktrees, PR volume stops tracking headcount. But merge throughput still tracks something much more rigid: how many candidate states of main your CI can validate per hour. That number is governed by test suite duration, runner capacity, flake rate, and queue mechanics — none of which got faster when your agents did.

A team I worked with spent three days hunting a 22% latency regression in their agent loop. They blamed a new tool router. They blamed a switched model version. They blamed the JSON schema validator they had quietly upgraded the week before. They eventually found the culprit two layers below their code: a runner image had rolled forward, the new image defaulted the cpufreq governor to schedutil instead of performance, and the burstiness of an agent's tool-call loop made schedutil's ramp-up latency visible in p95. The model was fine. The agent was fine. The kernel changed its mind about how to clock the CPU between micro-bursts of work, and the entire benchmark moved.

This is the failure mode most agent teams never see, because they never look. Your CI benchmark numbers are not measurements of the model or the agent. They are measurements of a stack that happens to include a model, a network, a shared VM, a hypervisor scheduler, a cache hierarchy with unknown neighbors, and — most quietly — a frequency-scaling policy that gets to decide whether a given millisecond of compute runs at 1.0 GHz or 3.6 GHz.

The agent's diff was green on your laptop. Tests passed, lint passed, the dev server hot-reloaded clean. You let it open the PR, and ninety seconds later CI is red on a step that has nothing to do with the change: a missing CLI, an env var the agent never declared, a Node version that resolves differently because your .nvmrc resolves through a global shim that the runner does not have. The agent did not write a broken diff. It wrote a diff that depends on your machine, and your machine and the runner are not the same computer.

"Works on my machine" was a human bug. The fix was discipline — pin versions, write Dockerfiles, read the CI logs. Coding agents inherited the bug at scale and removed the discipline that used to compensate, because the agent does not know which of the things it relied on came from the repo and which came from the warm sediment of your shell history. Every developer's laptop is a uniquely configured environment that the agent absorbs without naming. Then the same agent runs in a runner that is none of those things, and the failure mode looks like the agent's fault when it is actually an environmental contract that nobody wrote down.

A flaky test gets quarantined at 3:17 AM. The on-call rotation does not page, because nothing failed — the agent decided the failure was noise, opened a small PR labeled chore: quarantine flaky test, marked the change as a self-merge under the ci-bot service account, and went back to watching the queue. Six days later a customer reports that a feature has been broken since Tuesday. The test was not flaky. It was the only thing standing between a real regression and production, and the agent's confidence threshold was set high enough to make a decision but low enough to be wrong.

This is the part of agentic CI that the marketing decks skip. Wiring an agent into your pipeline to triage failures, downgrade dependencies on security alerts, and propose dependency bumps is straightforward in 2026 — the tools exist, the integrations are one config file away, and the productivity story is real. The part that nobody writes a runbook for is the new operational class you just created: an actor with merge rights that runs at 3 AM with no human in the synchronous loop, and an SRE handbook that assumed humans were the source of intent.

Open a prompt file in any production LLM repo and watch the reviewer's eyes glaze over. The diff is fifteen lines of natural language with a tweaked few-shot example, a reworded instruction, and a stray trailing space the editor left behind. There is no syntax check that ran on it, no linter complaining about contradictory instructions, no scanner that noticed the few-shot example contains a real customer's email address from last Tuesday's support trace, and no smoke eval that confirmed the change didn't tank latency on the prompts the system actually serves. The reviewer approves on vibes — the same way teams approved HTML template diffs in 2008 — and then production telemetry catches the regression six hours later.

The inner-loop tooling around code has had two decades to mature. The inner-loop tooling around prompts is somewhere between "we have a .md file in git" and "we ran promptfoo once after onboarding." The gap is widening because prompts are now the higher-leverage edit in many systems: a thirty-line system-prompt change moves more behavior than a thousand-line service rewrite, and it ships through a review process that treats it like a Word document.

The fastest way to discover that your AI team has no eval discipline is to ask three engineers, in separate Slack DMs, "did your last prompt change improve quality?" — and watch them answer yes, all three of them, with three different numbers, against three different slices, on three different laptops, none of which is reproducible by anyone else in the room. That isn't an evals problem in the textbook sense. The textbook says you don't have evals. The reality is worse: you have too many evals, each of them privately owned, each of them measuring something real, and none of them rolling up into a single number the org can plan against.

This is the shadow eval anti-pattern, and most AI teams ship with it for longer than they admit. It looks productive — every engineer has a notebook, every PR comes with a screenshot of a pass rate, every standup mentions a "win on the long-tail slice" — and it survives quarterly reviews because the bar for "we do evals" is so low that running anything counts. But the org has no signal. Leadership cannot tell whether last month's three prompt edits moved the product forward or sideways, because the three engineers measured against three private slices and stopped tracking the previous baseline the moment they switched files.

A six-line system prompt edit lands in a pull request on Tuesday afternoon. The diff is in plain English. Two reviewers eyeball the new wording, agree it reads more naturally, hit approve. The PR merges in under a minute. By Friday, support is fielding tickets about an agent that suddenly refuses to summarize documents over a certain length, won't quote sources, and inexplicably starts every reply with "Certainly!" — a behavior nobody asked for and the diff didn't predict.

This is what happens when a team that has spent a decade learning to review code regresses to first-week behavior the moment the artifact is a prompt. The diff looks harmless because it reads like English, and English is what humans review with their eyes. The discipline that makes code review work — running the tests, examining the blast radius, treating "small changes" with appropriate skepticism — quietly does not transfer. The wording got better; the behavior got worse; nobody noticed until users did.

The number that decides whether a model goes to production is being produced by a Jupyter notebook running on a single engineer's MacBook, against a CSV that lives in a Slack DM, scored by a judge model that nobody pinned. Two weeks later, after the engineer has touched the notebook three more times and the API provider has silently shipped a minor model update, nobody on the team can reproduce the number — including the engineer who originally generated it. And yet that number is the gate. It decided that GPT-4o-mini was good enough to replace GPT-4 in the customer support flow. It decided the new prompt template shipped. It decided the fine-tune was promoted. The team is treating it like a load-bearing artifact and storing it like a sticky note.

This is the eval gap. The industry has spent five years writing about evaluation as a methodology problem — which scoring technique, which judge model, which rubric, which dataset — and almost no time writing about evaluation as an engineering problem. But the moment your eval suite starts gating production releases, it inherits every requirement that the rest of your production stack lives by: reproducibility, version control, ownership, observability, dependency management, latency and reliability budgets, and a pipeline that survives the engineer who built it leaving the team. Most teams skip this layer entirely and discover its absence only after a major incident, usually one where the eval score said green and the customer experience said red.

Most teams that say "we have evals" mean: there is a dashboard, somebody runs the suite weekly, and the numbers get pasted into a Slack channel that nobody reads. Reviewers approve a prompt change without ever seeing whether it moved the suite, and the regression shows up two weeks later in a customer ticket. The eval exists; the eval is not in the loop.

The fix is structural, not motivational. Evals only gate quality when they live where the change lives — in the pull request comment, next to the diff, with a per-PR delta and a regression callout that the reviewer cannot scroll past. Anywhere else, they are a performative artifact: real work was done to build them, and they catch nothing.

A team I worked with recently had three different "summarize this contract" prompts. One lived in a Notion page that the legal-tech squad copy-pasted into their service. One lived in a prompts/ folder in the customer-success backend, slightly modified to handle their tone preferences. One lived inline in a Python file inside the data team's notebook, hardcoded between two f-string interpolations. When OpenAI deprecated the model they all ran on, the migration plan involved Slack archaeology — each owner had to be tracked down, each variant had to be re-evaluated, and two of the three subtly broke in production for a week before anyone noticed.

This is what a prompt cookbook looks like at scale. Cookbooks make sense for ten prompts and one team. They become unmanageable somewhere around a hundred prompts and four teams. By the time you're running an AI organization, your prompts/ folder of .md files behaves exactly like vendored copy-paste code from 2008: every consumer has its own snapshot, drift is invisible, and breaking changes ripple outward in unpredictable ways.