24 posts tagged with "developer-tools"

There's a specific kind of argument you can only have with a language model. You paste in your code. It rewrites a working call into one that hasn't existed since a major version ago. You correct it. It apologizes, agrees with you, and on the next turn does the exact same thing. You are not fighting ignorance. You are fighting a confident, well-rehearsed memory of a different version of your stack — and that memory was reinforced by more training examples than your correction will ever carry.

This is the failure mode I've come to think of as fighting the prior. The model's parametric knowledge — everything it absorbed during training — contains the popular, the outdated, or simply the different version of the framework you're actually using. When your context and its prior disagree, the prior often wins. And unlike a plain hallucination, this one is dangerous precisely because it's plausible: the deprecated API used to be correct, so the code looks right, passes a casual read, and sometimes even compiles.

Your most productive engineer just gave notice. You know the drill: transfer the tickets, document the deploy process, hand off the on-call rotation, schedule the brain-dump sessions. The offboarding checklist has been refined over decades, and it covers everything the company thinks it owns.

Here's what the checklist misses in 2026: that engineer's home directory contains a personal CLAUDE.md tuned over eighteen months, a dozen custom skills that encode exactly how to navigate your gnarliest subsystem, memory files where their agent accumulated hard-won facts about your codebase, and harness settings calibrated through hundreds of sessions of trial and error. None of it lives in the repo. None of it transfers with the account.

On their last day, IT deactivates the laptop, and months of compounded agent configuration — the difference between their agent shipping features unattended and everyone else's agent flailing — evaporates without anyone noticing it existed.

Somewhere on your laptop right now sits the most detailed record of engineering decision-making your team has ever produced, and nobody has read a word of it. Every agent session — every Claude Code run, every Cursor conversation, every hours-long refactoring saga — gets logged as a transcript. It contains the three approaches that were tried and abandoned, the constraint that forced the ugly workaround, the moment you overruled the model and why. Then the session ends, the PR merges with a one-line commit message, and all of that context goes into a JSONL file that no one will ever open.

We used to lose this information honestly. The reasoning behind a design choice lived in someone's head, decayed over months, and walked out the door when they changed jobs. There was nothing to preserve because nothing was written down. That excuse is gone. The reasoning is now written down, verbatim, timestamped, machine-readable — and we're treating it as disposable exhaust.

You are halfway through renaming a function when the file reloads under your cursor. The diff you were staging no longer matches the working tree. Your dev server hot-reloads twice for no reason you can see, and a test that passed ten minutes ago now fails in a file you never opened. Nothing crashed. Nothing warned you. You and your coding agent have just been editing the same working tree at the same time, and you found out the way most teams find out: through mystery diffs.

Databases solved this problem fifty years ago and gave it a name — concurrency control. Two writers touching shared mutable state need either a lock, an isolation boundary, or a merge protocol, and the choice among those is a design decision with known trade-offs. Yet most engineering teams adopting coding agents never make that decision explicitly. They drop a second writer into a single working tree, keep the habits of a single-writer world, and then file the resulting weirdness under "AI being flaky." It is not flakiness. It is a race condition, and you are one of the racers.

Count the readers of your stack traces. For most internal tools, the answer used to be "one tired engineer, occasionally." Today the highest-volume reader of your error output is almost certainly a language model inside a retry loop. Coding agents read your linter warnings, your CLI usage strings, your API error bodies, and your test failures thousands of times a day — far more often than any human ever will. And unlike the human, the agent takes every word literally.

That changes what an error message is. It is no longer documentation of a failure. It is an instruction injected into the context window of the next attempt — a prompt you wrote months ago, now steering fleets of agents you've never met. A precise error converges the loop in one retry. A vague or misleading one sends the agent spiraling: wrong fixes, --no-verify workarounds, hallucinated flags, burned tokens. If you maintain a tool, a service, or a build system, you are already doing prompt engineering. You're just doing it in your error strings, and probably by accident.

Ask a coding agent to build a React component and it writes idiomatic, hook-shaped, accessibility-annotated code on the first try. Ask the same agent to use your in-house ORM — the one your platform team has maintained for six years, the one with excellent docs and a hundred internal consumers — and it hallucinates methods that don't exist, invents configuration options from some other library, and confidently ships code that compiles against an API it made up.

The difference isn't quality. Your ORM might be better-designed than half the open-source libraries the model handles flawlessly. The difference is training data. React has millions of public repositories behind it; your framework has zero. In the vocabulary of natural language processing, your internal framework is a low-resource language — and every consequence NLP researchers documented for low-resource languages now applies to your codebase.

There is a class of AI-generated bug that passes code review almost every time, and it isn't the hallucinated function or the fabricated package. It's the perfectly idiomatic code — idiomatic for the version of your stack that existed when the model's training data was frozen. The model writes a tailwind.config.js for a project running Tailwind v4, reaches for a class component lifecycle method in a hooks codebase, or calls an API that was deprecated three minor versions ago and removed in the one your lockfile actually pins. Nothing about the code looks wrong. It looks like code from a well-regarded tutorial. The tutorial is just from 2024.

Call it the training-cutoff bug class: defects that exist not because the model reasoned badly, but because the model's knowledge of your dependencies has a timestamp and your lockfile doesn't care. An ICSE 2025 study of seven code models across eight popular Python libraries found deprecated-API usage rates of 25–38% in plausible completions — and when the surrounding code already contained outdated patterns, that rate climbed to 70–90%. These aren't rare edge cases. They're the default failure mode of asking a frozen artifact to write for a moving target.

When an agent joins your team, the nearest analogy in every engineering manager's head is the new hire. So the playbook writes itself: give it a sandbox and read-only logs, scope the first tasks small, pair with it, expect a ramp-up period, and grow it into bigger work as trust accumulates. It feels responsible. It feels like the same patient management that turned your last junior into a senior.

It is also a category error — not a slightly imperfect analogy, but a wrong one. A junior engineer is a person who does not yet know your system. An agent is a stateless function that will never know your system, no matter how many times it touches it. Those are different kinds of things, and the management instincts that work for one quietly misallocate your attention on the other.

The reason this matters is that the metaphor doesn't just mislead — it tells you to invest in the wrong place. "Grow the agent" is not a strategy. The agent is fixed. Everything you can actually change lives outside of it.

Most teams building AI coding assistants reach for the same off-the-shelf RAG pipeline they use for document retrieval: chunk the source files by token count, embed the chunks, store them in a vector database, query by semantic similarity. The pipeline works well enough on prose. On code, it quietly fails — and the failures are hard to see in aggregate metrics, because the retrieved chunks look plausible right up until the model generates code with the wrong return type, calls a function with the wrong signature, or misses a dependency that only exists three hops down the call graph.

The problem isn't the embedding model or the vector database. It's the chunking strategy. Code is not prose. It has structural properties — dependency graphs, call chains, type signatures, scope hierarchies — that token-based chunking destroys before the retriever ever sees them. Fixing this requires rethinking how you decompose code before it ever reaches the embedding step.

There's a specific failure mode that quietly destroys AI product metrics without anyone noticing. Your dashboard shows a 34% suggestion acceptance rate, strong DAU, and growing feature engagement. What the dashboard doesn't show is that 60% of those accepted suggestions get immediately rewritten, the users who "engage" most are the ones who click the AI output, select all, and type their own response anyway, and the feature has zero measurable effect on downstream task completion.

This is the quiet quitter pattern: users who systematically route around an AI feature while still generating all the surface metrics of engaged users. They don't disable the feature — they just ignore its output. In your analytics, they look identical to your best AI users.

A 2025 controlled trial gave experienced developers access to AI coding tools and measured whether they got faster. The developers predicted a 24% speedup. After completing the study, they reported feeling roughly 20% faster. Objective measurement showed they were actually 19% slower.

This isn't a story about AI hype. It's a story about tacit knowledge — the undocumented "why" that lives inside every mature codebase and cannot be recovered by reading the code alone. AI agents are remarkably productive in greenfield systems precisely because there is little tacit knowledge to violate. They degrade in mature codebases for exactly the same reason.

The default mental model for an AI coding tool is a panel inside VS Code. A chat box, a few inline suggestions, maybe an "apply diff" button. That framing is two years out of date. The leading products in the category are not VS Code extensions; they are full editors that happen to look like VS Code on launch. Cursor is a fork. Windsurf is a fork. Zed is a from-scratch native editor. The pattern is not coincidence — it is what happens when an agent's surface area finally exceeds what the host editor's plugin API was designed to support.

If you are building a coding agent and still treating "ship a plugin" as the obvious distribution choice, you are about to hit the same wall the leaders walked into around 2024 and chose to climb. The wall has a name: the plugin API was built to add features to an editor controlled by humans, not to host an autonomous agent that wants to control the editor.