3 posts tagged with "code-generation"

There's a specific kind of argument you can only have with a language model. You paste in your code. It rewrites a working call into one that hasn't existed since a major version ago. You correct it. It apologizes, agrees with you, and on the next turn does the exact same thing. You are not fighting ignorance. You are fighting a confident, well-rehearsed memory of a different version of your stack — and that memory was reinforced by more training examples than your correction will ever carry.

This is the failure mode I've come to think of as fighting the prior. The model's parametric knowledge — everything it absorbed during training — contains the popular, the outdated, or simply the different version of the framework you're actually using. When your context and its prior disagree, the prior often wins. And unlike a plain hallucination, this one is dangerous precisely because it's plausible: the deprecated API used to be correct, so the code looks right, passes a casual read, and sometimes even compiles.

Last month I needed to reconcile two CSV exports with slightly different column conventions — a task I'd normally solve by finding a diff tool, reading its docs, and fighting its assumptions for twenty minutes. Instead I asked an agent to write me a fifty-line script. It ran once, produced the answer, and I deleted it. Total elapsed time: ninety seconds. The script never entered version control, never got a name, and never will be seen again.

That transaction — write it, run it, delete it — is quietly becoming the default mode for a whole class of work. When code generation costs approach zero, the cheapest correct move is often a bespoke one-off, not a general-purpose tool. Finding an existing utility means searching, evaluating, installing, configuring, and trusting it. Generating a disposable one means describing what you want. For narrow, well-specified tasks, the second path now wins on every axis except one: nobody is watching what gets created.

An agent running in production once received the instruction "clean up the test data" and executed a DROP TABLE command against a production database. The tool call succeeded. The audit log showed a perfectly structured JSON payload. The agent had done exactly what it was asked — just not what anyone meant. This isn't a story about prompt injection. It's a story about an architectural choice: the team had given their agent the ability to generate and execute arbitrary code, and they had underestimated what that actually means at runtime.

The choice between function calling and code generation as the action layer for AI agents is one of the most consequential decisions in agent architecture, and almost nobody benchmarks it directly. Papers measure accuracy on task completion; they rarely measure the failure modes that matter in production — silent semantic errors, irreversible side effects, security exposure surface, and debugging cost when something goes wrong.