Notebooks Are Hostile Territory for Coding Agents

Give a coding agent a Python module and it operates on solid ground: the file on disk is the program. Read it, edit it, run it, observe the result — the loop closes. Give the same agent a Jupyter notebook and every one of those assumptions quietly breaks. The agent edits cell 12 with full confidence, not knowing that you re-ran cell 3 an hour ago with different data, that a variable defined in a since-deleted cell is still alive in the kernel, and that the output it just read under cell 7 was produced three kernel restarts ago by code that no longer exists.

A notebook is a REPL wearing a file costume. The .ipynb on disk looks like source code, but the thing that actually determines behavior — the kernel's accumulated memory — is invisible, unserialized, and shaped by the exact sequence of human clicks that produced it. Agents are trained on the contract that code determines behavior. Notebooks void that contract, and most agent harnesses don't even know it.

The File Is Not the Program

In a normal codebase, the mapping from source to behavior is a function: same files, same inputs, same behavior. Everything agents are good at — reading code to predict what it does, editing code to change what it does, running tests to verify the change — leans on that mapping.

A notebook breaks the mapping in three distinct ways:

• Out-of-order execution. Cells run in whatever order the human clicked them. The execution counters ([3], [17], [5]) record the sequence, but nothing enforces top-to-bottom order, and the counters reset on restart. A large-scale study of over 860,000 notebook executions on GitHub found that only about 24% of notebooks ran without errors top-to-bottom, and barely 4% reproduced their stored outputs. Roughly 36% showed direct evidence of out-of-order execution.
• Deleted code, surviving state. Delete a cell that defined df_clean and the variable lives on in the kernel. Downstream cells keep working — until the next restart, when they collapse. The code on screen is now a lie: it references a definition that exists nowhere in the file.
• Stale outputs. Cell outputs are snapshots from whenever the cell last ran, not reflections of current code. Edit a cell without re-running it and the saved output below it describes a previous version of the program.

For a human, these are familiar annoyances managed with habit and memory: "I know I re-ran the loading cell, I know that plot is stale." The human carries a mental model of kernel state that the file doesn't record. An agent has no access to that mental model. It has the file — and the file is the least trustworthy artifact in the room.

Why Agents Fail Worse Than Humans Here

It's tempting to say notebooks are equally bad for everyone. They aren't. Hidden state punishes agents disproportionately, for reasons built into how agents work.

First, agents read outputs as ground truth. An agent debugging a data pipeline will read the stored output of a cell — a dataframe preview, an error traceback, a printed shape — and reason from it. If that output is stale, the agent is now reasoning from a fabricated observation. Humans discount stale outputs with context ("that's from before lunch"); agents have no timeline to discount against. A misleading observation is often worse than no observation, because it actively steers the agent's hypothesis in the wrong direction.

Second, agents verify by re-running, and re-running is exactly what notebooks make unsafe. The standard agentic loop — edit, execute, check — assumes execution is cheap and idempotent. In a notebook, re-running a cell can double-apply a transformation (df = df.dropna() twice is fine; df["price"] = df["price"] * 1.1 twice is a silent bug), re-fire an API call, or clobber state that a later cell depended on. The agent's core verification move is a mutation with side effects on an environment it can't inspect.

Third, the JSON format fights the tooling. An .ipynb file is a JSON document with embedded source strings, base64 images, and metadata. Agents that edit files as text either mangle the JSON or produce diffs no human can review. Harnesses have grown notebook-specific tooling and MCP servers that expose cell-level read/edit/execute operations, and those genuinely help — but they solve the format problem, not the state problem. An agent with perfect cell-editing tools still can't know what's in the kernel.

The result is a trap for exactly the workflow notebooks are supposed to serve. Data science is the domain where people most want agent help — exploratory analysis, feature engineering, plot iteration — and it's the domain where the medium most undermines the agent.

"Restart and Run All" Is the Only Contract an Agent Can Verify

There is one operation that collapses all the hidden state: restart the kernel and run every cell from the top. After a clean top-to-bottom run, the file and the program are momentarily the same thing. Every output is fresh, every variable traceable to visible code, every execution counter sequential.

That makes "restart and run all succeeds" the only property of a notebook an agent can actually check. Everything else — "the notebook currently works," "the outputs are correct," "my edit is safe" — is a claim about kernel state the agent cannot observe.

This suggests a discipline for agent-driven notebook work that mirrors how we treat databases, not source files:

Loading…