Prompt Archaeology: Recovering Intent from Legacy Prompts Nobody Documented

You join a team that's been running an LLM feature in production for eighteen months. The feature is working — users like it, the business cares about it — but nobody can explain exactly what the prompt does or why it was written the way it was. The engineer who wrote it left. The Slack thread where they discussed it is buried somewhere in a channel that no longer exists. The prompt lives in a database record, 900 tokens long, with no comments and no commit message beyond "update prompt."

Now you've been asked to change it.

This situation is more common than the industry admits. Prompts are treated like configuration values: quick to write, invisible in code review, and forgotten the moment they start working. The difference is that a misconfigured feature flag announces itself immediately. A misconfigured prompt will silently degrade behavior across a subset of edge cases for weeks before anyone notices.

Recovering intent from an undocumented production prompt is archaeological work. You're trying to reconstruct what someone was thinking from the artifacts they left behind. The methodology for doing that systematically — and the documentation format that prevents the next engineer from facing the same problem — is what this post is about.

Why Prompts Lose Their Documentation

The root cause isn't laziness. It's that prompts don't live in the same workflow as code.

When engineers write code, they open a file in version control, write a commit message, go through code review, and merge a pull request. Every one of those steps is an opportunity to leave context. When engineers write prompts, they typically iterate in a playground, copy the result into a database or config file, and move on. There's no diff. There's no review. There's often no record that anything changed at all.

Analysis of over 1,200 production LLM deployments shows that prompt updates are the single biggest trigger for production incidents — more than code deploys, more than infrastructure changes. Three words added to improve conversational tone caused structured-output error rates to spike within hours, halting a revenue-generating workflow until engineers manually rolled back the change. The prompt wasn't documented. The behavior that broke wasn't documented. The rollback took hours because nobody knew what "correct" looked like.

This is the prompt documentation debt that teams accumulate quietly. Every undocumented prompt is a ticking timer — not on when it will break, but on how hard the debugging will be when it does.

Starting the Excavation: Reading the Artifact

When you inherit an undocumented prompt, your first instinct is probably to read it carefully and try to infer intent from the text. That works for simple prompts. For complex production prompts, it's unreliable.

Long prompts accumulate layers. The original intent might be buried under subsequent patches. A section that looks redundant might be handling a specific edge case someone discovered nine months ago. Removing it will break production on a Tuesday for inputs nobody thought to test.

Start with the observable outputs instead of the text itself. Pull a sample of real production requests and their corresponding outputs. A few hundred is better than a handful. Look for:

• Output format consistency: Is the prompt always producing the same structure, or does format vary? Variance is often a sign of under-specified output constraints.
• Length distribution: Unusually short outputs often indicate refusals, formatting failures, or outputs being truncated. Unusually long outputs often indicate missing length constraints.
• Language patterns: Does the prompt have a distinct voice? Does it hedge? Does it always include caveats? These behavioral signatures were usually intentional.
• Failure signatures: Are there output patterns that appear when the model was clearly confused — hedge phrases, admissions of uncertainty, topic deflections? These tell you where the prompt was weakest and what it was trying to handle.

This output audit gives you a behavioral profile before you touch the prompt at all.

Behavioral Probing: Mapping the Edges

Once you have a behavioral profile from production outputs, the next step is systematic probing — designing inputs specifically to reveal how the prompt behaves at its limits.

Think of it as a diagnostic test suite for an unknown system. You're not testing for correctness yet (you don't know what correct is). You're mapping the response surface.

Probe along input dimensions. If the prompt handles customer support queries, test it with short queries vs. long ones, simple vocabulary vs. technical vocabulary, polite vs. frustrated tone, English vs. other languages, well-formed questions vs. rambling ones. Document which dimensions produce clean outputs and which produce variance.

Probe the edge cases that bite production systems. These are specific categories worth testing deliberately:

• Empty or near-empty inputs
• Inputs that contain the same keywords as the system prompt
• Inputs in languages the prompt wasn't obviously designed for
• Inputs with conflicting signals (polite phrasing, hostile intent)
• Inputs that are technically within scope but unusual (a question about returns in a prompt designed for product recommendations)

Test sensitivity to small variations. Paraphrase the same input a dozen ways. If the output changes significantly based on phrasing rather than meaning, the prompt is fragile. If it stays consistent, it's robust. This distinction matters when you're deciding what you can safely change.

Test temperature stability. Run the same input at multiple temperature settings. High variance at temperature 0 is unusual — if you see it, you're often looking at a prompt that relies on lucky sampling to produce good outputs rather than reliable instruction-following.

Each probe produces a data point. Together, they give you a map of what the prompt was designed to handle and where it breaks down.

Reconstructing Intent from Evidence

After behavioral profiling and probing, you should have enough evidence to form hypotheses about what the prompt was trying to accomplish.

Look for negative space: what the prompt explicitly prohibits (often the legacies of specific incidents), what it defers to humans on, what topics it refuses to engage with. These are the scars from production failures. Each one represents a decision someone made at some point for a reason you now need to infer.

Cross-reference with surrounding systems. What consumes this prompt's output? If the output is parsed by a downstream service, the schema it expects tells you what the prompt was trying to produce. If the output is shown directly to users, look at user behavior metrics — where do users act on the output vs. where do they ignore it or ask follow-up questions? That behavioral data contains intent that the prompt text doesn't.

Talk to whoever is closest to this feature's history. Not to get the "correct" answer — memory is unreliable, especially about prompt changes — but to validate your hypotheses. "I think this section was added to handle cases where users ask about competitors — does that match anything you remember?" is a useful question. "What does this prompt do?" is not.

One more useful technique: deliberately break the prompt in controlled ways. Remove a section and test whether outputs change. If removing a paragraph has no measurable effect, it was probably dead code — added for a case that doesn't appear in production traffic, or made redundant by a later change. If removing it causes failures, you now know what that section was protecting against.

Documenting What You Find

The goal of all this excavation is not just to understand the prompt yourself — it's to prevent the next engineer from having to repeat the work.

The documentation format doesn't need to be elaborate. A Markdown file alongside the prompt (or stored as metadata wherever the prompt lives) with these fields covers 90% of what matters:

Purpose: One to three sentences. What task is this prompt performing? What's the intended output? This sounds obvious but almost never gets written down.

Key behaviors: A bulleted list of behaviors the prompt deliberately implements — things it always does, things it always avoids, things it handles in a specific way. These are the things you discovered through probing that aren't obvious from the prompt text alone.

Input assumptions: What inputs is this prompt designed for? What's the expected length range, language, domain, format? What inputs are explicitly out of scope?

Output format specification: This is the field that causes the most production incidents when missing. Document exactly what the output structure should be — field names, types, whether the output is Markdown or plain text or JSON, what length bounds apply. If the output is parsed downstream, document the schema.

Known edge cases: A list of specific inputs or input patterns that behaved unexpectedly during testing, with notes on what the correct behavior is (or should be). This is the scar tissue map.

Version history: Date, author, one-line summary of what changed and why. This is the field that prevents the "three words changed, production broke" scenario — not because you can prevent the change, but because you can find it immediately when debugging.

The documentation doesn't need to be perfect. A 70% accurate document is infinitely more useful than no document. Write what you know. Mark what you're uncertain about. Leave it better than you found it.

The Organizational Problem Underneath

Individual documentation practices only go so far. The deeper problem is that most organizations don't have a workflow that forces prompt documentation to happen.

Code review doesn't catch prompt changes if prompts live in a database. CI/CD pipelines don't run prompt tests if no tests exist. Postmortems attribute incidents to "prompt change" without capturing what the change was or why the original prompt wasn't understood well enough to predict the failure.

The teams that handle this well treat prompt changes with the same rigor as code changes. They store prompts in version control. They run a test suite before any production prompt changes. They require a written rationale for any change beyond fixing a typo. And they maintain a behavioral test corpus for each production prompt — a set of canonical inputs and expected outputs that any prompt change must pass before deployment.

This sounds heavyweight. For a single prompt, it is. At the scale of a production system with dozens of prompts driving different features, it's the only thing that prevents the archaeology problem from compounding indefinitely.

Start the Audit Now

If you have production prompts running today that you couldn't confidently explain to a new engineer in five minutes, you have undocumented prompts. The time to document them is not when you need to change them — it's now, while behavior is stable and you can use production traffic to validate your understanding.

Pick the highest-stakes prompt in your system. Run the behavioral profiling. Do the probe testing. Write the documentation. Then do the next one.

The archaeology is painful the first time. Done systematically, it also produces the documentation infrastructure that makes every future change faster and every future incident faster to debug. The goal is to be the last team that has to excavate — and to make sure you leave something better behind.