The Pointer Your Agent Mistook for a Value: Reference vs Value in Tool Outputs

A search tool returns ten document IDs. An asset tool returns an S3 presigned URL. A database tool returns a row handle. A file tool returns a path. Each of those returns is, formally, a pointer — a small string that names a value the agent does not yet possess. The model's downstream behavior depends entirely on whether it knows that and dereferences before reasoning, or whether it treats the pointer as if it were already the thing.

The failure mode is invisible from the trace. The tool call succeeded. The return is well-formed. The model emitted plausible-looking output. Nothing in the log says "the agent reasoned about a filename and called it a document." The pointer-vs-value confusion sits underneath the visible behavior, in a layer your tool schema never named.

This is not a model defect. The model has no native concept of indirection. It has tokens, and it produces more tokens conditioned on the ones in front of it. If the token in front of it is s3://bucket/report-q3.pdf, the model has a choice — call a fetch tool to get the contents, or compose on top of the string as if "report-q3.pdf" were itself an adequate summary of the report. Both are valid continuations of the prompt. Which one wins depends on the prompt's framing, the recent context, the model's training, and whatever the temperature roulette spits out this round. Your tool catalog never told it which to pick.

The Indirection Layer Your Tool Registry Never Named

Most tool registries describe inputs in detail and outputs in passing. The input schema is the contract the agent has to satisfy to call the tool. The output schema, when it exists at all, tends to be a flat description: "returns a list of document IDs," "returns a URL," "returns a row." That description is enough for a human reading the docs to know what to do with the return — go fetch the document by ID, GET the URL, query the row. It is not enough for a model.

The missing dimension is whether the return is meant to be reasoned about or resolved. A document ID is not a document. A URL is not the page. A row handle is not the row. Every one of those returns assumes a subsequent dereference step the tool author had in mind and the model was supposed to infer. Most of the time the model does infer it. Some of the time, predictably, it doesn't.

You can see the shape of the problem most clearly in mixed catalogs. A team adds a search_docs tool that returns IDs. Then later a summarize_doc tool that takes an ID and returns text. Then later a get_doc_metadata tool that takes an ID and returns a small JSON blob. The agent now has three tools that traffic in document IDs, and the relationship between "I have an ID" and "I have content" is encoded nowhere — it lives entirely in the agent's prompt-shaped intuition about which tool to call next. When the intuition is right, the agent looks competent. When it isn't, the agent summarizes documents by their filenames.

The Failure Modes You Will See in Production

The catalog of pointer-vs-value failures is small and recurring once you know to look for it:

• The agent answers a question about a document by reasoning over its title or filename, because the title was the only string in context.
• The agent compares two database rows by their primary keys, declaring them "different records" without ever fetching either row's fields.
• The agent describes an image by parsing the URL — "looks like a Q3 dashboard chart" because the path contained q3-dashboard — without ever calling a vision tool on the bytes.
• The agent treats a presigned download link as if the link itself were the asset, generating output as though it had read the file.
• The agent answers questions about a long PDF after only seeing a metadata blob (filename, page count, MIME type) that the previous tool returned.
• The agent cites a reference by its identifier — "doc_4f2a1" — in user-facing output, because the ID was the strongest string in context and the model promoted it to evidence.

Each of these traces looks fine in your observability dashboard. The tool was called. The tool returned a 200. The model produced an answer. The bug lives in the gap between "the model has a string that names content" and "the model has the content," and nothing in the trace marks that gap as an error.

The deeper failure mode is that the model's behavior here is prompt-shaped. The same tool, returning the same ID, in two different prompts will get dereferenced correctly in one and composed-as-if-resolved in the other. You cannot regression-test this with a single prompt against the tool. The bug is in the joint distribution of prompt, history, and tool surface, and it migrates as the prompt evolves.

Type the Indirection Explicitly

The fix is the obvious computer-science one: name the indirection in the type system the model sees. If your tool returns a pointer, mark it as a pointer in the schema. The agent's planner can then reason about the dereference as a step that needs to happen, instead of a step that may or may not happen depending on whether this prompt context made it salient.

Concretely, the patterns that work:

Loading…