Skip to main content

Your Agent's Memory Needs Two Clocks

· 11 min read
Tian Pan
Software Engineer

Somewhere in your agent's memory store sits a fact like "the billing API returns XML." It reads as timeless truth. But it is actually two claims welded together: the API returned XML during some window of the past, and your agent observed this at some moment — possibly a different window, possibly long after the migration to JSON. The memory record keeps neither timestamp. When the agent acts on that fact and breaks something, you will ask the only question that matters in a post-incident review: what did the agent believe at the moment it acted? And your memory layer, having collapsed both timelines into one flat string, cannot answer.

Database engineers solved this problem decades ago and gave it an unglamorous name: bitemporal modeling. Every fact carries two independent clocks — when it was true in the world (valid time) and when the system learned it (transaction time). Financial systems, insurance ledgers, and audit-grade databases have run on this distinction for years. Agent memory systems, almost universally, ignore it. That omission is now the root cause of a whole family of failures we keep misdiagnosing as "hallucination."

One Fact, Two Missing Timestamps

The standard agent memory pipeline extracts salient facts from conversations and tool outputs, embeds them, and stores them as propositions: "user prefers Python," "the staging cluster has 12 nodes," "the customer's plan renews monthly." Each proposition is stored the way a fortune cookie states wisdom — floating outside time.

But every observation an agent makes has two temporal coordinates, and they routinely diverge:

  • Valid time: the interval during which the fact was true in the world. The staging cluster had 12 nodes from the March scale-up until the June consolidation.
  • Transaction time: the moment the fact entered the agent's memory. The agent learned about the 12 nodes in April, from a tool call that happened to enumerate them.

When your memory layer stores only the proposition, it silently asserts that both clocks read "forever" and "now." That is how an agent ends up confidently describing a cluster topology that was decommissioned three weeks ago — not because the model hallucinated, but because the memory system faithfully served a fact whose validity had quietly expired. The retrieval was correct. The fact was stale. Without a valid-time interval, those two situations are indistinguishable.

The divergence gets worse with second-hand knowledge. An agent that reads a runbook learns facts whose valid time started long before its transaction time — and may have ended before it too. An agent told "we're migrating the API next quarter" learns a fact whose valid time hasn't started yet. Flat memory stores flatten all of this into an eternal present.

The Update That Destroys the Evidence

Most memory frameworks do have a story for change. Mem0, one of the most widely adopted memory layers, runs an LLM-driven reconciliation step: new facts are compared against existing neighbors, and the system decides per pair whether to ADD, UPDATE, or DELETE. When new information contradicts an old memory, the old memory is rewritten or removed.

This feels like hygiene. It is actually evidence destruction.

Consider the incident review again. Your agent issued a refund it shouldn't have, because it believed the customer was on the annual plan. Two weeks before the incident, the customer had downgraded to monthly. Somewhere along the way, the memory system UPDATEd the plan fact. Now the questions cascade:

  • Did the agent act before or after the memory was corrected?
  • If after — why did retrieval surface the old belief anyway?
  • If before — was the correction available upstream and simply not ingested yet?

An overwrite-based store cannot distinguish these cases, because the act of correcting the memory destroyed the record that the agent once believed otherwise. You know what the memory says now. You cannot reconstruct what it said then. The audit trail for an autonomous system's beliefs — the thing regulators, security teams, and your own postmortem process increasingly demand — was deleted by the very mechanism meant to keep memory accurate.

Bitemporal systems refuse this trade. In XTDB, transaction time is immutable by construction: you can correct the past, but the correction is itself a new transaction, and the pre-correction view remains queryable forever. Zep's Graphiti engine brought the same discipline to agent memory: every edge in its knowledge graph carries explicit validity intervals, and when contradictory information arrives, the superseded fact is invalidated — stamped with an end time — rather than deleted. You can query what's true now, or what was believed true at any earlier moment. Nothing is lost; things merely stop being current.

The design rule falls out cleanly: superseding a fact must never overwrite the record that the agent once believed it. Corrections append; they do not erase.

The As-Of Query Is Your Incident Report

Once both clocks exist, a query shape becomes available that flat memory cannot express: the as-of query. Show me the facts about the customer's plan, as they were valid on June 10th, as known to the agent on June 10th at 14:32 — the moment it approved the refund.

Bitemporal databases treat this as a routine operation. XTDB's canonical demo is a criminal investigation: border-crossing records arrive late and get corrected by humans, and investigators need to ask "who did we believe was in the country on day 2, given only what we knew by day 3?" — deliberately excluding corrections that arrived on day 4. The two time axes let you separate the history of the world from the history of your knowledge of it.

Substitute "agent" for "investigator" and this is precisely the post-incident workflow. Execution traces tell you what the agent did — the tool calls, the outputs, the final action. But traces alone don't tell you what the agent knew, because retrieval results depend on the memory state at query time, and that state has since moved on.

Replaying the decision against today's memory is forensically worthless; the memories have been consolidated, corrected, and re-embedded since. What you need is the memory store's state as-of the decision's transaction time — and only an append-only, two-clock store can give it to you.

This reframes memory design as an observability problem. Teams instrument agents with tracing for prompts, tool invocations, and reasoning chains, then attach those traces to a memory layer that mutates in place — an immaculate flight recorder wired to a cockpit whose instruments are repainted after every flight. The trace says the agent read memory #4821. It cannot say what memory #4821 contained at that instant. Bitemporal memory closes that gap with one schema decision instead of a bolted-on snapshotting subsystem.

Staleness Becomes a Query, Not a Vibe

There's a second payoff, and it addresses the failure mode that current benchmarks say models handle worst: knowing when a memory has gone bad.

The STALE benchmark tests exactly this — whether an agent can recognize that a previously valid memory has been invalidated by later observations, especially when the contradiction is implicit (a user who mentions a leg injury has implicitly invalidated the "commutes by bicycle" memory, though no one said so). The results are sobering: the strongest frontier model evaluated reached only about 55% overall accuracy, and performance splits sharply between recognizing staleness when asked directly and applying that recognition to downstream behavior — one open model scored 76% on direct state resolution but 39% on adapting its actual behavior. Worst of all is premise resistance: when a query presupposes the outdated fact, models overwhelmingly go along with it.

The instinctive fix is to make the model smarter about staleness — better prompts, reflection loops, an LLM judge that periodically sweeps memory for contradictions. That's fighting a structural problem with inference. If facts carry valid-time intervals, a large share of staleness stops being a reasoning problem and becomes a filter:

  • A fact whose valid-time interval is closed is historical by definition. It can inform context ("the user used to commute by bike") but should never be served as current state.
  • A fact with an open interval and an old transaction time is suspect in proportion to the volatility of what it describes. "User's birthday" observed two years ago is fine. "Staging cluster has 12 nodes" observed two years ago is almost certainly false, and the memory layer can say so — or trigger re-verification — before the fact ever reaches the context window.
  • A newly ingested fact that contradicts an open-interval fact doesn't need an LLM debate about which to keep. It closes the old interval, records the supersession link, and both survive.

The research frontier is converging on the same conclusion from the model side: the mitigation proposed alongside STALE performs belief adjudication at write time — evaluating and revising older memories when new evidence arrives, then constraining retrieval to adjudicated facts — and lifts accuracy from single digits to 68% on the same backbone. Write-side temporal bookkeeping beats read-side cleverness. That is exactly the bet bitemporal databases made forty years ago.

Building the Two-Clock Memory Layer

You don't need to adopt a temporal database wholesale to get most of the value. The essential moves fit into whatever store you already run:

  • Four timestamps per fact, minimum: valid_from, valid_to (nullable, meaning "still true as far as we know"), recorded_at (immutable, set at ingestion), and superseded_by (a link to the fact that closed this one). This is the classic bitemporal column set with lineage.
  • Append-only writes. Correction, contradiction, and decay all express as closing intervals and inserting successors, never as UPDATE or DELETE on the fact row. Storage is cheap; a destroyed belief history is unrecoverable.
  • Default retrieval is "now as-of now." The common path serves currently valid facts given current knowledge — same latency profile as before, just with an interval predicate. The historical axes exist for audits, evals, and debugging, not for every query.
  • Expose the clocks to the model. When a retrieved fact enters the context window, include when it was observed and its validity status ("observed 14 months ago, unverified since"). Premise-resistance failures soften considerably when the context itself flags the fact's age instead of presenting it as gospel.
  • Support retroactive corrections. Agents learn about the past constantly — a user mentions they changed jobs in January. That's a valid-time edit months in the past, recorded at today's transaction time. Event-sourcing practitioners have handled exactly this shape (late-arriving and corrective events) for years; borrow their discipline rather than reinventing it.
  • Replay in evals. Once as-of queries exist, your eval harness can re-run past decisions against the exact memory state the agent had — turning "the agent did something weird last Tuesday" from an anecdote into a reproducible test case.

The cost is real but bounded: more rows, an interval predicate on the hot path, and ingestion logic that adjudicates conflicts instead of blindly appending. Teams pay far more than that for tracing infrastructure that records everything except what the agent believed.

Memory You Can Cross-Examine

The industry is currently obsessed with making agent memory bigger — longer horizons, more sessions, richer consolidation. The harder and more valuable property is making memory accountable: able to state not just what the agent knows, but since when, on what observation, and what it believed before. Autonomy without that property is unauditable by construction; no amount of trace logging around the memory layer compensates for a memory layer that repaints its own history.

Bitemporality is not an exotic research direction. It's a well-worn pattern with production-grade implementations, already proven in the one agent-memory system that made it a first-class citizen, and it directly targets the staleness failures that benchmarks show models cannot solve on their own. Two clocks per fact. Append, never overwrite. Make "what did the agent believe when it acted?" a query — because the first time a serious incident forces you to ask it, the answer will either be one WHERE clause away, or gone forever.

References:Let's stay in touch and Follow me for more thoughts and updates