Designing an Agent Runtime from First Principles

Most agent frameworks make a critical mistake early: they treat the agent as a function. You call it, it loops, it returns. That mental model works for demos. It falls apart the moment a real-world task runs for 45 minutes, hits a rate limit at step 23, and you have nothing to resume from.

A production agent runtime is not a function runner. It is an execution substrate — something closer to a process scheduler or a distributed workflow engine than a Python function. Getting this distinction right from the beginning determines whether your agent system handles failures gracefully or requires a human to hit retry.

The Three Problems That Break Simple Loops

Before building anything, you need to understand exactly why a naive while agent_not_done: step() loop fails in production. There are three compounding problems.

Latency. LLMs are slow. A multi-step task with ten tool calls might spend four to eight seconds per LLM invocation. If any of those calls can be done in parallel, blocking on each sequentially is leaving seconds of wall-clock time on the table. Real users notice two seconds of unnecessary waiting. The loop has no concept of parallelism — it serializes everything by default.

Reliability. Long-running tasks fail. Networks drop. APIs rate-limit. Services go down. The longer your agent runs, the higher the probability that something interrupts it. A bare loop offers no recovery path. If it throws on step 18 of 25, you start over from step 1. Every LLM call is paid twice. More importantly, any side effects already taken — files written, emails sent, records modified — may execute twice.

Non-determinism. LLMs produce different outputs across runs, even at temperature 0 (practically speaking). You cannot replay history reliably. What you can do is checkpoint state so that recovery resumes from a saved snapshot rather than re-running previous steps. But the loop has no checkpointing model, so replaying is your only option.

These three problems compound. A task that fails under load, requires restart, replays non-deterministically, and does so at high latency is not a production system. It is a prototype wearing a suit.

Why Graphs, Not Chains

The first architectural decision in a serious agent runtime is how to represent the execution structure. The naive answer is a sequential list of steps. The better answer is a directed graph.

A graph lets you express two things that a list cannot: parallelism and cycles.

Parallelism matters because independent subtasks — fetching data from two APIs, calling two tools that don't depend on each other — should execute concurrently. In a list, you serialize them. In a graph, you wire them as sibling nodes, and the runtime executes them simultaneously.

Cycles matter because agents are not pipelines. They loop. An agent decides whether to continue or terminate based on its current state, not a predetermined step count. A list cannot represent this without hacks like index-jumping. A graph represents it naturally: a node can have an edge back to a prior node.

This is why graph-based execution engines — originally developed for distributed data processing — map surprisingly well to agent workflows. The same properties that made them useful for processing billions of edges (parallelism, fault tolerance, iterative convergence) are exactly what agent runtimes need.

The Execution Model: Super-Steps and Deterministic Concurrency

Once you commit to a graph representation, you need an execution model that handles parallel nodes correctly. The naive approach — run everything in parallel with shared mutable state — creates race conditions. Two nodes updating the same field simultaneously produce non-deterministic outcomes depending on which write wins.

The solution comes from a decades-old algorithm for large-scale parallel graph computation. The core idea is to structure execution as discrete, synchronized rounds called super-steps.

Here's how it works:

1. Plan: Identify which nodes are eligible to execute. Eligibility is determined by subscriptions — a node runs when the channels it reads have been updated.
2. Execute: Run all eligible nodes in parallel. Critically, each node gets an isolated copy of the state at the start of the super-step. No node can see another node's writes during execution.
3. Reconcile: Collect all updates from the completed nodes and apply them to the shared state in a deterministic order. No write timing races — the reconciliation happens after all execution completes.
4. Checkpoint: Serialize the resulting state. This snapshot is now the recovery point.

The isolation in step 2 is the key insight. By giving each node a private copy of state, you eliminate inter-node data races entirely. Two nodes that both write to the same field will have their writes merged deterministically in step 3 — the runtime defines the merge semantics, not execution timing.

This model gives you correct parallel execution without locks. It also gives you a clean point for checkpointing: after each super-step, the state is fully settled and serializable.

Checkpointing Is Not Optional

Many teams treat checkpointing as a nice-to-have. They discover it is load-bearing when their first 30-minute agent task fails at minute 28.

Good checkpointing in an agent runtime means serializing complete execution state after each super-step. "Complete" includes not just the agent's working memory but the full graph state: which nodes have run, what outputs they produced, what channels contain. A checkpoint is a snapshot sufficient to resume execution from that exact point.

The performance implication of this design matters. Loading a checkpoint should be O(1) with respect to history length — you load only the latest snapshot, not replay all prior events. History queries are a separate concern; they scan the snapshot log but should not affect hot-path execution latency.

This bounded recovery cost is what makes checkpointing practical. If resumption required replaying all prior steps, long-running tasks would face a growing recovery time that eventually made resumption slower than restarting. A snapshot-based approach keeps recovery time constant regardless of how many steps the agent has already completed.

In practice, the difference looks like this: a 200-step task that fails on step 199 recovers in the same time as a 5-step task that fails on step 4. The step count does not compound into recovery cost.

Human-in-the-Loop as a First-Class Primitive

Most agent frameworks bolt on human review as an afterthought — a flag that tells the loop to print a message and wait for stdin. This breaks in any real deployment. You cannot keep a process alive indefinitely waiting for a human who might respond in five minutes or five hours.

The correct model treats human interruption as an expected event, not an exceptional one. When an agent needs human input, it checkpoints state, signals the pause, and terminates its process. Nothing is kept alive. When the human responds, the runtime resumes from the checkpoint — the agent continues as if it had never stopped.

This separation of concerns is important. The agent process does not manage human latency. The agent does not need to know whether the human responded in 10 seconds or 10 hours. The checkpoint guarantees that context is preserved across the gap. The resumption is indistinguishable from recovering from a server crash.

This model also enables meaningful oversight patterns. Rather than a binary "approve or reject," you can model partial state edits — a human reviews the current state, corrects specific fields, and the agent resumes with the corrected values. Or escalation — an agent pauses, a human escalates to a different workflow, the original agent resumes with additional context injected.

Designing these interaction models requires thinking about the state schema before you build the approval flow. The checkpoint is the interface. Whatever a human reviewer needs to see, understand, and modify must be represented in the checkpointed state.

Streaming and Observability from the Start

The other failure mode in agent systems is opacity. The agent starts, runs for 90 seconds, and either produces output or errors. You have no idea what it was doing in between.

An agent runtime built on super-steps has natural observability hooks. Each step boundary is an event: what nodes ran, what state changes they produced, what the resulting state looks like. These events are the substrate for streaming output and distributed tracing.

Streaming output is not just a UX feature. In production systems, streaming is how you build reliable confidence that long-running tasks are making progress. An agent that streams partial results as it goes allows downstream systems — dashboards, monitoring tools, human operators — to detect stalls early rather than after a timeout fires.

The step boundary also maps cleanly to trace spans. Each node execution is a span with a defined start, end, inputs, and outputs. LLM calls within nodes are child spans. Tool calls are child spans. The graph structure maps to the trace hierarchy. This gives you distributed-tracing-style debuggability for non-deterministic agent behavior, which is otherwise nearly impossible to diagnose.

Tacking on observability after the fact is painful. A runtime designed around step boundaries makes it structural.

What to Not Bake In

One of the less obvious design choices in a production runtime is what to intentionally leave out.

Specific LLM providers, tool schemas, memory backends — these change. Provider APIs evolve. New models arrive. Memory architectures vary by use case. A runtime that bakes in assumptions about any of these becomes outdated rapidly.

The runtime's job is execution semantics: how nodes are scheduled, how state flows between them, how checkpoints are taken, how failures are handled. Everything above that layer — the specific agents, the tools they call, the LLMs they invoke — belongs to the user's code, not the runtime.

This separation keeps the runtime stable as the rest of the stack evolves. It also makes the runtime testable in isolation. You can write deterministic unit tests for scheduling logic, checkpoint/resume behavior, and state reconciliation without mocking LLM calls.

The principle extends to APIs. If the execution engine is decoupled from the developer-facing graph construction API, you can iterate on both independently. Deprecating a graph-building convenience method does not require changing the underlying scheduler. Optimizing the scheduler does not break existing graph definitions.

Building for Scale That Isn't There Yet

The argument against building a proper runtime early is that the overhead isn't justified until you have real scale. This is usually wrong.

The properties that make a runtime scalable — O(1) checkpoint loading, isolated node execution, deterministic concurrency — are also what make it debuggable and reliable at small scale. You get the debugging benefits immediately. The scaling benefits arrive for free when load increases.

The cost of not building these properties in is that you retrofit them later, under pressure, in production. Checkpoint semantics retrofitted into a running system require careful migration of all existing state. Deterministic concurrency retrofitted into code that assumed sequential execution requires careful auditing for previously hidden race conditions.

These are hard problems to fix under load. They are straightforward to get right from the start, when the graph structure is still flexible and the state schema is not yet frozen by production data.

Agent systems are still early enough that the runtime choices made now will shape what is and isn't possible two years from now. A production-grade execution model is not premature optimization — it is the foundation that makes everything above it reliable.