For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Learn how Agent Substrate suspends idle Actors to snapshots and resumes them on demand.

An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. Agent Substrate runs each agent inside an Actor, the unit that Agent Substrate suspends and resumes. To free compute resources while an agent waits, Agent Substrate writes the idle Actor to a snapshotSnapshotThe stored state that an Actor suspends to, held in object storage. Resuming restores the Actor from its most recent snapshot, which is what makes suspending idle agents cheap.Learn more and releases the capacity that the Actor held on its WorkerWorkerA pre-started, sandboxed pod that hosts at most one Actor at a time. Actors are multiplexed across a pool of Workers over time.Learn more. The next message to the agent restores the Actor. The SessionSessionA running conversation with one Agent. Unlike the resources it is built from, a Session is not a Kubernetes resource: kagent's gRPC API creates it and its PostgreSQL database tracks it.Learn more stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again.

Actor lifecycle operations

Agent SubstrateAgent SubstrateThe runtime that kagent runs agents on. It multiplexes many sandboxed Actors onto a smaller pool of pre-started Workers, suspending idle ones to snapshots.Learn more provides three lifecycle operations, and each one moves an Actor between states that you can observe on the Actor record.

• Suspend: Writes the Actor’s state to a durable snapshot in snapshot storage, then releases the capacity that it held on its Worker. A running Actor is snapshotted on its Worker. A paused Actor’s node-local snapshot is uploaded instead.
• Pause: Takes a short-term snapshot whose files stay on the node. Pausing pins the Actor to that node, because the following resume is prioritized onto the node that holds the snapshot files.
• Resume: Restores a suspended or paused Actor onto a Worker, from its latest snapshot. The common path restores from a snapshot rather than cold-booting the workload.

The following diagram traces an Actor through those operations, and shows the further path that opens once a snapshot is pinned by a tag.

    flowchart LR
    pool["WorkerPool"] --> worker1["Worker"]
    worker1 -->|hosts| actor["Actor<br>(running)"]
    actor -->|suspend| snapshot["ActorSnapshot<br>(immutable)"]
    snapshot -->|resume| worker2["Any Worker with room<br>in the pool"]
    snapshot -->|pinned by| tag["ActorSnapshotTag<br>(retention pin)"]
    tag -->|seeds| newactor["New Actor"]
  

An Actor reports its position in this cycle through its state, which is one of RESUMING, RUNNING, SUSPENDING, SUSPENDED, PAUSING, PAUSED, CRASHED, or DELETING. Only a suspended Actor can be deleted.

What a snapshot captures

An ActorTemplateActorTemplateThe compiled, immutable definition that the kagent controller produces from an Agent. Every Actor is created from one.Learn more’s snapshot configuration decides how much of an Actor a given snapshot holds. Two scopes exist.

• Full: Captures process memory, the root filesystem changes layered on top of the container image, and any attached durable volumes. A Full snapshot holds everything that is needed to resume the Actor hot, with its in-memory state intact.
• Data: Captures only the contents of attached durable volumes. Process memory and the rest of the root filesystem are discarded, which makes the snapshot much cheaper to write and store.

kagent sets the scope automatically on every ActorTemplate that it generates, and it sets Full so that process memory survives both a pause and a suspend. Each Actor also gets a single durable volume named data. The one snapshot setting that you author is the storage location, which you set on the HarnessHarnessA Kubernetes custom resource defining how an agent is allowed to run: its runtime, workload image, and WorkerPool and snapshot storage. An Agent pairs it with the AgentTemplate it runs.Learn more.

One scope governs both operations. The ActorTemplate’s onCommit setting selects what a suspend uploads to snapshot storage, and a pause captures this same scope on the node.

Durable volumes and resume

A DurableDir volume is the per-Actor application data surface. Its contents are preserved by the Data scope, so they survive a suspend and resume cycle independently of process memory. How many volumes an ActorTemplate can declare depends on its sandbox class. A microvm template can declare several, because they are subdirectories of a single shared filesystem. A gvisor template is limited to one, until gVisorgVisorA user-space kernel that isolates a workload from the host kernel by intercepting its system calls. A default kagent installation compiles ActorTemplates to the gvisor sandbox class.Learn more accepts more than a single durable mount.

Golden and per-Actor snapshots

Two kinds of snapshot serve different purposes, and both appear in a normal installation.

• The golden snapshot is shared by every Actor of one ActorTemplate. Agent Substrate captures it once by booting the workload temporarily when the template is created. Every new Actor of this template starts from this snapshot rather than from a cold start. kagent waits for the golden snapshot before an AgentTemplateAgentTemplateA Kubernetes custom resource defining what an agent does: its model, system prompt, tools, skills, and plugins. It runs only once an Agent pairs it with a Harness.Learn more becomes ready, and the controller reports waiting for the ActorTemplate golden snapshot while the capture runs.
• The last snapshot belongs to one Actor. Agent Substrate writes it on every suspend and restores this same Actor from it on the next resume. It keeps the files and memory that this one Session built up, so the Session continues from where it stopped.

Agent Substrate writes every snapshot to object storage, so Actor state is durable and portable across the cluster. The supported backends are Google Cloud Storage and Amazon Simple Storage Service (S3), and the S3 client also reaches S3-compatible stores that accept a custom endpoint and path-style addressing. For the settings that select and configure each client, see Configure snapshot storage. A HarnessHarnessA Kubernetes custom resource defining how an agent is allowed to run: its runtime, workload image, and WorkerPool and snapshot storage. An Agent pairs it with the AgentTemplate it runs.Learn more names the location for its Actors’ snapshots in its substrate.snapshotPolicy section.

spec:
  substrate:
    workerPoolRef:
      name: kagent-default
    snapshotPolicy:
      # The bucket that the Agent Substrate chart creates in its bundled object store.
      # If your Substrate installation uses your own object storage, use that location instead.
      location: s3://ate-snapshots/kagent/

Suspension between turns

kagent does not wait for an Actor to go idle for a long stretch before suspending it. It suspends the Actor at every turn boundary, as soon as nothing in the Session is running.

A turn reaches such a boundary when its task enters a terminal state, or when the task stops to wait on a person, which is the INPUT_REQUIRED and AUTH_REQUIRED case. At that point kagent suspends the Actor durably and records the exact snapshot that the suspend produced.

The SessionSessionA running conversation with one Agent. Unlike the resources it is built from, a Session is not a Kubernetes resource: kagent's gRPC API creates it and its PostgreSQL database tracks it.Learn more’s own state does not change while this happens. It stays READY throughout, because suspension is a property of the runtime underneath the Session rather than of the Session itself. A caller that lists Sessions sees a ready Session whether or not an Actor is running for it.

Resuming on demand

A suspended Session resumes when the next message arrives for it. The kagent controller hands that message to Agent Substrate, which restores the Actor from its last snapshot onto a Worker with room, and then delivers the message. No caller needs to know whether the Actor was running beforehand, and nothing in kagent asks for the resume explicitly.

Agent Substrate resolves the Actor’s address on every request, so the Actor stays reachable at one name no matter which Worker it is placed on. For more information, see Sandboxing.

Because kagent suspends at every turn boundary, every message to a Session pays for one resume. An idle Session costs no compute, and the next message waits for its Actor to come back.

Checkpoints

A checkpoint marks one point in a conversation and keeps it. You can create a checkpoint when a Session reaches a state worth returning to, such as a long investigation that is only partway through. You can then start a second Session from that point, which is known as a forkForkA second Session created from a checkpoint, continuing the conversation from the point that the checkpoint pinned. A fork inherits the checkpoint's revision, so later edits to the Agent do not change what it runs, and new turns append only to the fork, leaving the original's history untouched.Learn more. The fork continues the conversation from where the checkpoint marked it, new turns append only to the fork, and the fork gets its own conversation and task identifiers rather than inheriting the source’s. To create a checkpoint and fork a Session from it, follow the Agent Substrate example.

Create a checkpoint only after a turn finishes. The request names the task that the turn ended on, and it fails when a newer turn has already started or while that turn’s snapshot is still being written.

Underneath, a checkpoint pins a snapshot that would otherwise be discarded. Agent Substrate treats the snapshot written on each suspend as transient and collects it once a newer one supersedes it. Creating a checkpoint attaches a TagTagAn Agent Substrate resource that gives one snapshot a stable, human-meaningful name, so callers do not need to track Substrate's internal snapshot identity. It also acts as a retention pin: Agent Substrate does not collect a snapshot while a tag names it, and only the tag's visibility scope can change afterward.Learn more to the snapshot that the Session most recently suspended to, and Agent Substrate keeps any snapshot that a tag still names. Deleting the checkpoint removes the tag and releases the snapshot.