Snapshot and Restore Mechanics in Firecracker

Firecracker micro-VM isolation gives each of the AI agents its own kernel and machine state that persists, without using containers. Snapshot and restore make that model practical by reviving a paused machine. Lazy page-in, copy-on-write memory and diff snapshots keep revival cheap, but reliable deployment also requires correcting the time and randomness inherited from the snapshot.

What a Firecracker snapshot actually contains and why restore is not a boot

The restore sequence is to launch a fresh Firecracker process, map vm.mem, load vm.state, and unpause the vCPUs. The memory file holds guest RAM, while the state file supplies the CPU and device state that interprets it. Both files are needed because registers point into RAM, and RAM needs the execution state that gives those addresses meaning. A restorable snapshot consists of that pair plus the rootfs.

Consequently, the guest resumes mid-instruction, with processes still running, files still open and its page cache still warm. In a cold boot, Firecracker must start before the kernel can be loaded and booted, init can run and the guest agent can start. Restore still needs a fresh Firecracker process, but skips those guest startup stages, including the systemd handshake. Maritime's persistent agent platform uses this restore-not-boot architecture to take a snapshot of each agent after initialization and restore its paused state when it wakes. Its under-a-second resume path brings an existing machine back to life.

Why MAP_PRIVATE makes restore O(1) in RAM image size

The crucial distinction is between mapping a memory image and reading it. With MAP_PRIVATE, “load a 2 GiB snapshot” does not mean reading 2 GiB from disk before execution can begin. The kernel establishes virtual-memory bookkeeping for a file-backed mapping without eagerly reading its contents. Mapping a 2 GiB file and a 200 GiB file therefore costs about the same at this stage. The O(1) claim is limited to this mapping operation relative to image size.

Once the guest resumes, demand paging supplies the memory it actually touches. Access to a missing page causes the CPU's MMU to raise a page fault, which the kernel resolves before the instruction retries transparently. A minor fault finds the page already in the host page cache and connects it to this mapping without disk I/O. A major fault requires a disk read and puts that cost on the first access. Page faults are not free, but their cost is deferred rather than charged against the entire image before execution.

A guest with 2 GiB of RAM that touches only a few hundred megabytes to become ready pays to read that working set. Untouched parts of vm.mem need never be read. This distinction underpins per-agent pricing models, including Maritime's, because a large configured memory image does not require an equally large read at every wake. It does not make active memory consumption independent of concurrency.

How copy-on-write memory semantics let many VMs share one snapshot

On a read fault, the kernel can map a 4 KiB page directly from the file through the host page cache without copying it. Other guests restored from the same snapshot can share that identical page. On a write fault, the kernel creates a private 4 KiB copy for the guest making the change. The backing file stays pristine, so one guest's memory writes do not alter the baseline used by another.

This copy-on-write behavior turns a latency optimization into a fleet primitive. With 50 running VMs restored from the same snapshot, pages can stay shared without making 50 separate copies, and each VM gets its own copies of the pages it writes. The rootfs can follow the same principle through XFS reflink cloning of its ext4-formatted image file. Disk blocks remain shared with the template until the guest writes them. Memory sharing and disk sharing are separate mechanisms that together avoid copying an entire sandbox for each instance.

The golden-image pattern that follows is to cold-boot once, snapshot once and restore many times. Relative paths and per-sandbox working directories let concurrent restores use the same snapshot without path conflicts. A sandbox fork() illustrates the same principle by snapshotting a particular running sandbox and restoring a child with the same MAP_PRIVATE mapping. The child inherits the parent's RAM and disk state but allocates private memory only for pages it changes.

Full snapshots vs. diff snapshots and when each belongs in a workflow

A full snapshot and a diff snapshot serve different purposes. A full snapshot writes the whole guest RAM image and gives it a memory baseline that can be restored on its own. A diff writes only pages dirtied since the base snapshot, using KVM's dirty-page log to identify them. That smaller artifact is not independently restorable because unchanged pages still belong to its base. To restore the full memory state, the diff must be layered onto that base, along with the matching machine and disk state.

The workflow follows from that dependency. Use a full snapshot to bake a golden image for many restores, and use diffs for incremental checkpoints within a long-running agent session. A sequence of diffs over a full base supports rollback without copying all guest RAM at every checkpoint. Mixing up those roles can lead to copying more than needed or to a checkpoint that lacks the state needed to restore it.

DeltaBox demonstrates why incremental state capture fits agent execution, where consecutive checkpoints differ only slightly. Its local checkpoint work can hide under inference, with template-fork rollback in roughly 2 ms. On the disk side, Tensorlake implements copy-on-write image layering inside Firecracker's block device. Its one-file-change example demonstrates the same changed-bytes principle for disk snapshots that KVM's dirty-page mechanism applies to RAM.

Streaming pages from object storage with userfaultfd for cross-host restores

Local file mappings need the memory image to be on the destination host. Cross-host restore preserves the lazy page-in principle through a different kernel path, using userfaultfd to send unresolved page faults to a userspace handler. Firecracker's snapshot load API exposes the backend choice through mem_backend.backend_type. The "File" backend maps a local memory file, while the alternative delegates page delivery to the handler.

When the guest touches a missing page, the handler fetches a surrounding chunk from object storage using an HTTP Range GET. It then installs the data with UFFDIO_COPY, allowing the guest to continue. The fetch is still a real cost, and the faulting access depends on its completion. As with local demand paging, that cost depends on which pages the guest touches, without a need to read all of its configured RAM.

Snapshots can live in object storage, and a destination host can restore one while fetching memory on demand. A guest can consequently reach its ready state before most of its RAM image crosses the network. For fleet design, snapshot placement no longer has to bind an agent permanently to the host that captured it.

The clock drift problem: why a restored VM wakes up in the wrong time

HTTPS requests can fail right after restore because the guest thinks a server certificate's notBefore is in the future. The symptom resembles a network failure, but the cause is the restored clock. If a template was baked at 09:00 Monday and restored at 09:00 Wednesday, the guest can wake believing it is Monday 09:00. Its clock then advances from the wrong starting point. This is a step error that persists as the machine runs.

The error spans the guest's timekeeping layers. The TSC, or Time Stamp Counter, supplies the per-CPU tick count used for fast timekeeping. The kvmclock shared memory page publishes a reference TSC value and the scale and shift needed to convert ticks into nanoseconds. That page is captured in vm.mem, but kvmclock is a coherency mechanism rather than a service that synchronizes wall-clock time. Wall-clock time derives from a base value and elapsed time, so restoring frozen timekeeping state can restore the wrong “now.”

That wrong starting point affects more than certificates. JWTs and OAuth tokens can fail validity checks, or expired tokens can seem valid, because the checks use a date in the past. Scheduled tasks can fire immediately when their deadlines were close to the captured moment. Uptime accounting also needs attention because /proc/uptime resets to seconds since restore.

The fix is to force an NTP sync or push the correct wall-clock time into the guest immediately after restore, before workload code runs. VMGenID exposes a generation identifier that lets guest software detect restoration and trigger clock correction. Clock repair belongs on the resume path because the step error is not self-healing.

Entropy reuse: the randomness footgun that restored VMs share by default

Restoring identical state also restores anything that was assumed to be unique. A process that seeded its PRNG before snapshotting leaves that seed in vm.mem. Every clone then inherits the same state and can produce the same sequence of apparently random values. The scope includes OpenSSL's internal state, session tokens, startup-generated UUIDs and cryptographic nonces. AWS explicitly warns about the equivalent initialization-state problem in Lambda SnapStart, where captured state is shared across restored environments.

The corrective work is to identify startup PRNG and key-generation calls and make them re-seed on resume as well as on first boot. Entropy sources must be re-seeded before workload code that needs randomness runs, and unique state must be reinitialized. VMGenID gives guest software a signal it can use to do that work on resume. Credentials or nonces made inside the snapshot window must be treated as compromised when more than one clone will use them. Sandbox fork() sharpens the same risk in tree-search or RL rollback because each branch inherits the parent's entropy state unless it re-seeds on creation.

What speculative sandbox prewarming changes about restore latency in MCP workloads

MCP-based agents invoke external sandboxes, creating a choice between retaining every sandbox in memory and restoring each only when requested. Always-warm execution avoids wake latency but wastes memory. Always-cold execution saves idle memory but makes the request wait. Speculative prewarming attempts to restore the needed sandbox before the tool call arrives.

SpecBox, published in July 2026, uses keyword matching and streaming semantic embedding to identify pending tool demands during model generation. It overlaps sandbox bootstrapping with inference and uses context-aware stochastic prefetching over a sandbox dependency graph to forecast later switches. SpecBox reports lower peak memory use than always-warm execution without reintroducing cold-start latency.

How do you give each AI agent its own isolated kernel in a per-user fleet?

A per-user fleet assigns each user a persistent VM with its own kernel and injects identity at runtime. The shared base image has no particular user's identity in it. Each VM receives its own dedicated endpoint, with tokens scoped so a stolen credential for one tenant's endpoint is useless against another's. The VM boundary supplies structural isolation for untrusted agent code, while copy-on-write sharing reduces the cost of supplying separate machines.

An idle agent can be parked as a snapshot, costing storage rather than a running allocation of RAM and vCPU time. Host capacity then follows active concurrency. Files, installed packages, browser sessions, credentials and in-memory state are still there after the snapshot-restore cycle. Under-a-second wake latency lets that state be kept for interactive use, as long as the pages arrive and the resume-time corrections finish within the wake path. Cheap mapping alone is not the whole readiness contract.

The remaining choices concern where snapshots live, how checkpoints retain their bases, and when the scheduler restores an agent. Local mappings favor available files, userfaultfd permits remote demand paging, and speculative prewarming moves likely restores ahead of requests. For teams that choose a managed layer over running their own fleet, Maritime uses this pattern with persistent Firecracker micro-VMs on bare metal, each with its own disk and kernel. Its architecture also fits the shared-baseline copy-on-write model that makes concurrent agent instances economical. The decisive design is to preserve an isolated machine while idle, then restore only the work and memory needed to make it useful again.