Agent Wakeup Latency Budget for Interactive Products

Wakeup latency has to be set as a limit before the architecture is built. Most teams build the other way around: they pick an LLM, wire up a set of tools, connect a speech layer if the product talks, and only then run the end-to-end number that tells them whether any of it actually works for a real person. By that point the structural decisions that produced the number are the ones hardest to undo. The wakeup event, the moment a dormant agent has to come back to life, rebuild its context, and produce a first response before any LLM token or tool call or streamed word reaches the user, is the start of that chain, and it is usually the one piece nobody measured. Treating it as a fixed budget line, sized before anything else is chosen, changes which decisions get made first and which get made last.

What human tolerance permits, layer by layer

The ceiling on that budget comes from how people actually take turns talking, and it's tighter than most developers assume. An agent that takes 900 ms to respond isn't just slow by a little; it has moved outside the whole range of what any caller has ever experienced from a human being on the other end of a conversation.

Practitioners who build and run conversational agents report four rough bands that match this research without being controlled studies themselves, calibrated observation rather than settled science. Under 500 ms, a response feels like conversation, and under 300 ms, the pause is hard to tell apart from a human one. Past 800 ms, practitioners consistently describe something different happening: the conversational contract breaks, and the user stops treating the exchange as a dialogue. That 800 ms mark is roughly four times the human conversational mean from the Stivers data, which is part of why it functions as a defensible ceiling rather than an arbitrary round number.

Some of that budget disappears before an agent touches a single request. ITU-T G.114 caps one-way transmission delay at 150 ms for what it calls transparent interactivity, and treats 400 ms as the outer planning limit for acceptable voice communication. That delay belongs to the network carrying the call, not to anything the agent does, and it comes out of the 800 ms ceiling before processing starts.

Production numbers show how hard that ceiling is to hit even on tuned systems. Most shipped voice agents are already past the 1.2-second point where callers start to abandon the call, at least at the tail end of their latency distribution, even when the median looks acceptable on a dashboard.

How the latency budget breaks into layers

Once the ceiling is fixed, the next step is dividing it across the layers that actually consume it. The LLM's first token typically takes 100 to 250 ms with models built for low latency, and this is usually the single largest piece of the budget; in a stack stitched together from separate vendors, the model call alone can eat more than half the total time available. Network transport between components adds another 20 to 60 ms, compressed significantly when the stack is co-located.

Vendor-published numbers tend to describe one layer and get read as a description of the whole pipeline. Vapi publishes a target of under 500 ms at the median as a performance standard rather than a measured result across its whole platform; when Cekura ran a fixed set of components through it across more than a thousand turns, the measured number came back at 2.34 seconds. Both figures come from real measurements, describing different things about the same pipeline. They are just not measuring the same thing, and a team that designs around the smaller number without checking what it covers will miss its own ceiling by a wide margin.

Before any of those four layers can run, the agent's compute environment has to exist and be ready to receive a request. If the agent has been asleep, that environment has to come back either from a snapshot or from a cold boot, and this step happens before speech-to-text, before the model call, before anything that shows up in a typical latency dashboard. Most frameworks never account for it because they assume the agent was already running when the clock started.

Planning-heavy, multi-hop agents stretch this budget well past what conversation allows. Products that need both real planning and real conversation have to either narrow what the agent is allowed to do inside a single turn or use signals like typing indicators and partial responses to keep the wait from feeling broken.

Why cold-start and restore times decide product viability

Cold-starting an isolated agent environment and restoring one from a snapshot sit in entirely different latency bands, not different points along one scale. A Firecracker micro-VM boots from nothing in roughly 125 ms with about 5 MB of memory overhead, which is fast for bringing up a full kernel, but that number describes the bare VM before any application state, installed packages, or prior agent context exist inside it. Container-image cold starts at managed scale run far longer: before Amazon Bedrock AgentCore added snapshot-restore, its V1 cold starts ran 5.4 to 30 seconds across container images between 200 MB and 2 GB. An agent that has to re-clone a repository, reinstall packages, or rebuild session state on top of a cold VM boot adds hundreds of milliseconds to several seconds on top of that, which by itself exceeds the entire 800 ms ceiling before any pipeline layer has run.

Snapshot-restore avoids almost all of that cost by never discarding the state in the first place. Firecracker's restore mechanism pauses a running VM, preserves its memory and filesystem exactly as they were, and brings it back in 5 to 30 ms, with installed packages, written files, and session context intact. One tuned open-source implementation combining Firecracker's snapshot restore with pre-provisioned root filesystems measured guest state restore at roughly 2.6 ms, with the full end-to-end cold start, including network namespace setup, a veth pair, a TAP device, and NAT egress provisioning, landing around 90 ms. At managed scale, Amazon Bedrock AgentCore Runtime V2, generally available September 18, 2026, cut cold starts from the V1 range of 5.4 to 30 seconds down to a 75th-percentile figure of 1.9 to 2.0 seconds across the same image sizes. That is a substantial improvement from a major managed platform, and it still consumes most of the available 800 ms before the pipeline has started a single step.

Only snapshot-restore at the Firecracker-level tier, restoring in the 5 to 30 ms range, leaves enough of the 800 ms ceiling for speech-to-text, the model call, speech synthesis, and network transport to all fit inside it. Maritime's approach, restoring each agent from a snapshot of its own micro-VM in under a second, is one way of putting that constraint into the infrastructure layer directly rather than discovering it as a bottleneck after launch.

The idle-cost trap: why keeping agents warm to avoid wakeup latency has its own ceiling

Keeping every agent running at all times eliminates the delay of waking a dormant agent, but it raises costs that balloon once a product has one agent per user instead of one agent shared across many. Most agents, in a per-user model, spend the overwhelming majority of their time doing nothing between turns, and paying for compute during all of that idle time is not a sustainable cost structure once the user count grows past a small pilot.

The trade-off runs in both directions and has to be sized deliberately. Consumption rates in V2 rose across the board, CPU from $0.0895 to $0.1276 per vCPU-hour and memory from $0.00945 to $0.0169 per GB-hour, and the mechanism built to reclaim idle cost only applies to sessions that stay idle for at least 120 seconds following a memory spike. A short customer-support exchange that starts and finishes quickly never reaches that 120-second window, so it never benefits from the reclaim.

Patterns built around suspend-and-resume, where compute drops to zero during genuine waits rather than idling at reduced cost, can cut idle spend substantially, but only on infrastructure that can restore state fast enough on the way back up. The cost and the latency have to be solved together: a fix for one that ignores the other just moves the failure from the budget to the user experience or back again.

Sizing the wakeup budget for your specific product type

The budget looks different depending on what the product is asking the agent to do: treating a voice agent and a coding assistant as the same sizing problem produces the wrong number for both.

Voice and conversational agents operate under the tightest version of this budget. The ceiling is 800 ms voice-to-voice, with abandonment starting around 1.2 seconds. Network and carrier overhead alone consumes 70 to 160 ms before any processing happens, leaving roughly 600 to 700 ms combined for speech-to-text, the model call, and speech synthesis. Streaming isn't optional here: batch transcription alone adds 600 to 1,200 ms before the model call can even start.

Chat and text-based agents get a looser ceiling, roughly 1 to 2 seconds before the interface starts to feel broken, extended somewhat by typing indicators and partial streaming that give the user something to watch while they wait. The wakeup allocation is comfortable under 200 ms and tolerable up to around 500 ms as long as the first streamed token appears promptly once the wait starts. A returning user's agent should pick up with its full session history intact rather than starting over, so persistent disk and memory have to be part of the architecture, not just a fast VM restore on its own.

Code-execution and tool-heavy agents get the most room, but not unlimited room. What matters more for this category is environment state: an agent that has installed packages and written files to disk has to come back to exactly that state on wakeup rather than rebuilding it, which makes persistent disk a requirement for correctness and not just a latency optimization.

If any one layer's realistic minimum is larger than what's left for it, the architecture needs to change before any code gets written.

Infrastructure characteristics that must match the wakeup budget before anything else is chosen

Once the wakeup budget has a number attached to it, infrastructure choices can be judged against that number as a pass-or-fail test.

Restore latency from snapshot, the time between a sleep event and a running VM with its context intact, needs to be sub-30 ms for voice products and sub-200 ms for text-based ones. This is the single number worth asking any infrastructure provider for directly. Isolation has to be real isolation: shared containers don't provide kernel-level separation for code an agent is running on its own, so hardware virtualization like Firecracker running on KVM is the requirement, and it has to hold that isolation without adding to boot or restore time. The wake-to-first-token path needs to be measured as one number end to end, not as two separate figures added together after the fact; a 5 ms restore followed by a 300 ms context-loading step is a 305 ms wakeup cost, not a 5 ms one, and reporting it as the smaller number hides where the budget actually went.

Network physics and environment setup consume part of the 800 ms ceiling before an agent processes anything at all, which most latency research skips past in favor of studying inference alone. An agent built on infrastructure that holds its state persistently and wakes from sleep in under a second keeps considerably more of that ceiling available for speech-to-text, the model call, and speech synthesis, the layers where a caller's actual sense of the conversation gets shaped. That's the difference between a target that feels like a real exchange and one that consistently falls short of it.

Maritime's setup, each agent in its own Firecracker micro-VM on bare metal with a persistent disk, sub-second wake from snapshot, full package installation privileges, and pricing based on storage at rest rather than idle compute, is built to answer each of those questions directly, which is what makes running a fleet of per-user agents economically workable without leaving them running around the clock.

What to measure after launch

A latency budget set before launch is only a hypothesis until production data confirms it's holding, and the numbers that confirm it aren't the ones vendors publish about their own layer.

Wakeup latency needs to be logged as its own measurement, separate from everything downstream of it. Cold starts and snapshot restores should be distinguished in that log rather than merged together; a spike in cold-start events signals a misconfigured warm pool or snapshot schedule, not a slow restore mechanism.

The numbers that matter are p95 and p99, not the median. A product can have an excellent p50 and a p99 that's badly broken, and reporting on the median alone hides that problem until user complaints reveal it.

Without that breakdown, finding which layer blew its allocation turns into guesswork every time the total comes in high.

Alerts belong on p95 wakeup latency specifically, not on the average of the whole end-to-end number: wakeup drifts quietly as snapshot files grow larger, as the VM scheduler gets reconfigured, or as infrastructure load shifts underneath the product. The same discipline that set the budget before the architecture was built, fixing the ceiling from human tolerance and allocating it downward layer by layer, has to carry into production as the discipline of measuring each layer and alerting when one of them starts spending more than its share. That's what keeps a budget line from quietly turning back into a number nobody checks until it's already failed.