High Availability (HA) is the cornerstone of an efficient, scalable, performant system. Without it, zero downtime and migrating/scaling state for systems would be impossible. In this case, the "system" is Agents, and the ability to accomplish this is with sandboxing.
In this blog post, you'll learn how to accomplish this for your Agents with Agent Substrate.
Prerequisites
To follow along with this blog post from a hands-on perspective, you will need:
- A GKE or kind cluster.
- Substrate installed (you can learn how to do that here).
If you don't have a cluster running, you can still follow along from a theoretical perspective and implement once a cluster is at your fingertips.
How The Actor Teleports
In Agent Substrate, you'll notice that unless an Actor is being used, it's in a SUSPENDED state. This is a mechanism built into Substrate on purpose to enable efficiency for Agents. The goal is for Agents to be "on" only when necessary.
- Create two actorspaces.
kubectl ate create atespace tenant-a
kubectl ate create atespace tenant-b
- Create two actors, each with the same ID and using the same template, and place them in their respective actorspaces.
kubectl ate create actor agent-1 --template=ate-demo-counter/counter --atespace=tenant-a
kubectl ate create actor agent-1 --template=ate-demo-counter/counter --atespace=tenant-b
- Ensure that both Actors are deployed.
kubectl ate get actors -A
- With both Actors deployed, drive traffic to them to ensure they're operational.
kubectl -n ate-system port-forward svc/atenet-router 8000:80 &
# Hit tenant-a's agent-1 three times
for i in 1 2 3; do
curl -s -H "Host: agent-1.tenant-a.actors.resources.substrate.ate.dev" http://localhost:8000/
done
# Hit tenant-b's agent-1 once
curl -s -H "Host: agent-1.tenant-b.actors.resources.substrate.ate.dev" http://localhost:8000/
You'll see an output similar to the below:
hello from: <sandbox-ip> | preserved memory count: 3 | preserved file counter: 3
hello from: <sandbox-ip> | preserved memory count: 1 | preserved file counter: 1
This proves out the tenancy piece. You gave the Actors the same name (which means the same Actor ID), the same template, and sameWorkerPool, but tenant-a's count is 3 and tenant-b's is 1. Neither tenant can address or observe the other's actions.
Suspend and Move Actor
In this section, you will do two things:
Suspend the actor: the two tenants each run an actor with the same ID (agent-1), and Substrate keeps its state, addressing, and snapshots completely separate.
State teleport: once the Actor is suspended (checkpointed to object storage, Worker released) and resumed onto a different Worker, its state continues exactly where it left off.
This proves that the Pod is cattle and the actor's state is the stable thing.
- Get the Actors current state so you can prove that the IP changes when you move the Actor. With the command below, you'll get the IP.
kubectl ate get actors --atespace=tenant-a
curl -s -H "Host: agent-1.tenant-a.actors.resources.substrate.ate.dev" http://localhost:8000/
- Suspend the Actor.
kubectl ate suspend actor agent-1 --atespace=tenant-a
- The counter demo's
WorkerPool has 5 replicas; parking a few filler actors in the pool takes the old slot. Use 3 fillers, not more. Tenant-b'sagent-1may still be running (idle-suspend takes about a minute), and 3 fillers + tenant-b. The slot you're saving for the wake-up accounts for all 5 workers. The placement picks any free worker, which makes a different worker likely, but not guaranteed. If the Agent lands on the same Worker, suspend it and repeat with another filler.
for i in 1 2 3; do
kubectl ate create actor filler-$i --template=ate-demo-counter/counter --atespace=tenant-b
curl -s -H "Host: filler-$i.tenant-b.actors.resources.substrate.ate.dev" http://localhost:8000/ > /dev/null
done
💡
Substrate has no explicit "migrate actor" API. An actor's snapshot lives in object storage, and its next resume simply restores it onto whichever free worker the placement logic picks (a random shuffle over free Workers with no affinity to or away from the previous pod). The demo wants to prove a cross-worker resume; that session identity travels with the actor rather than being tied to the Worker. However, after suspension, the Actors old Worker returns to the free pool, so a random pick could land it right back on the same pod and demonstrate nothing. The filler step rigs the odds: each kubectl ate create actor filler deliberately creates a brand-new actor (not a move of the suspended one), and the curl triggers its first resume purely to claim a Worker slot, occupying the old worker so the target Actors wake-up probably lands elsewhere.
- Wake the Actor.
curl -s -H "Host: agent-1.tenant-a.actors.resources.substrate.ate.dev" http://localhost:8000/
- Get the output from the Actor.
kubectl ate get actors --atespace=tenant-a
You will see that the Actor is fully restored on another Worker with the same state.
Cleanup
for i in 1 2 3; do
kubectl ate suspend actor filler-$i --atespace=tenant-b 2>/dev/null
kubectl ate delete actor filler-$i --atespace=tenant-b
done
kubectl ate suspend actor agent-1 --atespace=tenant-a 2>/dev/null
kubectl ate delete actor agent-1 --atespace=tenant-a
kubectl ate suspend actor agent-1 --atespace=tenant-b 2>/dev/null
kubectl ate delete actor agent-1 --atespace=tenant-b
kubectl ate delete atespace tenant-a
kubectl ate delete atespace tenant-b
# Leave the system-managed `ate-golden` atespace alone - the control plane
# uses it for golden-snapshot actors.
# Stop the router port-forward
kill %1 2>/dev/null
Wrapping Up
Pods fail for multiple reasons, and they also move from one Worker Node to another. To ensure that Actors keep state intact (e.g - the Agent they're running), there needs to be the ability to move them from one Worker to another without losing data. In this blog post, you learned how to ensure Actors can move between Workers while still keeping state and Actor Identity.
SOCIAL SHARE CARD GENERATOR