Agent Substrate Day

Running Agent Substrate
on EKS

What it takes on AWS, which services to use,
and where AWS can help the project

Carlos Santana

Sr. Specialist Solutions Architect, Amazon Web Services

linkedin.com/in/csantanapr

Agenda

Agent Substrate and its architecture on AWS
Cluster, gVisor, microVM, mode 3 on AWS Nitro Sandbox
Snapshots on S3, IAM roles, state on RDS
Secrets, Bedrock and per-actor volumes
Two demos, lessons, production checklist

Everything shown runs on

Agent Substrate v1.0.0EKS 1.37Amazon Linux 2023 m8i (nested virt)S3RDS for PostgreSQL 18 Secrets ManagerECRBedrock EFSS3 Files

Every Service is ClusterIP: no public endpoints

The path for the next half hour, from what Agent Substrate is to what we learned.

What Agent Substrate is, and how it maps onto AWS.
Then the pieces, one at a time:
- the cluster on EKS 1.37 and its two sandbox classes, gVisor and microVM, plus a third mode on AWS Nitro Sandbox
- snapshots on S3, and one IAM role per component
- state on RDS, with rotating logins
- actor secrets and Bedrock, through the egress gateway
- per-actor volumes on Amazon EFS and S3 Files
Two demos, then what we learned: the lessons, where AWS can help, and a production checklist.
point at the right card Everything shown runs on the published release images from ghcr.io/agent-substrate/substrate.
Plus a small local installer patch for S3 and EKS paths.
No application endpoint is public: every Service is ClusterIP, and every demo goes through a port forward.

Next: the idea behind Agent Substrate, in one slide.

Thousands of idle agents share a small pool of worker pods

One EKS cluster, three node groups, managed AWS services around it

One EKS cluster, and every cloud need maps to a managed AWS service.

Three managed node groups on AL2023: system, gvisor and microvm. point at the three boxes
Everything the project needs from a cloud maps to a managed AWS service. point at the right column
The components that call AWS use EKS Pod Identity, not static keys:
- ate-api-server, atelet and the LLM gateway
- the Secrets Manager credential provider and the EBS CSI driver
Actors get no AWS identity at all.
Secrets Manager holds two kinds of secret:
- the database logins
- the secrets actors use, which the egress gateway injects into their requests, so the actor never holds them

Next: the whole install on one page.

Everything runs in private subnets, with no public application endpoints

Agent Substrate on Amazon EKS in us-west-2: a VPC with public and private subnets; in the private subnets the EKS cluster with system, gVisor and microVM node groups running ate-api-server, ate-controller, ate-idp-server, atenet-router, atenet-egress, External Secrets with the Reloader, OpenClaw, efs-csi-controller-ate, agentgateway, atelet and worker pods; plus RDS for PostgreSQL and the rotation Lambdas; with Amazon S3, S3 Files, EFS, EBS, ECR, Secrets Manager, Bedrock, IAM with Pod Identity, CloudFormation and CloudWatch Logs.

The whole install on one page: everything runs in private subnets, with no public load balancers.

Outside AWS, only two things: your laptop, and ghcr.io, where the Substrate release images come from.
You reach the EKS API with IAM, and the Substrate API through kubectl port-forward.
The system node group runs the control pieces:
- ate-api-server, with its state in RDS and its snapshots in S3
- the controller and the identity server
- the router that brings traffic in to actors
- the egress gateway, the only way out
The gVisor and microVM node groups run atelet and the worker pods that host the actors.
Each pod that calls AWS has its own IAM role through Pod Identity. Actors have no AWS identity.
Bedrock is reached only through agentgateway. The database logins rotate through Secrets Manager.
Reading the picture, if there is time:
- dashed lines are TLS over the public internet
- rotation Lambdas rotate the owner and readwrite secrets every 30 days
- nodes run AL2023 with IMDSv2 and a hop limit of 1; the microvm launch template turns on nested virtualization
- EFS and S3 Files access points are per actor volume, in the optional demo
- in the cluster, not drawn: k8s-credential-provider and podcertcontroller (mTLS pod certificates)
- add-ons, not drawn: vpc-cni (NetworkPolicy), CoreDNS, kube-proxy, Pod Identity Agent, EBS CSI and EFS CSI
- the full-size image is in the repo README, so people can read the small labels later
- the editable source is agent-substrate-on-eks.drawio, next to this deck

Next: every Substrate dependency, mapped to an AWS service.

Every Substrate dependency maps to an AWS service

Agent Substrate needs	On AWS	Status in v1.0.0
PodCertificate + ClusterTrustBundle	EKS 1.37 (both GA)	works, no flags
Snapshot object store	Amazon S3	works
microVM guest assets	S3 asset bucket	works; small patch for S3
PostgreSQL	Amazon RDS for PostgreSQL	works
Workload identity	EKS Pod Identity	works
Actor image pulls	Amazon ECR	works
Nested virt for microVM	Intel 7th/8th gen (m8i here)	works
Per-actor volumes (CSI)	Amazon EFS and S3 Files	works; ExternalVolumes preview
Egress credential injection	AWS Secrets Manager	works
Metrics	Amazon Managed Service for Prometheus	ships a GKE PodMonitoring

Read the green column first: almost everything works on a stock EKS cluster with the published release.

The orange rows are where you need a little help today.
microVM guest assets: the stock installer stages them to Google Cloud Storage. A small installer patch in this repo stages them to S3.
Per-actor volumes work through the ExternalVolumes preview and a second EFS CSI controller behind Envoy.
Metrics: the install ships a GKE PodMonitoring object, so on EKS you wire Prometheus yourself.
Each orange row comes back on the "where AWS can help" slide.
Detail behind the rows, if asked:
- Kubernetes needs PodCertificate and ClusterTrustBundle
- S3 through the objectstore-plugin sidecar, with ATE_STORAGE_BACKEND=s3
- the asset bucket is read-only for atelet
- RDS: bring your own DSN and the server CA
- Pod Identity for atelet, ate-api-server and the LLM gateway
- ECR through the kubelet ecr-credential-provider that atelet runs
- nested virtualization with NestedVirtualization=enabled
- volumes through the EFS CSI driver add-on, with an mTLS-fronted controller
- Secrets Manager through the aws-sm-credential-provider plugin

Next: the cluster config.

One node group per sandbox class, network policy on eksctl · EKS 1.37 · AL2023

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata: { name: substrate-day,
            region: us-west-2, version: "1.37" }
addons:
  - name: vpc-cni  # + enableNetworkPolicy: "true"
  - name: eks-pod-identity-agent
  - name: aws-ebs-csi-driver
managedNodeGroups:
  - name: system   # control plane components
    instanceType: m8i.xlarge
    disablePodIMDS: true   # IMDSv2, hop limit 1
  - name: gvisor
    instanceType: m8i.2xlarge
    disablePodIMDS: true
    taints:
      - { key: ate.dev/sandboxClass, value: gvisor,
          effect: NoSchedule }
  - name: microvm
    instanceTypes: [m8i.2xlarge]
    launchTemplate: { id: lt-… }  # nested virt, hop limit 1
    taints:
      - { key: ate.dev/sandboxClass, value: microvm,
          effect: NoSchedule }

One tainted node group per sandbox class
Turn on VPC CNI network policy
mTLS needs no feature flags in 1.37
Private subnets for nodes and data stores

The eksctl config, shortened: one tainted node group per sandbox class, and network policy on.

One managed node group per sandbox class, each tainted ate.dev/sandboxClass=<its class>.
Substrate's worker pods carry the matching toleration on their own, so nothing else lands there.
Turn on network policy in the vpc-cni add-on. Substrate ships default-deny NetworkPolicies for ate-system and one per WorkerPool.
Without enforcement, those policies do nothing.
PodCertificateRequest and ClusterTrustBundle are GA in Kubernetes 1.37, so Substrate's mTLS identity layer needs no feature flags and works on a stock cluster.
disablePodIMDS sets IMDSv2 with a hop limit of 1, so pods can't use the node's credentials.
The microvm group gets the same settings, plus nested virtualization, from its launch template.

Next: the two sandbox classes these node groups run.

Two sandbox classes, one cluster

gVisor · soft isolation

Userspace kernel intercepts syscalls
Runs on any node, no special hardware
Native checkpoint/restore of the process tree
Class gvisor; default SandboxConfig gvisor-default

microVM · VM isolation

Kata guest on Cloud Hypervisor, its own guest kernel
Needs /dev/kvm; atelet advertises it as ate.dev/kvm
Memory snapshot, restored on demand with userfaultfd
Class: microvm, guest assets from S3

Same ActorTemplate shape for both. Pick per workload with sandboxClasses on the WorkerPool.

Two sandbox classes run side by side in one cluster.

gVisor is soft isolation: a userspace kernel, runsc, intercepts the actor's syscalls. point at the left stack
gVisor runs on any node, with no special hardware, and checkpoints and restores the process tree natively.
microVM is VM isolation: Kata runs the actor with its own guest kernel inside Cloud Hypervisor. point at the right stack
microVM needs /dev/kvm. atelet advertises it as the ate.dev/kvm resource.
Its memory snapshot is restored on demand with userfaultfd, and its guest assets come from the S3 asset bucket.
The ActorTemplate looks the same for both. The WorkerPool's sandboxClasses picks which one a workload gets.

Next: a 3rd mode on AWS Nitro Sandbox, the installer's opening animation.

gVisor, microVM, and a 3rd mode on AWS Nitro Sandbox

The installer's opening animation: the Agent Substrate mascot, the title, and actors dropping onto gVisor, microVM and AWS Nitro Sandbox nodes in an Amazon EKS cluster

An interactive installer on top of the same scripts

A loop through the installer's six screens: Welcome, Check your setup, Settings with the demos and plugins checked, Review with the config file and the commands in order, Provision with its live checklist and log at three points in the run, and Done with what was created and the step times

What it adds

Checks first: tools, account, Region, vCPU quota, nested virtualization
Pick the cluster name, Region, node groups, demos and plugins
Live checklist from the scripts' own steps
Teardown from the Done screen
Same create-cluster.sh and deploy.sh underneath

An interactive installer, the same idea as the interactive installer for GKE.

A small terminal UI in Go with no install logic of its own: it runs create-cluster.sh and deploy.sh, and reads their step lines.
First it checks your setup:
- the tool versions
- which account and role you are signed in as, and the Region
- whether the cluster name is free
- that the instance type supports nested virtualization, and the EC2 vCPU quota
On Settings you pick the cluster name, Region, profile and node groups, then the demos and plugins: EFS, S3 Files, login rotation, Mode 3 and the credential provider. point at the checked boxes
Then it shows a review of the settings and the commands it will run.
While it installs, it turns the scripts' steps into a live checklist with times, next to the log.
The Done screen prints how to reach the cluster, and has a Teardown action that runs clean.sh and delete-cluster.sh.
The default choices take about 30 minutes; this run, with every demo and plugin, took about an hour. It lives in the installer folder of the talk repo.
The loop walks the six screens in order, about 40 seconds; talk through each step as it comes up.
It is the fastest way to deploy; the same scripts still run on their own.

Next: the third mode, on AWS Nitro Sandbox.

A third mode: each worker Pod is an AWS Nitro Sandbox upstream proposal

Mode 3 runs on images from the proposal branch, ghcr.io/csantanapr/substrate. The installer turns it on.

Mode 3 is a proposal for upstream, not part of v1.0.0. The worker Pod itself runs as an AWS Nitro Sandbox.

point at the left two columns The two modes in v1.0.0: gVisor is soft isolation, a userspace kernel; microVM is VM isolation, Cloud Hypervisor with its own guest kernel.
point at the right column In mode 3, each worker Pod is its own Nitro sandbox, on RuntimeClass isolated-sandbox. That is hardware isolation from AWS Nitro, per Pod.
EKS Auto Mode launches the nodes for these Pods. No node group and no extra runtime to install.
The sandbox can't see the node's files, so atelet reaches the worker over mutual TLS on the Pod network, through the atelet-workers Service on port 8087, and moves sandbox assets and snapshots through a staging service.
Only if asked: the WorkerPool still names the gvisor sandbox class; networkTransport.runtimeClassName is what puts its workers in Nitro sandboxes, behind the NetworkWorkers preview gate.

Next: how mode 3 runs on EKS Auto Mode.

Mode 3 on AWS Nitro Sandbox: hardware isolation per worker Pod

apiVersion: ate.dev/v1
kind: WorkerPool
spec:
  networkTransport:
    runtimeClassName: isolated-sandbox  # from EKS Auto Mode

Why it matters

Hardware isolation per Pod: each worker Pod is its own AWS Nitro Sandbox
A Pending Pod is the request: Auto Mode launches an instance that fits
VPC networking, Pod Identity and network policies work as for any Pod
Try it: the installer's isolated-sandbox Mode 3 plugin

Mode 3 on AWS Nitro Sandbox in EKS Auto Mode.

point at the left The worker Pod asks for RuntimeClass isolated-sandbox, which Auto Mode already provides. While it is Pending, Auto Mode's built-in Karpenter picks and launches an instance that can run it.
NodePool ate-isolated is a copy of the built-in general-purpose NodePool, with a label and a taint so only worker Pods land there.
point at the right Each worker Pod runs in its own Nitro sandbox: hardware isolation from AWS Nitro, per Pod.
No node group and no extra runtime to install. The Pods keep full VPC networking, EKS Pod Identity and network policies.
Several sandboxes share one instance, so a node holds many workers.
It needs EKS Auto Mode; it does not run on managed node groups.

Next: how microVMs run without bare metal.

Nested virtualization runs microVMs without bare metal

Recent Intel families in us-west-2:
c7im7ir7ic8im8ir8ii7ix8i
Turned on per instance with CpuOptions
No bare metal needed for microVMs
Check a node for /dev/kvm and vmx

# launch template data for the microvm node group
{
  "CpuOptions": { "NestedVirtualization": "enabled" },
  "MetadataOptions": {
    "HttpTokens": "required",
    "HttpPutResponseHopLimit": 1 },
  "BlockDeviceMappings": [{ "Ebs": {
    "VolumeSize": 200, "Encrypted": true } }]
}

# find instance types that support it
aws ec2 describe-instance-types --filters \
  Name=processor-info.supported-features,\
Values=nested-virtualization

# on a microvm node
crw-rw-rw- 1 root kvm 10, 232 /dev/kvm
kernel 6.18.56-125.164.amzn2023.x86_64

The microVM sandbox needs KVM, and nested virtualization provides it without bare metal.

Without nested virtualization, KVM means bare metal instances.
Recent Intel families support it, plus their flex and d variants. This demo uses m8i.2xlarge.
You turn it on per instance with CpuOptions in the launch template, so the microvm node group has its own template.
The same template keeps IMDSv2 with a hop limit of 1 and an encrypted 200 GB volume.
create-cluster.sh checks the instance type with the describe-instance-types filter shown here, before it creates anything.
On a node, /dev/kvm exists and the CPU exposes vmx. point at the bottom of the code
Only if anyone asks: on Windows instances, it disables Credential Guard's Virtual Secure Mode. No effect on AL2023 nodes.

Next: where suspended actors live: S3.

Suspended actors live in S3, moved by a sidecar

objectstore-plugin sidecar moves every snapshot
ATE_STORAGE_BACKEND=s3 with Pod Identity
Read-only asset bucket for microVM assets

Bucket best practices

Block Public AccessBucketOwnerEnforceddefault encryptionTLS-only bucket policyabort incomplete uploads

# a golden (tag) snapshot in S3
atespaces/ate-demo-counter/tags/<uid>/
  manifest.json
  pages.img.zstd          # process memory
  pages_meta.img.zstd
  fs/pages.img.zstd       # durable volumes
  rootfs/fscheckpoint.pb.zstd
  rootfs/multitar.img.zstd
  rootfs/pages.img.zstd   # root filesystem writes

# asset bucket (read-only for atelet)
microvm-assets/amd64/<sha256>   # cloud-hypervisor
microvm-assets/amd64/<sha256>   # virtiofsd
microvm-assets/amd64/<sha256>   # guest kernel
microvm-assets/amd64/<sha256>   # guest rootfs

A suspended actor is a set of objects in S3, moved by a sidecar.

Neither atelet nor ate-api-server talks to storage directly: an objectstore-plugin sidecar does every transfer.
Select S3 with ATE_STORAGE_BACKEND=s3. The sidecar uses the AWS SDK default chain, so its credentials come from Pod Identity.
The snapshot location is one cluster-wide flag: --snapshot-location s3://bucket/.
point at the code A golden snapshot for a template tag holds process memory, durable volumes and root filesystem writes.
An actor's own suspends go under atespaces/<a>/actors/<uid>/snapshots/<name>/.
A separate asset bucket holds the microVM guest kernel, rootfs, VMM and virtiofsd, content addressed by sha256, read-only for atelet.
Bucket settings:
- Block Public Access on, and BucketOwnerEnforced
- default encryption: SSE-S3 here; use SSE-KMS with a Bucket Key if you need control of the key
- a bucket policy that denies requests without TLS
- a lifecycle rule that aborts incomplete multipart uploads

Next: which IAM role makes those S3 calls.

Identity: EKS Pod Identity, one role per ServiceAccount

# trust policy, scoped to this account and cluster
{ "Principal": { "Service": "pods.eks.amazonaws.com" },
  "Action": ["sts:AssumeRole", "sts:TagSession"],
  "Condition": {
    "StringEquals": { "aws:SourceAccount": "<account>" },
    "ArnLike": { "aws:SourceArn": "arn:aws:eks:…:cluster/substrate-day" } } }

aws eks create-pod-identity-association \
  --namespace ate-system --service-account atelet \
  --role-arn arn:aws:iam::<account>:role/substrate-day-us-west-2-atelet

Lesson: atelet pulls actor images with its own role

ECR pull goes on atelet's Pod Identity role, not the node role

ServiceAccount	Allowed
`ate-system/ate-api-server`	snapshot bucket: get, put, delete, list
`ate-system/atelet`	snapshot bucket: get, put (no delete) · asset bucket read · ECR pull
`llm-gateway/llm-gateway`	`bedrock:InvokeModel*` on its pinned inference profiles
actors	nothing: no role, no keys, no IMDS

Every component that calls AWS gets its own IAM role through EKS Pod Identity, one per ServiceAccount.

The trust policy trusts the Pod Identity service. aws:SourceAccount and aws:SourceArn limit it to this account and this cluster.
Then one association binds the role to a namespace and ServiceAccount.
The table: point at it
- ate-api-server: get, put, delete and list on the snapshot bucket, because it makes tag copies and garbage collects
- atelet: get and put, no delete; read on the asset bucket; ECR pull on the demo repositories
- the LLM gateway may invoke only its pinned inference profiles
Actors get nothing: no role, no keys, no IMDS.
Lesson: atelet pulls actor images itself. It runs the node's ecr-credential-provider inside the atelet pod.
So the provider uses atelet's Pod Identity role, not the node role.
Give it ECR pull on your private actor repositories, or those templates' golden snapshots fail at image pull with a 401.

Next: the same roles, drawn per S3 action.

Each component gets only the S3 actions it needs

The two sidecars do different jobs, so each gets only the S3 actions it needs.

Neither ate-api-server nor atelet calls S3 itself: each one's objectstore-plugin sidecar does, with its pod's Pod Identity role.
Control sidecar, next to ate-api-server:
- copies a golden snapshot when a tag is made, and deletes snapshots nothing refers to any more
- so it needs list, get, put and delete on the snapshot bucket, and nothing on the asset bucket
Node sidecar, next to atelet on every worker node:
- uploads a snapshot when an actor suspends, and fetches it when the actor resumes, so it needs get and put, but not delete
- it also reads microVM guest assets and checks each one's sha256 before atelet sees it, so the asset bucket is read-only for it
The operator stages those assets once, at install time, with their own credentials. No pod can write them.
ECR pull sits on atelet's role too, because atelet runs the node's ecr-credential-provider. Actors get nothing.
Why it matters: worker nodes run untrusted code, so they are the likeliest to be compromised. From there, you can't delete snapshots.
The next step is per-actor prefixes:
- on GCS, the credential exchange hands each transfer a token scoped to one actor
- on S3, it isn't implemented yet
- STS session policies or S3 Access Grants would close that gap; it's on the "where AWS can help" list

Next: where actor state lives: RDS.

Actor state lives in RDS: private, TLS verify-full, two logins

Private subnets; 5432 only from the cluster
Two logins: owner and readwrite
sslmode=verify-full against the RDS CA bundle
Connection strings as env vars, never flags

# install: logins read from Secrets Manager
export ATE_API_POSTGRES_OWNER_CONNECTION_STRING=\
"postgres://substrate_owner_user:…@<rds>:5432/atepg\
?sslmode=verify-full\
&sslrootcert=/run/postgres-server-ca/server-ca.pem"
export ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING=…

ate-setup deploy ate-system \
  --image-repo ghcr.io/agent-substrate/substrate \
  --image-tag v1.0.0 \
  --snapshot-location s3://<snapshots>/ \
  --ateapi-postgres-server-ca-file global-bundle.pem \
  --sandbox-classes gvisor,microvm

Substrate keeps actor and template state in PostgreSQL, and Amazon RDS runs it.

The instance is in the private subnets. Its security group allows port 5432 only from the EKS cluster security group.
RDS manages the master password in Secrets Manager. It's used once, to bootstrap the roles.
Substrate then uses two logins, each with its own secret: owner for migrations and DDL, readwrite at runtime.
Connections use sslmode=verify-full against the RDS CA bundle, which ate-setup mounts at /run/postgres-server-ca.
The connection strings go through environment variables only, never command-line flags, because other users can read flags in ps.
Two bootstrap tips for RDS:
- the master user is rds_superuser, which is not a real superuser, so grant substrate_owner to it first
- run setup.sql in a single transaction, with --single-transaction

Next: rotating those two logins.

Rotate the DB password, and ate-api-server follows

A rotation reaches ate-api-server only through a restart, so the job is to make that restart automatic.

ate-setup puts both connection strings in one Kubernetes Secret, ate-api-server-secret-envvars. ate-api-server reads it as environment variables when it starts.
First, Secrets Manager rotates the owner and readwrite logins, with the alternating-users strategy:
- native RDS rotation covers only the master user, so the app logins need a rotation function
- AWS hosts the code: one CloudFormation transform creates the Lambda in your private subnets, and it logs in as the master user
- it flips between two database users, user and user_clone, so the login the running pods hold keeps working while new pods start
- both users hold the same Substrate role, so either one can SET ROLE to it
- single-user rotation changes the password in place, and new connections fail until the rollout finishes
Second, External Secrets Operator syncs the secret into the cluster:
- its controller has its own Pod Identity role, and checks Secrets Manager every minute
- it templates the full DSN from the secret's username and password
- nothing changes in the ate-api-server pod
Third, the ESO Reloader watches that Secret, and does a rolling restart of ate-api-server when it changes.
With two replicas, the API stays up the whole time.

Next: the wiring, with no change to the Substrate manifests.

Rotation wiring, with no change to the Substrate manifests

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: ate-db, namespace: ate-system }
spec:
  refreshInterval: 1m
  secretStoreRef: { kind: SecretStore, name: aws }  # Pod Identity
  target:
    name: ate-api-server-secret-envvars  # what ate-setup made
    creationPolicy: Merge               # keep its other keys
    template:
      data:
        ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING: >-
          postgres://{{ .user }}:{{ .pw | urlquery }}@<rds>:5432/…
  data:
  - secretKey: user
    remoteRef: { key: <cluster>/ate-readwrite, property: username }
  - secretKey: pw
    remoteRef: { key: <cluster>/ate-readwrite, property: password }
  # the owner login: the same, with ate-owner
---
apiVersion: reloader.external-secrets.io/v1alpha1
kind: Config
spec:
  notificationSources: [{ type: KubernetesSecret }]
  destinationsToWatch:
  - { type: Deployment, deployment: { names: [ate-api-server] } }

Rotation

AWS-hosted PostgreSQLMultiUser Lambda
Every 30 days, or rotate-secret now

External Secrets Operator

Templates both DSNs
Controller only: nothing listening

ESO Reloader

Rolls ate-api-server on Secret change

The rotation wiring needs no change to the Substrate manifests.

The ExternalSecret writes into the Secret ate-setup already created. point at the top of the YAML
creationPolicy: Merge means it updates only the keys it templates, and leaves the schema key alone.
The SecretStore has no auth block, so ESO uses its controller's Pod Identity role: GetSecretValue on this cluster's two login secrets only.
The template URL-encodes the password, because it sits inside a URL. It builds both DSNs from the username and password, and keeps sslmode=verify-full and the CA path from the RDS slide.
The rotation Lambda is the AWS-hosted PostgreSQLMultiUser function, in the private subnets with its own security group.
ESO runs as a controller only: no webhook, nothing listening. Its IAM is secretsmanager:GetSecretValue on <cluster>/ate-*.
The ESO Reloader Config lives outside the Deployment, so nothing goes into the ate-setup manifests, and a redeploy can't drop it:
- it watches Secrets, and rolls any Deployment in its list that reads the changed Secret through envFrom
- it restarts by stamping a last-reloaded annotation on the pod template
Upstream: if ate-api-server read its password from a mounted file on each new connection, rotation would need no restart.

Next: secrets for the actors themselves.

Actors use API keys they never hold

An agent can use an API key without ever holding it. Follow the numbers on the diagram.

1. The actor sends its request with a placeholder in the Authorization header.
Every request out of an actor goes through atenet-egress, which terminates TLS and matches the actor's EgressPolicy.
2. The rule names a secret by URI, so the gateway asks the credential provider for it, over mTLS, passing the actor's identity.
3. The provider is a small plugin we wrote, credprovider-aws-secret-manager, running in ate-system:
- it admits only the egress gateway
- it checks a default-deny policy that maps each atespace to the secrets it may use
- then it calls GetSecretValue with its own Pod Identity role
- that role can read only secrets tagged ate.dev/injectable=true, so even a bug in the policy can't reach the database logins
4 and 5. The value goes back to the gateway, which swaps it into the header and sends the request on.
The actor never sees the secret: not in its memory, its files, or its snapshots in S3.
A 30-second cache keeps Secrets Manager calls per secret, not per request. A rotation reaches actors within that window, with no redeploy.

Next: the configuration behind it, and its three gates.

Actor secrets: the wiring and its three gates

# EgressPolicy on the actor
rules:
- https:
    hostnames: ["api.github.com"]
    effects:
      replaceHeaders:
      - header: Authorization
        prefix: "Bearer "
        credentialUri: ate-secret://secretsmanager.amazonaws.com/
          accounts/<id>/regions/us-west-2/
          secrets/ate/team-a/github/stages/AWSCURRENT
---
# provider policy: atespace -> secrets, default deny
policies:
- atespace: team-a
  allowedSecrets:
  - { account: "<id>", region: us-west-2, namePrefix: ate/team-a/ }

Three gates

mTLS: only atenet-egress may ask
Provider policy per atespace
IAM: ate/*, tag ate.dev/injectable

On EKS

Pod Identity role, no keys

Rotation

AWSCURRENT cached 30s, no redeploy

Three pieces of configuration put three gates between an actor and a secret.

The actor's EgressPolicy names the host, the header and the secret URI: account, Region, name and stage. point at the top of the YAML
The provider's own policy maps each atespace to its secrets by account, Region and name prefix. Everything else is denied: the atespace must be granted the secret.
The provider's IAM role only reads secrets under ate/ that carry the ate.dev/injectable tag: GetSecretValue on ate/* with ate.dev/injectable=true.
point at the Three gates card So the gates are mTLS (only atenet-egress may ask), the provider policy, and IAM.
On EKS, the provider gets its role through Pod Identity, and ships a NetworkPolicy that only lets the egress gateway in.
It is registered with the gateway next to the bundled Kubernetes Secrets provider (k8s.io), so both work at once.
Rotation: AWSCURRENT is cached for 30s. Rotate in Secrets Manager, and actors pick it up with no redeploy.
The full example policy is demos/credprovider-aws/egress-policy.example.yaml. The plugin is github.com/csantanapr/credprovider-aws-secret-manager.

Next: Bedrock, where a header swap is not enough.

Actors reach Bedrock through a gateway, with no AWS credentials

Lesson: block IMDS for pods

Set disablePodIMDS: true (hop limit 1) on every node group, so the gateway signs with its own role, never the node instance role.

Why a gateway, not header injection

SigV4 signs each request's body hash, so signing lives in a proxy that holds the role.

Actors reach Bedrock through a gateway, and hold no AWS credentials.

The actor talks to a model with a normal Anthropic or OpenAI SDK. follow the flow left to right
It leaves through atenet-egress: the EgressPolicy allows one host, the in-cluster LLM gateway, and swaps the placeholder for a signed actor JWT.
The gateway is agentgateway, a ClusterIP service. It signs the call to Bedrock with SigV4, using its own Pod Identity role.
That role may invoke the pinned inference profiles only:
- InvokeModel on each profile ARN
- and on the profile's foundation models, only when bedrock:InferenceProfileArn is that profile
Model claude maps to the default profile. claude-haiku, claude-sonnet, claude-opus: the newest active profile per family, read at deploy time.
Lesson: block IMDS for pods.
- eksctl managed node groups get a launch template, and with one the IMDS hop limit defaults to 2, so pods can reach IMDS
- with it open, agentgateway's credential chain signed with the node instance role and got a 403
- set disablePodIMDS: true on every node group, for a hop limit of 1, and AWS_EC2_METADATA_DISABLED=true on the gateway, so it fails closed
Why not inject credentials at egress, like other secrets?
- egress credential providers swap in fixed header values
- SigV4 signs the body hash of every request, so signing has to live in a proxy that holds the role

Next: per-actor volumes on EFS.

Per-actor EFS volumes follow the actor to any zone CSI · ExternalVolumes preview

Per-actor EFS volumes follow the actor to any node, in any Availability Zone.

Snapshots carry an actor's memory and its own disk. Some agents want a shared, durable file system:
- a repository checkout, a dataset, files that outlive the actor
Substrate does this with CSI external volumes, still behind the ExternalVolumes preview. It doesn't use PersistentVolumeClaims.
The control plane calls the CSI controller itself: on an actor's first resume, ate-api-server calls CreateVolume. point at the top row
The EFS driver creates an access point just for that actor, owned by the user it runs as. Deleting the actor deletes it.
Each time the actor resumes, atelet asks the EFS node plugin on that node to mount it, over TLS.
EFS is regional, with a mount target in every AZ: resume in another zone, and the files are still there. point at node A, then B
Amazon S3 Files works through the same driver, and the files also show up as objects in an S3 bucket.

Next: the configuration, and a captured run across zones.

EFS wiring, and an actor moving between zones

# StorageClass: one access point per actor
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap    # s3files-ap for S3 Files
  fileSystemId: fs-…
  directoryPerms: "700"
  uid: "<uid>"              # EFS_UID: the actor's user
  gid: "<gid>"
  basePath: /ate
---
# CSIDriverConfig: ate-api-server finds the controller
driverName: efs.csi.aws.com
controllerEndpoint:
  tcp://efs-csi-controller-ate.kube-system.svc…:50052
tls:
  enabled: true
  usePodIdentity: true
  serverName: efs-csi-controller-ate.kube-system.svc
---
# ActorTemplate
volumes:
- name: data
  external: { template: { capacityBytes: 1073741824,
                          storageClassName: efs-ate } }
containers:
- volumeMounts: [{ name: data, mountPath: /home/counter }]

Storage	Sandbox	Zones
EFS	gVisor	us-west-2c → us-west-2b
EFS	microVM	us-west-2a → us-west-2b
S3 Files	gVisor	us-west-2b → us-west-2c
S3 Files	microVM	us-west-2b → us-west-2a

A captured run: same access point, new zone

Best practices

Give the access point the actor's uid
Run actors as non-root

This is all the configuration it takes, plus a captured run across zones.

The StorageClass makes the EFS driver create an access point per volume under /ate, owned by the actor's uid (EFS_UID, EFS_GID).
The demo's counter image runs as root, so the demo sets 0. A non-root image is better.
The uid matters: Substrate drops actor capabilities to a small default set, so even root in the sandbox can't write another uid's directory.
The CSIDriverConfig points ate-api-server at the controller: today a second controller, same image as the EKS add-on, fronted by Envoy with mTLS.
Why a second one: the add-on's controller only listens on a socket inside its own pod.
The ActorTemplate just declares the volume and where to mount it.
point at the table A captured run on this cluster, for both storage types and both sandbox classes:
- each row: the zone it was suspended in, then the zone it resumed in
- we counted on one node, suspended, emptied that node, and resumed
- every actor came back in a different Availability Zone, and its file kept counting, from the same access point
Best practices:
- match the access point's uid to the actor's user: actors run without CAP_DAC_OVERRIDE
- run actors as non-root; on S3 Files, root needs ClientRootAccess
- turning on the preview reinstalls Substrate and rolls workers: plan it

Next: the demo.

Demo: suspend frees the worker, and every count keeps going gVisor and microVM

Watch for

Counts in memory, a file and rootfs: 3, 3, 3
Suspend: SUSPENDED, the worker is free
Next request resumes it: all three read 4

kubectl ate create actor demo-gvisor \
  -a ate-demo-counter --template counter
kubectl ate proxy actor demo-gvisor \
  -a ate-demo-counter 8001:80 &
curl localhost:8001
kubectl ate suspend actor demo-gvisor -a ate-demo-counter
curl localhost:8001     # resumes

== demo-gvisor (gVisor) memory: 3 | file: 3 | rootfs: 3 $ kubectl ate suspend actor demo-gvisor \ -a ate-demo-counter state: ACTOR_STATE_SUSPENDED memory: 4 | file: 4 | rootfs: 4 state: ACTOR_STATE_RUNNING == demo-microvm (Kata + Cloud Hypervisor) memory: 3 | file: 3 | rootfs: 3 $ kubectl ate suspend actor demo-microvm \ -a ate-demo-counter-microvm state: ACTOR_STATE_SUSPENDED memory: 4 | file: 4 | rootfs: 4 state: ACTOR_STATE_RUNNING

Condensed from a run of this demo on the cluster above

Suspend frees the worker, and the next request brings back every count.

Tell the room what to watch before you run it. Point at the Watch for card.
One counter actor per sandbox class: gVisor and microVM.
Each one counts in three places: in memory, in a file on a durable volume, and in its root filesystem (rootfs).
You reach it with kubectl ate proxy actor, a localhost-only forward through atenet-router, so nothing is exposed.
Call it a few times: all three counters read 3.
Suspend it with kubectl ate suspend actor: the state turns SUSPENDED, and the worker pod is free for another actor.
- the snapshot is in S3, and the actor's state is in RDS
Send one more request. It resumes the actor from its snapshot, and all three read 4. pause Same result on the microVM.

Next: the same pattern, with a real agent.

Demo: an always-on agent that sleeps between messages OpenClaw

An always-on gateway holds the chat channel
Each agent is a suspendable actor
It thinks with Bedrock and holds no keys

Watch for

The actor answers, then suspends to S3
The next message wakes it from its snapshot
A new conversation gets a new actor

Flow

chat message → gateway → atenet-router → actor →
atenet-egress → LLM gateway → Bedrock → reply

actor oc-agent and its egress policy exist actor answers /readyz through atenet-router actor said: Pong, here I am ready to help whenever you need me. actor suspended, snapshot in S3 actor resumed on demand from its snapshot actor said (via gateway): Pong, right on time. gateway created a new actor + egress policy and woke it

Condensed from a run of this demo on the cluster above

This is the pattern for real agents: an always-on gateway, and an agent that sleeps between messages.

OpenClaw is split in two. An always-on gateway holds the chat channel, for example WhatsApp; each conversation's agent is a suspendable actor.
The gateway creates actors and suspends them when they go idle. The next message resumes them from their snapshot in S3.
On EKS, the agent thinks with Bedrock through the LLM gateway, so the actor holds no AWS keys.
The images are in ECR. The gateway and its dashboard are ClusterIP services behind a port forward.
Watch the log: the actor answers, it is suspended, the next message resumes it on demand, and it answers again.
The flow, hop by hop: atenet-router uses mTLS with a pod certificate, the actor resumes from S3, and atenet-egress carries the actor JWT.
A new conversation gets a brand-new actor, with its own egress policy.
Tip for your own apps: the v1 router adds forwarded headers.
- an app with strict proxy attribution, like OpenClaw, should listen on loopback behind a small front door that drops them

Next: what works on EKS today, and the rough edges.

What works on EKS today, and the rough edges

Works with v1.0.0

Published images on stock EKS 1.37 and AL2023
gVisor and microVM suspend and resume
S3, RDS, ECR and Bedrock with Pod Identity
Actor secrets injected from Secrets Manager
Per-actor EFS and S3 Files volumes (preview)

Rough edges

ate-setup assumes GKE for any real cluster
S3 uses the pod's role, not per-actor tokens
RDS needs a role grant before setup.sql
DB password rotation needs a rolling restart
EFS needs a second CSI controller

Left: what works on EKS with v1.0.0. Right: the rough edges.

Works with the published images on a stock EKS 1.37 cluster with AL2023 nodes.
These all use Pod Identity:
- S3 snapshots and microVM assets
- RDS with verify-full TLS
- ECR pulls through atelet's credential provider
- Bedrock through the LLM gateway
Also working:
- gVisor and microVM both suspend and resume
- NetworkPolicy is enforced by the VPC CNI
- actor secrets come from Secrets Manager, injected on egress
- per-actor EFS and S3 Files volumes work across nodes and zones, behind the preview gate
Right: GKE assumptions in the install:
- ate-setup treats every non-kind cluster as GKE: it wants a GCP transfer service account, uploads assets with gcloud, and writes gs:// asset URLs
- this repo carries a small installer patch for that
- the base install applies a GKE PodMonitoring
- atelet's credential provider hostPaths are GKE paths; on AL2023 they are /etc/eks/image-credential-provider
Gaps on the AWS services:
- on S3, snapshot transfers use the pod's credentials, where GCS gets per-actor scoped tokens
- on RDS, pre-create substrate_owner and grant it to the master user before setup.sql
- ate-api-server reads both database passwords once at start, so a rotation needs a rolling restart
- ate-api-server calls the CSI controller over gRPC, but the EFS add-on's controller only listens on a socket inside its pod
- so a second, Envoy-fronted controller serves Substrate

Next: most of these come back on the next slide.

Where AWS can help the project next

Neutral install seams

ate-setup deploy --component: one piece at a time
ate-setup render: manifests to your own tooling

Per-actor AWS access

Per-actor S3 credentials for snapshot transfers
Actor JWT to AWS token exchange

Plugins and drivers

Secrets Manager provider in the plugins repo
Password file for the database logins
EFS CSI endpoint Substrate can call

Most rough edges have a fix the project can take, and AWS can help with each one.

Install: neutral seams, not an EKS mode. The project wants ate-setup to stay vendor neutral.
- ate-setup deploy --component installs one component at a time, so a platform can own pieces like the database or object storage itself
- ate-setup render prints the manifests instead of applying them, so Helm, kustomize, Terraform or CDK can take them from there
- both are open upstream
Everything AWS-specific stays outside the project: a reference blueprint in eksctl, Terraform or CDK, like the repo for this talk.
- with the IAM, buckets, RDS and node groups, so a blank account gets to a running install in one command
- metrics for Amazon Managed Service for Prometheus belong in that blueprint too
Per-actor access:
- per-actor S3 credentials in the credential exchange, with STS session policies scoped to the actor's prefix, or S3 Access Grants
- that gives parity with the GCS per-actor tokens, so a compromised node can't read every snapshot
- an actor JWT to AWS token exchange with AssumeRoleWithWebIdentity, for per-actor AWS access when an actor really needs it
Plugins and drivers:
- the Secrets Manager credential provider is built as credprovider-aws-secret-manager; next is donating it to agent-substrate-plugins
- that gives production actor secrets without Kubernetes Secrets
- a password file for ate-api-server's database logins, re-read on each new connection, so Secrets Manager rotation needs no restart
- an EFS CSI controller endpoint Substrate can call: a TLS listener in the driver, or an add-on option
- then per-actor EFS and S3 Files volumes work with the add-on alone

Next: the production checklist.

Production checklist for EKS

Cluster

EKS 1.37+
Node group per sandbox class, tainted
Nested-virt Intel (m8i, c8i, r8i, 7i) for microVM
VPC CNI network policy on, IMDSv2 hop limit 1

Data

S3: private, TLS-only, encrypted, lifecycle
Separate read-only asset bucket
RDS private, verify-full, owner/readwrite split
DB logins rotated, synced, rolled out

Identity & access

Pod Identity role per ServiceAccount
No credentials in actors
--ateapi-authz-bootstrap-owners: your operators
ClusterIP only, private EKS API endpoint

If you take one slide home, take this one.

Cluster:
- EKS 1.37 or newer, with one tainted node group per sandbox class
- a nested-virtualization Intel family for microVMs
- network policy on in the VPC CNI, and IMDSv2 with a hop limit of 1
Data:
- S3 buckets: private, TLS-only, encrypted, with a lifecycle rule; a separate read-only asset bucket
- a private RDS instance with verify-full TLS and the owner and readwrite split
- the logins live in Secrets Manager, rotate with the alternating-users strategy, and are synced by External Secrets; the Reloader rolls ate-api-server
- EFS access points are owned by the actor's uid
Identity and access:
- a Pod Identity role per ServiceAccount, and no credentials in actors
- Bedrock through a gateway, with pinned profiles
- set --ateapi-authz-bootstrap-owners to your operators; this demo uses the shared-client default
- keep every Service ClusterIP, and operate with port forwards
- make the EKS API endpoint private or CIDR-restricted
For production, also turn on Multi-AZ RDS, longer backup retention and deletion protection.

Next: thank you, and where to help.

Thank you

Agent Substrate runs on EKS today.
Help us make it boring:
plugins, neutral install seams, per-actor credentials.

github.com/agent-substrate/substrateagent-substrate-plugins

Carlos Santana

linkedin.com/in/csantanapr

QR code for github.com/csantanapr/eks-agent-substrate — Scripts, demos and slides
**github.com/csantanapr/eks-agent-substrate**

Running Agent Substrateon EKS

Agenda

Everything shown runs on

Thousands of idle agents share a small pool of worker pods

One EKS cluster, three node groups, managed AWS services around it

Everything runs in private subnets, with no public application endpoints

Every Substrate dependency maps to an AWS service

One node group per sandbox class, network policy on eksctl · EKS 1.37 · AL2023

Two sandbox classes, one cluster

gVisor · soft isolation

microVM · VM isolation

gVisor, microVM, and a 3rd mode on AWS Nitro Sandbox

An interactive installer on top of the same scripts

What it adds

A third mode: each worker Pod is an AWS Nitro Sandbox upstream proposal

Mode 3 on AWS Nitro Sandbox: hardware isolation per worker Pod

Why it matters

Nested virtualization runs microVMs without bare metal

Suspended actors live in S3, moved by a sidecar

Bucket best practices

Identity: EKS Pod Identity, one role per ServiceAccount

Lesson: atelet pulls actor images with its own role

Each component gets only the S3 actions it needs

Actor state lives in RDS: private, TLS verify-full, two logins

Rotate the DB password, and ate-api-server follows

Rotation wiring, with no change to the Substrate manifests

Rotation

External Secrets Operator

ESO Reloader

Actors use API keys they never hold

Actor secrets: the wiring and its three gates

Three gates

On EKS

Rotation

Actors reach Bedrock through a gateway, with no AWS credentials

Lesson: block IMDS for pods

Why a gateway, not header injection

Per-actor EFS volumes follow the actor to any zone CSI · ExternalVolumes preview

EFS wiring, and an actor moving between zones

Best practices

Demo: suspend frees the worker, and every count keeps going gVisor and microVM

Watch for

Demo: an always-on agent that sleeps between messages OpenClaw

Watch for

Flow

What works on EKS today, and the rough edges

Works with v1.0.0

Rough edges

Where AWS can help the project next

Neutral install seams

Per-actor AWS access

Plugins and drivers

Production checklist for EKS

Cluster

Data

Identity & access

Thank you

Running Agent Substrate
on EKS