TL;DR: Kubernetes schedules LLM workloads well, but it does not give them the isolation boundary they need once they start calling tools, executing code, or handling tenant data.
Open Source Summit North America made one thing obvious: the cloud native crowd has moved from "can Kubernetes run LLM workloads?" to "what breaks when we trust Kubernetes too much?"
That is the right question.
The default Kubernetes security model assumes a pod is mostly an application packaging unit. It gives you namespaces, cgroups, seccomp, AppArmor, service accounts, admission control, and network policy. All of that matters. None of it changes the central fact that normal containers share the host kernel.
For a stateless API, that tradeoff is usually fine. For an LLM tool runner that can read files, call APIs, invoke Python, shell out to package managers, and chain actions across systems, that boundary starts looking thin.
The uncomfortable version is this: vanilla Kubernetes is orchestration, not containment.
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: kata
handler: kata
Then each workload declares the boundary it needs:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tool-using-agent
spec:
replicas: 3
selector:
matchLabels:
app: tool-using-agent
template:
metadata:
labels:
app: tool-using-agent
spec:
runtimeClassName: kata
serviceAccountName: llm-agent
containers:
- name: agent
image: ghcr.io/example/tool-agent:2026.05
My rule of thumb:
| Workload | Runtime | Why |
|---|---|---|
| Plain inference API | runc or gvisor | Low tool risk, latency sensitive |
| Retrieval worker with narrow egress | gvisor | Better syscall boundary with less operational change |
| Agent that calls tools | kata | VM boundary per pod, Kubernetes friendly |
| Arbitrary code execution | Firecracker style microVM | Treat it like untrusted tenant compute |
gVisor is the easiest first step because it integrates as an OCI runtime through runsc. Kata is the better fit when the isolation requirement is stronger and a VM per pod is acceptable. Firecracker is the most interesting boundary for code execution, but it is also the one I would least casually bolt onto an existing cluster without a real operations plan.
The Minimum Policy Set
The runtime is only one layer. I would not run LLM workloads without this set:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: llm-worker-egress
spec:
podSelector:
matchLabels:
app: tool-using-agent
policyTypes: ["Egress"]
egress:
- to:
- namespaceSelector:
matchLabels:
name: model-gateway
ports:
- protocol: TCP
port: 443
- to:
- namespaceSelector:
matchLabels:
name: telemetry
ports:
- protocol: TCP
port: 4317
Also make the service account boring:
apiVersion: v1
kind: ServiceAccount
metadata:
name: llm-agent
automountServiceAccountToken: false
If the workload does not need Kubernetes API access, do not mount a token. If it does, bind only the exact verbs it needs.
Benchmark Plan
I am not going to fake GPU numbers from a laptop. The package needs a real GPU node before publishing final performance claims.
This is the harness I would run:
SOCIAL SHARE CARD GENERATOR