Managing AI Agents in Production the Kubernetes Way

You wouldn't run a microservice in production without health checks, scaling policies, resource limits, and a GitOps workflow. But somehow, AI agents get deployed as one-off scripts, hardcoded API calls, or fragile background jobs with none of that.

The agents are becoming production workloads. The ops practices haven't caught up.

The problem with how we run agents today

Ask any engineering team running AI agents in production how they manage them and you'll hear the same story:

• The agent lives in a Python script that someone runs manually


• Scaling means copy-pasting the script and hoping the API rate limits hold


• "Monitoring" is checking if the process is still running


• Rolling back a bad system prompt means editing a file on a server somewhere


• Cost overruns happen because there are no hard limits anywhere

This is where microservices were in 2012. We solved those problems with containers, orchestration, and the Kubernetes ecosystem. We haven't applied those lessons to agents yet.

Agents are just workloads

Here's the mental model shift: an AI agent is not fundamentally different from any other long-running service. It has:

• A runtime (the model + prompt instead of application code)


• Resource consumption (tokens instead of CPU/memory)


• A health state (is it producing correct output, not just running)


• Scaling requirements (more traffic = more instances)


• Configuration (prompts, model settings, tool access)

Kubernetes already knows how to manage all of this. The missing piece was a way to express agents as first-class Kubernetes resources.

That's what we built: agentops-operator.

What it looks like

Instead of a Python script, you define an agent like this:

apiVersion: agentops.agentops.io/v1alpha1

kind: AgentDeployment

metadata:

  name: research-agent

spec:

  replicas: 3

  model: claude-sonnet-4-20250514

  systemPrompt: |

    You are a research agent. Gather and summarise information

    accurately. Always cite your sources.

  limits:

    maxTokensPerCall: 8000

    maxConcurrentTasks: 5

    timeoutSeconds: 120

  livenessProbe:

    type: semantic

    intervalSeconds: 60

Apply it:

kubectl apply -f research-agent.yaml

kubectl get agdep

# NAME             MODEL                      REPLICAS   READY   AGE

# research-agent   claude-sonnet-4-20250514   3          3       45s

Three agent pods, running, healthy, managed by Kubernetes. Scale to 10:

kubectl patch agdep research-agent --type=merge -p '{"spec":{"replicas":10}}'

That's it.

The parts that aren't just "Kubernetes but for agents"

Semantic health checks

Standard Kubernetes liveness probes check if a process responds to HTTP. They have no idea whether your LLM is producing useful output or hallucinating garbage.

agentops-operator introduces semantic health checks — a secondary LLM call that validates actual output quality:

livenessProbe:

  type: semantic

  intervalSeconds: 60

  validatorPrompt: "Reply with exactly one word: HEALTHY"

Pods that fail the semantic probe are removed from routing until they recover. Your load balancer never sends tasks to a degraded agent. No other Kubernetes tooling does this.

Hard resource limits at the infrastructure level

Token limits aren't in your application code where a developer can accidentally remove them. They're enforced at the infrastructure level, applied to every pod the operator creates:

limits:

  maxTokensPerCall: 8000

  maxConcurrentTasks: 5

  timeoutSeconds: 120

A runaway agent cannot burn your API budget beyond what you've declared in the YAML.

GitOps for prompts

A system prompt change is a pull request. A rollback is git revert. The audit trail of who changed what, when, and through which approval process is standard Kubernetes, no custom tooling required.

This matters for compliance teams. It also matters when a prompt change goes wrong at 2am and you need to roll back immediately without touching a server.

Multi-agent pipelines as config

Chaining agents together usually means writing custom orchestration code, managing queues, and handling failures by hand. With AgentPipeline, it's declarative:

apiVersion: agentops.agentops.io/v1alpha1

kind: AgentPipeline

metadata:

  name: research-then-summarize

spec:

  input:

    topic: "AI in healthcare"

  steps:

    - name: research

      agentDeployment: research-agent

      inputs:

        prompt: "Research this topic: {{ .pipeline.input.topic }}"

    - name: summarize

      agentDeployment: summarizer-agent

      dependsOn: [research]

      inputs:

        prompt: "Summarize these findings: {{ .steps.research.output }}"

  output: "{{ .steps.summarize.output }}"

The operator executes the DAG, passes outputs between steps, and updates the pipeline status as each step completes. No orchestration code. No custom queue logic.

Try it in five minutes

Prerequisites: Docker, kind, kubectl, Go 1.31+

git clone https://github.com/agentops-io/agentops-operator.git

cd agentops-operator

make dev ANTHROPIC_API_KEY=sk-ant-...

This creates a kind cluster, builds the images, deploys Redis and the operator, and creates the API key secret — all in one command. When it finishes:

kubectl apply -f config/samples/agentops_v1alpha1_agentdeployment.yaml

kubectl get agdep -w

To test the LLM actually responding, submit a task via Redis:

kubectl exec -it -n agent-infra redis-0 -- \

  redis-cli XADD agent-tasks '*' prompt "What is the capital of France? One sentence."



kubectl exec -it -n agent-infra redis-0 -- \

  redis-cli XREAD COUNT 10 STREAMS agent-tasks-results 0

# "The capital of France is Paris."

Tear down:

make dev-down

Who this is for

This isn't for solo developers experimenting with LLMs like Claude Code or a simple API call will serve you better.

This is for engineering teams running agents in production where the problems are operational:

• Platform engineers who need a standard way to manage agents across multiple teams without introducing a parallel management layer


• Team leads who want cost controls enforced at the infrastructure level, not by developer discipline


• SREs who need agents to participate in standard alerting and on-call workflows


• Security and compliance teams who need audit trails and RBAC on agent configuration

If your team is already running Kubernetes, your platform team already knows the tooling. agentops-operator plugs agents into that existing system.

What's next

The project is early — v0.0.1, single contributor, Apache 2.0. The core works: AgentDeployment, AgentService, AgentConfig, and AgentPipeline are all functional.

On the roadmap:

• Parallel step execution in pipelines


• KEDA-based autoscaling on queue depth (because CPU is the wrong signal for LLM workloads)


• Multi-model support (OpenAI, Gemini)


• Helm chart

If this is solving a problem you have, contributions are welcome. Open an issue, read CONTRIBUTING.md, and jump in.

GitHub: agentops-io/agentops-operator

Docs: agentops-io.com