incident response is becoming an agent review workflow

The least useful sentence during an incident is "the AI says it found the problem."

Great.

What did it look at?

Which logs?

Which metrics?

Which permissions did it use?

What did it rule out?

What action is it recommending, and who is taking responsibility for that action at 2 a.m. when everyone is a little too confident and a little too tired?

This is why I found AWS's recent walkthrough of AWS DevOps Agent diagnosing Amazon EKS control-plane performance issues more interesting than the usual "AI for operations" story. The example is very specific: API server throttling, API Priority and Fairness seat exhaustion, audit logs, CloudWatch metrics, and a recommendation path for remediation.

That specificity matters.

The story is not "AI fixes Kubernetes."

The story is that incident response is becoming a review workflow.

kubernetes incidents are mostly evidence problems

Kubernetes is extremely good at turning one problem into five symptoms.

A deployment slows down. The API server starts throttling. Controllers fall behind. Webhooks add latency. A noisy client hammers the cluster. Someone notices pods are not scheduling quickly. Someone else sees a dashboard that is red but not very helpful. The actual cause is somewhere between application behavior, cluster configuration, cloud limits, and a control plane doing exactly what it was designed to do under pressure.

This is not a place where "summarize the incident" is enough.

The hard part is collecting the right evidence in the right order.

When an engineer investigates an EKS control-plane issue, they are usually stitching together several sources:

• CloudWatch metrics for API server request volume, latency, and throttling


• audit logs showing which clients are making which calls


• Kubernetes events and controller behavior


• API Priority and Fairness signals


• recent deploys, automation jobs, and scripts


• human context that is rarely stored in the same system

The value of an agent is not that it has magical intuition.

The value is that it can do the boring correlation work quickly, consistently, and with enough structure that another person can inspect it.

That last part is the difference between help and risk.

the agent should produce an argument

I do not want an operations agent to produce a vibe.

I want it to produce an argument.

Something like:

"API server requests increased sharply at this time. The increase came mostly from this identity. The requests were concentrated in this verb and resource. APF metrics show this priority level exhausting seats. Audit logs confirm repeated calls from this workload. No matching control-plane health event suggests a regional service issue. The likely cause is this client behavior. Recommended actions are these, with these tradeoffs."

That is reviewable.

It can be wrong, but it is wrong in a way a human can challenge.

This is the shape I like in the AWS example. The DevOps Agent is not just a chatbot sitting beside a dashboard. It is using cloud-native evidence: metrics, logs, audit trails, and cluster-specific context. It can point at the thing it saw.

That changes the human role.

The engineer is no longer starting from a blank page, trying to remember which query caught the same class of issue six months ago. The engineer is reviewing a proposed incident narrative.

Is the evidence complete?

Did the agent overfit to the first suspicious metric?

Did it miss a deploy that happened five minutes earlier?

Is the remediation safe during business hours?

Should we reduce client pressure, tune APF, roll back an automation, or just stop one bad actor before changing cluster-wide policy?

This is a much better use of AI than pretending the human can disappear from the loop.

access is part of the diagnosis

There is a security angle here that is easy to miss because the demo looks operational.

An agent that diagnoses production infrastructure needs access.

Maybe not write access. Hopefully not broad write access by default. But it needs enough read access to inspect logs, metrics, events, identities, configurations, and sometimes sensitive operational details. In Kubernetes, even read-only access can reveal a lot about the system.

So the agent's identity matters.

Which IAM role or access entry did it use? Which clusters can it inspect? Which namespaces? Which CloudWatch log groups? Can it see audit logs? Can it query across accounts? Can it read secrets by accident because someone treated "debugging" as a reason to grant everything?

This is where agentic ops becomes ordinary platform engineering again.

The agent needs least privilege. It needs audit trails. It needs clear boundaries between "observe," "recommend," and "act." It needs different permissions for development, staging, and production. It needs a way to explain which evidence it could not access, because missing evidence is itself important during an incident.

If the agent says "I did not find a bad client," but it could not read the audit log group where the bad client would be visible, that conclusion is not useful.

Confidence without access transparency is theater.

recommendations are not ownership

There is a dangerous version of operations AI where the system jumps from diagnosis to action too quickly.

The agent sees throttling. The agent recommends changing API Priority and Fairness settings. Someone clicks approve because the recommendation sounds plausible and the dashboard is red. The incident improves for ten minutes, then a more important workload is starved because the change moved pressure somewhere else.

Kubernetes has many knobs that are locally reasonable and globally surprising.

This is why I care about the distinction between recommendation and ownership.

An agent can recommend a remediation. It can explain expected effects. It can show blast radius. It can link to the metrics that should improve. It can draft a change. It can even prepare a rollback plan.

But the organization still needs to decide who owns the action.

During an incident, that ownership should be boringly explicit:

• the agent diagnosed this


• this human accepted the diagnosis


• this change was approved


• this system applied it


• these metrics were watched afterward


• this rollback path was available

That sounds bureaucratic until you have to write the postmortem.

Then it sounds like kindness.

the postmortem starts during the incident

The best thing an incident agent can do may not be fixing the issue.

It may be preserving the investigation.

Most incident timelines are reconstructed after the fact from Slack messages, dashboards, terminal history, deploy logs, memory, and a few timestamps everyone argues about later. The incident has already consumed attention. Now the team has to do archaeology.

An agent can help by making the timeline part of the workflow from the beginning.

It can record what evidence was checked, which hypotheses were considered, which ones were rejected, which query showed the turning point, which remediation was chosen, and what happened afterward. Not as a polished executive summary. As operational notes that can survive stress.

That matters because incident response is full of reasonable decisions that look weird out of context.

Why did the team restart that controller?

Why did they pause that automation?

Why did they choose not to change APF during the incident?

Why did they decide the noisy client was the cause and not merely another symptom?

If the agent keeps the evidence trail, the postmortem is less about memory and more about learning.

That is a very practical form of AI assistance.

this is where ops agents should get boring

I am skeptical of any operations AI story that jumps too quickly to autonomy.

Autonomous remediation will happen in narrow places. Some of it already does. Restart the known bad job. Scale the obvious queue worker. Revert the canary that just failed health checks. Block a clearly abusive client. Fine.

But most production incidents are not clean enough to deserve a big red "let the model handle it" button.

The better near-term product is more boring:

• gather evidence faster


• correlate across systems


• preserve the chain of reasoning


• make permissions visible


• propose bounded actions


• require human approval for risky changes


• turn the whole thing into a better postmortem artifact

That is still a big deal.

It means on-call engineers spend less time remembering where the evidence lives and more time deciding what the evidence means. It means new team members can learn from actual incident reasoning, not just final summaries. It means platform teams can encode common diagnostic paths without pretending every incident is identical.

Most importantly, it gives reviewers something concrete.

Not "the agent was helpful."

"Here is the investigation it performed, here is what it concluded, here is who accepted it, and here is what changed."

That is the kind of sentence production systems can live with.

the punchline

AWS DevOps Agent diagnosing EKS control-plane performance issues is a useful signal because it shows where ops AI is probably going.

Not straight to magic remediation.

Toward incident response as a structured, reviewable workflow.

Kubernetes incidents need evidence. Cloud providers already hold much of that evidence in logs, metrics, audit trails, identities, and configuration APIs. Agents can pull those pieces together faster than a tired human jumping between consoles. But the useful output is not a confident answer. It is a diagnosis that can be inspected.

What did the agent see?

What could it access?

What did it miss?

What action did it recommend?

Who decided to act?

If those questions are answered, an agent becomes a second reviewer in the incident process. Not the owner. Not the hero. Not the person carrying the pager.

A reviewer.

That may sound less exciting than autonomous operations.

Good.

Production needs fewer exciting surprises.

references

To test my projects, I use .