Building an Incident Triage Agent with Full Observability in SigNoz

The problem: AI agents are black boxes

AI agents chain LLM calls and tool calls together to make decisions. But

when something goes wrong — a slow response, an unexpected answer, a

runaway cost — you're stuck guessing. You can't debug what you can't see.

For the "Agents of SigNoz" hackathon, I wanted to build something that

addresses this directly: an AI agent whose entire decision-making process

is fully observable, end to end.

What I built

An Incident Triage Agent — a small AI agent that simulates what an

on-call engineer's assistant might do during a production incident:

1. 
   Plans an investigation approach (LLM call)


2. 
   Searches logs for relevant errors (tool call)


3. 
   Checks a runbook for the matching incident type (tool call)


4. 
   Scores severity based on what it found (tool call)


5. 
   Summarizes a recommended action for the engineer (LLM call)

Every one of these five steps is wrapped in its own OpenTelemetry span, so

a single incident investigation shows up in SigNoz as one connected trace

with all five child spans nested underneath it.

Tech stack

• 
  Python for the agent logic


• 
  Google Gemini API (gemini-3.5-flash) for the two LLM calls


• 
  OpenTelemetry SDK, exporting spans via OTLP/gRPC


• 
  SigNoz, self-hosted locally via the new Foundry installer

How I used SigNoz

Traces

Each span carries custom attributes beyond the defaults — token counts,

latency in seconds, an estimated cost, the severity level the agent

assigned, and even a simple heuristic flag for potential hallucination

(whether the final summary references something that was never in the

logs or runbook). Opening a trace in SigNoz's flame graph view immediately

shows where the time went — in my case, the two LLM calls consistently

took 10-15 seconds each, while all three tool calls combined took under a

second. That's not obvious from reading code; it's obvious from a trace.

Dashboards

I built two dashboards directly from trace data using SigNoz's query

builder:

• 
  Latency by Step — average duration grouped by span name, showing
  exactly which part of the agent's pipeline is the bottleneck


• 
  Severity Distribution — count of incidents grouped by the severity
  the agent assigned, useful for spotting patterns across many runs

Alerts

I set up a threshold alert on the planning LLM call: if agent.llm.plan

takes longer than 15 seconds on average, it fires. This is the kind of

thing you'd actually want in production — a way to know your agent is

degrading before someone notices the user-facing symptom.

What I learned

The most interesting finding, honestly, wasn't about SigNoz — it was

about my own agent. Once I could see the trace breakdown, it became

obvious that essentially all the latency in this pipeline comes from the

two LLM calls, not the tool calls. That's an unglamorous but important

insight: if I wanted to optimize this agent, the tool calls are not

where I'd spend my time.

That's the whole pitch for agent observability — it turns "the agent

felt slow" into "the agent's planning step specifically took 12 seconds,

here's the trace, here's the token count, here's the cost." You can't

have that conversation without instrumentation.

Try it yourself

Full code, setup instructions, and the Foundry deployment config are on

GitHub:

hackathon, Track 01: AI & Agent Observability.