We instrumented an AI agent swarm with SigNoz, and its own telemetry told us we were wrong about almost everything

Built for the WeMakeDevs Agents of SigNoz hackathon, July 2026.

Every number on our own landing page is a live ClickHouse query against the trace store. Marketing copy that drifts from the telemetry is impossible by construction.

Everything is served through Hugging Face Inference Providers. There are zero closed-model API calls in the system, which turned out to matter for reasons we did not anticipate (see the provider section below).

The critic is the load-bearing part. Frontend and backend are generated in parallel from the same contract, then an independent model reviews both for contract conformance, security and runtime bugs. Real catches route back to the agent that owns them, that agent patches its own file and the critic re-reviews only the delta. Two regeneration rounds, then it ships with an honest verdict either way.

Why we instrumented before we polished

A multi-agent system fails in ways a single-model tool does not. A call can succeed while producing an unusable artifact. A fallback can rescue a request so smoothly that nobody notices the primary is dead. Latency can triple because one role silently started thinking twice as long. None of that shows up in a request log.

So the very first thing that worked in this project was not code generation. It was a trace.

SigNoz is self-hosted through Foundry, which is a single-config install. Our casting.yaml and casting.yaml.lock are committed to the repo so the deployment is reproducible by anyone, judges included.

Three signals, and what each one carries

Traces. Every model call is a span named llm.<role> carrying GenAI semantic conventions:

span.setAttributes({

  'gen_ai.operation.name': 'chat',

  'gen_ai.request.model': model,

  'gen_ai.usage.input_tokens': usage.prompt_tokens,

  'gen_ai.usage.output_tokens': usage.completion_tokens,

  'devswarm.role': role

});

Two span events do the heavy diagnostic lifting. fallback_promotion records that a primary failed, which model took over and the verbatim reason. critic_catch records every issue the review agent found, with its target and severity. Both are events rather than separate spans on purpose: they belong to the call they describe, and they survive in the trace even when the call ultimately succeeds.

Metrics. Six counters and a histogram, because some questions are time series questions rather than trace questions: devswarm.tokens, devswarm.llm.calls, devswarm.llm.duration, devswarm.fallback.promotions, devswarm.critic.catches, devswarm.generations, devswarm.refinements. Labelled by role, model and outcome.

Logs. Structured records for the things a human reads during an incident: a fallback promoting, a doctor diagnosis, a generation completing with its verdict and catch count. Same resource attributes as the traces, so a log line and a span line up.

The whole trace layer is now extracted into a small library,

One generation as a flame graph. Planner, then frontend and backend in parallel, then the critic.

Command Center. Top row answers "is the swarm healthy", the role-health table answers "which role", while the fallback chart should trend to zero.

The Doctor reporting a healthy swarm. It read 180 minutes of its own traces to say so, and it is honest about sample size: "call volume is very low, so latency figures are not statistically meaningful".

Two hard-won SigNoz API notes, since we lost hours to both:

Alert rules must be created against /api/v2/rules with schemaVersion: v2alpha1, a notificationSettings block and at least one channel. The v1 endpoint accepts the request and returns "alert rule is not valid" with no indication of which field is wrong. Dashboards, by contrast, go to /api/v1/dashboards with a SIGNOZ-API-KEY header and behave exactly as documented.

Also: a cold Docker restart can leave ClickHouse replicas read-only until Keeper reconnects. It usually self-heals within a minute. If it does not, SYSTEM RESTORE REPLICA per table clears it.

Apps that are born observable

Every app the swarm generates ships instrumented. Alongside index.html and server.js, each generated folder gets an otel.mjs bootstrap, a package.json, and a signoz-dashboard.json scoped to that app's own service name. If a SIGNOZ_API_TOKEN is configured, the dashboard is created in SigNoz at generation time, before the user has opened the preview.

So the generated app appears in SigNoz as its own service, with RED metrics and a routes table, seconds after it exists. Twenty four of them are in our instance right now.

Worth saying because people assume otherwise: there is no image model anywhere in this pipeline. The swarm generates 227 inline SVG elements across the 26 apps it has built, an average of 8.7 per app, and every one of them was written as markup by a language model. The Vandercook press in the screenshot above is hand-drawn SVG, not a generated image. The only assets we ever image-generated are DevSwarm's own favicon and social card, which are branding for the tool rather than anything the swarm produces.

Before. Mono labels from a system stack, and card colours the backend invented at random.

Dashboards, alert rules and the Foundry casting.yaml are all in the repo under observability/, so the whole SigNoz side of this is reproducible rather than described.