Distributed tracing for multi-agent systems with OpenTelemetry

In this article, we'll discuss how distributed observability works for multi-agent systems, how we can propagate context across agents within our workflow, dive a little deeper into the OpenTelemetry Semantic Conventions for GenAI, and how we can implement observability in our agents and Microsoft Foundry so that we have a unified view across our agents.

If you want to take a look at the code while reading the article, you can view the complete sample to provide consistency across tools and integrations. Within Foundry, tracing captures information such as user inputs and agent outputs, tool usage, token consumption, and time signals such as duration and latency.

There are a couple of OpenTelemetry concepts we should discuss before moving on. Traces 'trace' the journey of a request or workflow through your application by recording events and state changes (which can include function calls, and system events). Spans are the building blocks of traces. They represent a single operation within a trace, and capture start and end times, attributes, and can be nested to show hierarchical relationships so we can see the full call stack and sequence of operations. Attributes are essentially key-value pairs attached to traces and spans, and provide contextual metadata.

How does it work?

Propagating context across agents

For multi-agentic systems, span context contains both a trace ID and span ID. The span context gets propagated across agent boundaries using the , which is a key-value store which lets you propagate any data we like alongside context. The trace is created in the with get_tracer().start_as_current_span() statement. as_current means that the span is created and it gets installed in the context. Any span started inside that block becomes a child of root, which uses the same trace ID, but its own span ID.

The SpanKind.SERVER value for the kind parameter starts the SERVER span deliberately. Without this, Application Insights has nothing to use as the start of a transaction, so we need to set this in order to see the end-to-end view.

In order to give our processor a rule for what to copy, we can provide a BAGGAGE_PREFIX namespace so that our key id becomes drill.id like so:

BAGGAGE_PREFIX = "drill."

Next, we need to set the Baggage. In OTel, contexts are immutable, so we use the set_baggage method to return a new context rather than modifying each one:

@contextmanager

def drill_context(**entries: str) -> Iterator[None]:

    ctx = context.get_current()

    for key, value in entries.items():

        ctx = baggage.set_baggage(f"{BAGGAGE_PREFIX}{key}", value, context=ctx)

    token = context.attach(ctx)

    try:

        yield

    finally:

        context.detach(token)

Baggage lives in the context, while attributes live in the spans. To bring them together, we need to use a span processor to hook the start of the span, and then use a BaggageSpanProcessor to copy the matching baggage across as attributes:

def _install_baggage_processor() -> bool:

    provider = trace.get_tracer_provider()

    add_span_processor = getattr(provider, "add_span_processor", None)

    if add_span_processor is None:

        return False

    add_span_processor(BaggageSpanProcessor(lambda key: key.startswith(BAGGAGE_PREFIX)))

    return True

The GenAI semantic conventions

The OpenTelemetry semantic conventions define a common set of semantic attributes which provide meaning to data when collecting, producing and consuming it. As far as agents are concerned, invoke_agent, chat, and execute_tool automatically. However, when we need to trace when the Commander agent hands work to our specialist, we need to implement that ourselves.

This is what happens inside _delegate(), the function our asyncio.gather calls once per specialist:

with get_tracer().start_as_current_span(

    "agent_to_agent_interaction",

    kind=SpanKind.CLIENT,

    attributes={

        "gen_ai.operation.name": "agent_to_agent_interaction",

        "gen_ai.agent.name": agent.name,

        "game_day.from_agent": COMMANDER_NAME,

        "game_day.to_agent": agent.name,

        "game_day.role": role.value,

    },

) as span:

    response = await agent.run(f"{brief}\n\n{SPECIALIST_ANGLE[role]}")

Because agent.run() sits inside this with block, the invoke_agent span that Agent Framework opens becomes a child of our hand-off span. That gives us the three levels we'll see in the portal later: the drill, the hand-off, then the specialist's own agent, chat and tool spans. Without it we'd have three invoke_agent spans sharing a parent and no record of who handed work to whom.

OpenTelemetry with Azure Monitor

Azure Monitor supports , which is a client that bundles all open-source and Microsoft components required for a full integrated experience with Azure Monitor for both AI Agents and normal applications.

To use OpenTelemetry in Python, we need to install the following . This sample is just using synthetic data to show what's possible. If you want to keep user prompts out of Application Insights, set this to false.

Building our observable multi-agent system

As I mentioned earlier, the Microsoft Agent Framework provides some of this out of the box. Once we've configured our instrumentation, these spans appear without any extra effort on our part. What's lacking is what choices our agents have made.

If we wanted to see what tool calls our agent makes, we want to be able to see that within our traces. For example, for our Logging Agent, we can configure this like so:

@tool(approval_mode="never_require")

async def query_logs(

    service: Annotated[str, Field(description="Service name from the service map, for example 'checkout-api'.")],

    min_level: Annotated[str, Field(description="Lowest severity to return: info, warn or error.")] = "warn",

) -> str:

    """Read recent log lines for one service in the incident window."""

    key = _clean(service).lower()

    level = _clean(min_level).lower()

    _count("query_logs", key)



    span = trace.get_current_span()

    span.set_attribute("game_day.tool.service", key)

    span.set_attribute("game_day.tool.min_level", level)



    ...

    span.set_attribute("game_day.tool.result_count", len(lines))

    return _ok(service=key, min_level=level, lines=lines)

Let's break this down:

• 
  @tool registers the tool as a function within the Microsoft Agent Framework.


• The _clean() method trims and caps every argument. Our tool arguments come from a model, so it's best to treat this as untrusted input.


• The trace.get_current_span() method grabs the execute_tool span that is already open. We're not creating a new span, we are adding to the one that already exists to get that unified span.


• The game_day.tool.* attributes record what tool the agent invoked, and what it got back.

Without those attributes, we wouldn't be able to see that the Logs agent looked at the wrong service, or that a query returned 0 results but it did some work anyway.

Seeing it work

Let's see this in action. We can run one of the drills like so:

$ python demo.py

This produces the following output:

[1] INC-A1C212  checkout latency spike     normal  sev1      48.5 s  drill 1b1b480ad6e2

    logs       24.9 s  Checkout-api p99 latency and error spike caused by payments-gateway connection-pool exhaustion leading to timeouts and a circuit-breaker opening.

    metrics    24.4 s  ~09:25 spike in checkout-api p99 and errors, traced to payments-gateway timeouts after a 09:20 deploy that reduced its connection pool.

    runbook    12.5 s  RB-014 (Downstream connection pool exhaustion) selected; confirm pool.max vs prior release and restore it if it was lowered.

    commander  routed to grok-4-1-fast-reasoning

    tool calls  lookup_runbook 1  query_logs 4  query_metrics 6    total 11



     - ~09:20: Deploy to payments-gateway reduced connection pool max from 200 to 20 (Metrics).

    - ~09:25: p99 latency jumps in payments-gateway (90ms to 7.66s) and checkout-api (~360ms to 1.9s then 7.9s), with error rates rising to 17% and 15% respectively; request volumes flat then falling (Metrics).

    - payments-gateway connection pool saturated: in_use=20 max=20 waiters=61, then pool acquire timed out after 8000ms with waiters=112 (Logs).

    - checkout-api slow downstream calls to payments-gateway: duration_ms=6902 (Logs).

    - checkout-api upstream timeouts on payments-gateway after 8000ms (Logs).

    - Circuit breaker opened on payments-gateway dependency after 23 failures (Logs).



    RB-014 "Downstream connection pool exhaustion" matches symptoms of latency step change, flat-then-falling volume, and upstream timeouts. Confirm current pool.max value against the previous release; since metrics evidence shows it was reduced from 200 to 20 by the 09:20 deploy, raise it back to 200 or roll back the release. Monitor p99 latency, error rates, and pool metrics post-mitigation to confirm recovery.



    SEVERITY: sev1

The header line shows the root span duration (48.5s). All 3 agents ran concurrently and have returned their findings back to, or reported back to, the incident commander. We can see that the Commander agent uses grok-4-1-fast-reasoning to produce the timeline and a mitigation path.

Within the tool calls, we can see how many calls each agent made to various tools, along with a total.

Within the classic Foundry portal, we can see the entire game_day_drill trace in the

Taking a closer look at this, we can see that the child spans all use the same trace_id, so looking at the metadata of game_day_logs, it uses a new span_id, but has the same trace_id:

using that trace_id:

We can also view custom properties:

where we can also view operational metrics from our agents, such as Agent runs, tool calls, models used and importantly, token consumption by model and input vs output tokens.

.

If you have any questions about this, please feel free to reach out to me on BlueSky!

Until next time, Happy coding! 🤓🖥️