Your multi-agent system processed a customer request. Agent A parsed intent. Agent B fetched data. Agent C analyzed options. Agent D recommended a path. Agent E executed. The customer got a wrong result.
Which agent failed? You open your APM dashboard. You see 50 agents processed messages. You see each agent's internal logs. But you cannot trace the causal chain from Agent A's initial parse to Agent E's final execution. The observability tool sees individual agents. It does not see the conversation that produced the decision.
This is the multi-agent observability gap. Traditional distributed tracing assumes predictable request lifecycles. Agent conversations are non-deterministic, recursive, and produce unbounded span trees. MLflow's 2026 analysis ranked "message-bus visibility" as the #1 missing capability in multi-agent tracing tools.
Why APM Tools Fail on Multi-Agent Systems
Datadog, New Relic, and Jaeger instrument services. They trace HTTP requests through microservices. Each span has a parent. The tree is bounded. You can read it.
Agent messaging breaks every assumption:
# Why traditional distributed tracing fails on agent messaging
class TraditionalTrace:
"""Assumes: request enters, traverses services, returns. Bounded tree."""
def trace_microservice_call(self):
# Span tree: A -> B -> C -> response
# Depth: 3-5 hops typically
# Deterministic: same input = same path
# Bounded: tree ends when response returns
return {"readable": True, "bounded": True, "deterministic": True}
class AgentConversationTrace:
"""Reality: agents loop, branch, recurse, and the tree is unbounded."""
def trace_agent_conversation(self):
# Agent A sends to B. B asks C for clarification. C asks A.
# A sends updated data to B. B now asks D. D loops back to B twice.
# B sends recommendation to E. E asks B to verify. B confirms.
# E executes. But E also messages F for backup.
trace_properties = {
"depth": "unbounded", # Loops mean infinite depth possible
"deterministic": False, # Same input may take different paths
"readable": False, # 200 spans in one trace = unreadable
"tree_structure": "DAG", # Not a tree. Directed acyclic graph (maybe cyclic)
"span_count": "50-500", # Per conversation, not per request
"parent_ambiguity": True, # Message has multiple causal predecessors
}
# APM dashboard shows: 200 spans, 50 agents, duration: 4.2s
# Human sees: chaos
# Question "why did Agent E output X?" requires:
# 1. Which messages influenced E's decision?
# 2. What was the content of those messages?
# 3. Which agent's output was wrong upstream?
# 4. Was the wrong output caused by bad input from further upstream?
# APM cannot answer any of these without MESSAGE-LEVEL causality tracking
return trace_properties
# The gap:
# APM traces: "Agent B received a request and responded in 200ms"
# What you need: "Agent B's response was based on messages from A and C,
# where C's data was stale because C cached a value from 30min ago"
# Only message-level tracing provides this causality chain.
Message-Level Trace Context: The Missing Primitive
The solution is not better APM. It is trace context propagation at the message layer itself. Every message carries: where it came from, what caused it, and what decision chain it belongs to.
// Message-level observability with rosud-call
import { RosudCall, TraceContext } from 'rosud-call';
const channel = new RosudCall({
agentId: 'orchestrator-prod',
network: 'base-mainnet',
observability: {
// W3C Trace Context propagation on every message
tracing: {
enabled: true,
propagation: 'w3c-tracecontext', // Standard, works with any APM
// Message-specific extensions
causalityTracking: true, // Track which messages caused which
decisionChainId: true, // Group related messages into decision chains
contentHashing: true, // Hash message content for audit without storing plaintext
// Automatic span creation per message
spanPerMessage: true,
spanAttributes: [
'message.type', // request | response | notification
'message.category', // data | analysis | recommendation | execution
'message.payment_intent', // Does this message carry payment context?
'peer.trust_score', // Trust level of sender at time of receipt
'decision.chain_id' // Which decision chain does this belong to?
]
},
// Causality graph (not just span tree)
causality: {
// Each message declares its causal predecessors
predecessorTracking: true,
// "This message was caused by messages X, Y, and Z"
maxPredecessors: 5,
// Enables: "trace backward from failure to root cause"
backwardTracing: true
}
}
});
// Sending a message automatically propagates trace context:
const msg = await channel.sendMessage({
to: 'analysis-agent',
message: { parts: [{ kind: 'text', text: 'Analyze EURC liquidity for settlement' }] },
// Trace context automatically attached:
// traceparent: 00-{traceId}-{spanId}-01
// causedBy: [msg-id-from-data-agent] (causal predecessor)
// decisionChainId: 'dc-2026-07-06-settlement-path'
});
// When the analysis agent responds, its response carries:
// - Same traceId (belongs to same distributed trace)
// - New spanId (its own processing)
// - causedBy: [msg.id] (this message caused by our request)
// - decisionChainId: same (same decision chain)
// Failure investigation becomes:
const failureTrace = await channel.traceDecisionChain('dc-2026-07-06-settlement-path');
// Returns: ordered list of all messages in this decision chain
// With causality arrows: "A caused B, B+C caused D, D caused E"
// Root cause: the message where output diverged from expected
Backward Tracing: From Failure to Root Cause
When Agent E produces a wrong output, you need to trace backward through the message chain to find where the error originated. Traditional APM cannot do this because it traces forward (request to response). Agent failures require backward tracing (output to cause):
// Backward tracing from failure to root cause
const investigation = await channel.investigateFailure({
failedOutput: {
agentId: 'executor-agent',
messageId: 'msg-final-output-wrong',
timestamp: '2026-07-06T05:00:00Z',
expectedCategory: 'settlement_in_DE',
actualCategory: 'settlement_in_FR' // Wrong!
}
});
// Backward trace result:
console.log(investigation);
// {
// rootCause: {
// messageId: 'msg-data-agent-response-3',
// agentId: 'data-agent-eu',
// issue: 'Stale cache: returned yesterday liquidity data',
// timestamp: '2026-07-06T04:59:52Z',
// dataAge: '18h (cache TTL was 24h, data was from yesterday 11:00)',
// impact: 'Analysis agent saw DE liquidity as lower than actual'
// },
// causalChain: [
// { step: 1, agent: 'data-agent', msg: 'Returned stale DE liquidity: 0.8M (actual: 2.1M)' },
// { step: 2, agent: 'analysis-agent', msg: 'Recommended FR (appeared deeper: 1.2M vs stale 0.8M)' },
// { step: 3, agent: 'executor-agent', msg: 'Executed settlement in FR based on recommendation' }
// ],
// timeToRootCause: '< 1 second', // vs hours of manual log correlation
//
// // Without message-level tracing:
// // timeToRootCause: '2-4 hours of manual investigation'
// // because: APM shows each agent "worked correctly" given its input
// // the error is in the MESSAGE CONTENT, not the agent PROCESSING
// }
Why Observability Must Be in the Messaging Layer
Three reasons observability cannot be bolted on as a separate tool:
Context propagation requires message awareness: Trace IDs must travel inside messages, not alongside them. A separate tool cannot inject trace context into messages it does not control.
Causality requires message-level knowledge: Knowing that "message B was caused by messages A and C" requires the messaging layer to track predecessors. External tools can only correlate by timestamp (unreliable).
Payment-bearing messages need enhanced tracing: Messages that carry payment intent need richer observability (amount, delegation reference, settlement status). Only the messaging layer knows which messages carry payment context.
SOCIAL SHARE CARD GENERATOR