Multi-agent systems in production experience issues in ways that traditional monitoring misses. For example, the agent can’t invoke its foundation model (FM) and returns an empty response. This could be because of a missing AWS Identity and Access Management (IAM) permission on an agent’s execution role that doesn’t throw a 500 error. A supervisor agent with a poorly scoped prompt doesn’t increase error rates but instead starts routing 20 percent of requests to the unintended specialist while the infrastructure metrics stay green.
Infrastructure monitoring and agent effectiveness monitoring require different approaches. for continuous agent quality assessment and provides API access to foundation models from leading AI companies including Anthropic, Meta, Mistral, and Amazon. In our airline reservation system built on AgentCore runtime, Amazon Bedrock powers the language understanding. For model availability by AWS Region, refer to Supported models by AWS Region in handles agent orchestration and manages interaction lifecycles, with built-in observability through OpenTelemetry instrumentation.
With the is a quality assessment framework integrated into AgentCore runtime. It continuously scores agent interactions using is an autonomous investigation tool that acts as an on-call engineer for your infrastructure. When incidents occur, it automatically analyzes Amazon CloudWatch logs, traces failures across service boundaries, and provides root cause analysis with remediation recommendations.
is an open source observability framework that provides standardized instrumentation for collecting traces, metrics, and logs. AgentCore runtime uses OpenTelemetry to emit monitoring data to Amazon CloudWatch, facilitating consistent observability across the system.
Figure 1: Swarm multi-agent pattern
Dual monitoring architecture
The following diagram shows how these components connect.
that connects through Amazon Bedrock AgentCore Identity, a capability of Amazon Bedrock AgentCore, to Amazon Bedrock AgentCore runtime, where the four-agent swarm handles user requests. AWS Amplify hosts the conversational interface, . We used the (FAST) as reference.
To use Amazon Bedrock AgentCore Evaluations, you need:
AgentCore CLI (
Workflow diagram for setting up the online evaluation configuration
Why not more metrics?
Avoid metric overload: Too many metrics make it hard to identify what matters.
Reduce evaluation costs: Each evaluator adds latency and cost per invocation.
Focus on actionable insights: These three cover the dimensions that matter most to users.
Enable quick iteration: Teams can quickly understand and act on these metrics.
Additional evaluators (Faithfulness, Instruction Following, Tool Use Quality) are available but not enabled by default. Teams can add them based on their specific needs.
Figure 3: Evaluation dashboard displaying average scores across sessions with distribution breakdown by score range (0.0–1.0)
The evaluation dashboard transforms raw Amazon CloudWatch logs and OpenTelemetry traces into an actionable view of how your agent is actually performing (Figure 3). Instead of sifting through thousands of JSON log entries across multiple log groups to piece together what happened in a single session, the dashboard surfaces session timelines, span hierarchies, and evaluation scores in a visual interface. You can filter sessions by date range and drill into individual traces to see exactly where an agent spent time or encountered errors. You can also run on-demand evaluations against specific sessions with built-in or custom evaluators.
Responsible AI safeguards
Evaluators like Correctness and Faithfulness catch hallucinated or inaccurate outputs after the fact, but because online evaluation runs asynchronously on a sample of sessions, a problematic response can still reach the user before it’s scored. For production agent systems,
Figure 4: Viewing trace and span data and performing on-demand evaluation against a session’s trace, spans, and tool calls
Workflow for on-demand evaluation:
On-demand evaluation follows a synchronous workflow where you request evaluation of a specific session and receive immediate results with scores and explanations.
Figure 5: AI pattern analysis based on the low-scoring sessions
Figure 7: The AWS DevOps Agent UI showing the topology graph construction and Amazon CloudWatch log analysis beginning simultaneously
Then, by examining the errors that occurred across the invocation chain, the AWS DevOps Agent identified the root cause.
Figure 9: The AWS DevOps Agent UI showing the prevention tab with specific, actionable remediation steps and a high confidence rating
At a high level, recommended actions were to add the required Amazon Bedrock permissions to the particular execution role, and scope permissions to the specific foundation model resource in use. These provide the user with things to think about for future development to help prevent failures in this paradigm.
The AWS DevOps Agent improves developer efficiency by automating failure diagnosis across the multi-agent system. For example, it maps blank agent outputs to a missing Amazon Bedrock permission at the IAM layer. The AWS DevOps Agent coordinates between Amazon Bedrock AgentCore runtime, Amazon Bedrock, AWS IAM execution roles, and Amazon CloudWatch Logs to correlate automatically with a failure. Its ability to pattern-match across the entire multi-agent call flow catches errors that would otherwise go unnoticed.
Beyond IAM permission issues, the AWS DevOps Agent is designed to handle the full range of complex failure modes common in multi-agent systems. Such examples include:
Model throttling under load: When subagents scale up and hit Amazon Bedrock token-per-minute limits, individual subagents begin failing intermittently. The AWS DevOps Agent correlates throttling metrics with agent invocation timelines to identify which model, Region, and traffic spike caused the issue.
Tool call failures masked by retry logic: If a subagent’s tool integration begins returning errors and the Supervisor Agent silently reroutes, the AWS DevOps Agent surfaces downstream tool errors from logs and connects them to degraded user experience.
Memory and context issues: If an agent loses access to its memory resource mid-session, responses become incoherent rather than failing outright. The AWS DevOps Agent detects anomalies in response quality patterns and correlates them with configuration changes.
Cross-agent communication breakdowns: When handoffs between agents fail because of network issues or authentication problems, the AWS DevOps Agent tracks the complete handoff chain and identifies exactly where communication broke down.
Note: Before adopting this architecture, keep the following in mind:
LLM-as-judge reliability: AgentCore Evaluations uses LLM-based scoring, which lacks ground truth. Treat scores as signals, not absolute measures. Calibrate evaluators with subject matter experts to align automated judgments with human expectations in your domain.
Service maturity: AWS DevOps Agent is actively evolving. Currently, webhook credential generation is done through the console, but Agent Space creation and management can be automated with the .
Latency trade-offs: Online evaluation adds processing overhead. Lower sampling rates reduce this overhead but may miss edge cases. Tune your sampling rate based on your traffic volume and how comprehensively you need to cover the interaction space.
Security: Careful IAM policy configuration is required. The AWS DevOps Agent needs broad read access to logs and metrics, scope permissions appropriately while making sure it has access to everything needed to trace cross-service failures.
Responsible AI controls: For production deployments, complement evaluation with to demo our dual-layer monitoring architecture for multi-agent systems.
Explore for building multi-agent systems.
Review the τ-Bench benchmark for evaluating agent architectures. tau-Bench includes a tau-airline domain that benchmarks exactly the kind of tool-calling and policy-following behaviors our system requires, making it a natural fit for validating quality improvements driven by AgentCore Evaluations.
About the authors
Vollständiger Original-Bericht
Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf aws.amazon.com.
SOCIAL SHARE CARD GENERATOR