How to Test an AI Agent's Tool Selection Without Trusting Its Own Logs

You have built an AI agent harness. It calls tools, routes requests, and returns results. Your team trusts its telemetry to tell you which tool was selected and why.

That trust is a liability.

An agent's own logs are self-reported. They tell you what the agent thinks it did, not what actually happened. A hallucinated tool name, a misrouted parameter, a silent fallback to a different function — none of these surface in the agent's own trace. You need an external witness.

Here is how to build one.

The Problem: Self-Reported Truth Is Not Truth

Most teams validate agent behavior by reading the agent's own output. They check the tool_calls field in the response, match it against an expected schema, and call it done.

This works until it doesn't.

Consider a common failure mode: the agent decides to call search_knowledge_base but the LLM formats the tool name as searchKnowledgeBase. The routing layer silently normalizes it, the call succeeds, and the agent logs search_knowledge_base. Your test passes. The actual execution path was different from what you verified.

Another pattern: the agent selects the correct tool but passes a parameter that the tool silently coerces. A date string gets parsed into a different timezone. A user ID gets truncated. The tool returns a result, the agent logs success, and your test never catches the drift.

The root cause is the same. You are testing the agent's intent, not its execution. Intent is cheap to fake. Execution leaves fingerprints.

The Solution: An External Observer

You need a layer that sits between the agent and the tools it calls. This observer records every invocation — tool name, parameters, response, latency — without the agent knowing it is being watched.

The observer does not trust the agent's logs. It trusts what it sees on the wire.

Here is the architecture at a high level:

1. Intercept every outbound call from the agent to a tool.
2. Record the raw request before any normalization or routing.
3. Compare the recorded call against your expected invocation.
4. Assert on the recorded data, not the agent's summary.

This is not middleware. It is a test harness that wraps the tool-calling layer.

Technical Detail: Building the Observer in Python

I will show you a minimal implementation using Python and a mock tool server. The same pattern works in TypeScript with Playwright's route interception or a custom fetch wrapper.

Start with a simple tool registry. Each tool has a name, a handler, and a schema.

from dataclasses import dataclass, field
from typing import Any, Callable, Dict
import json
import time

@dataclass
class Tool:
    name: str
    handler: Callable
    schema: Dict[str, Any]

class ToolRegistry:
    def __init__(self):
        self._tools: Dict[str, Tool] = {}
        self._invocations: list = []

    def register(self, tool: Tool):
        self._tools[tool.name] = tool

    def call(self, name: str, params: Dict[str, Any]) -> Any:
        # Record the raw invocation before any processing
        invocation = {
            "tool_name": name,
            "params": params,
            "timestamp": time.time(),
            "raw_name": name  # This is what the agent actually sent
        }

        tool = self._tools.get(name)
        if tool is None:
            invocation["error"] = f"Tool '{name}' not found"
            self._invocations.append(invocation)
            raise ValueError(f"Tool '{name}' not found")

        start = time.time()
        try:
            result = tool.handler(**params)
            invocation["result"] = result
            invocation["latency"] = time.time() - start
        except Exception as e:
            invocation["error"] = str(e)
            invocation["latency"] = time.time() - start
            raise
        finally:
            self._invocations.append(invocation)

        return result

    def get_invocations(self) -> list:
        return self._invocations

    def clear(self):
        self._invocations.clear()

The key detail: invocation["raw_name"] captures exactly what the agent sent. No normalization. No aliasing. If the agent sends searchKnowledgeBase, you record searchKnowledgeBase. Your test can then assert that the agent sent the canonical name, not a variant.

Now register a tool and simulate an agent call.

def search_kb(query: str, max_results: int = 5) -> list:
    # Simulated search
    return [{"id": 1, "title": f"Result for {query}"}]

registry = ToolRegistry()
registry.register(Tool(
    name="search_knowledge_base",
    handler=search_kb,
    schema={"query": "string", "max_results": "integer"}
))

# Simulate an agent call with a non-canonical name
try:
    registry.call("searchKnowledgeBase", {"query": "AI testing", "max_results": 3})
except ValueError:
    pass

invocations = registry.get_invocations()
print(invocations[0]["raw_name"])  # "searchKnowledgeBase"

Your test can now assert on raw_name directly.

def test_agent_uses_canonical_tool_name():
    registry.clear()
    # Simulate the agent's decision loop
    agent_decides_to_call("search_knowledge_base", {"query": "testing"})
    invocations = registry.get_invocations()
    assert len(invocations) == 1
    assert invocations[0]["raw_name"] == "search_knowledge_base"

This catches the normalization failure. If the agent sends a variant, the test fails.

Extending the Observer

The same pattern catches parameter drift. Record the raw parameters before the tool handler processes them. If the agent sends a string where an integer is expected, your test sees the raw string.

def test_agent_passes_correct_param_types():
    registry.clear()
    agent_decides_to_call("search_knowledge_base", {"query": "testing", "max_results": "3"})
    invocations = registry.get_invocations()
    params = invocations[0]["params"]
    assert isinstance(params["max_results"], int), "max_results should be int"

The agent's log might show max_results: 3 as an integer because the routing layer coerced it. Your observer shows the raw string. That difference matters when the tool's behavior depends on type.

What This Teaches

The principle is simple: test the boundary, not the summary.

An agent's internal logs are a summary of what it intended. The actual execution happens at the boundary between the agent and the tool. That boundary is where failures live. Normalization, coercion, fallback routing, silent retries — none of these appear in the agent's own trace.

By placing an observer at that boundary, you shift your testing from intent to execution. You stop asking "did the agent think it called the right tool?" and start asking "did the agent actually call the right tool with the right parameters?"

This is not a new idea. It is the same principle that makes contract testing valuable in microservices. You test the API contract, not the service's internal state. The agent is just another service with a particularly unreliable internal narrator.

A Note on Cost

An external observer adds latency and storage. Every invocation gets recorded, serialized, and stored for the duration of the test. In production, you might sample or aggregate. In test, you record everything.

The trade-off is worth it. A single undetected tool misrouting can cascade into hours of debugging. The observer pays for itself the first time it catches a failure the agent's logs missed.

Closing

Your team is probably testing the agent's intent right now. The logs look clean, the traces are green, and the demos work. But the real failures live in the gap between what the agent says it did and what actually happened.

Build an observer. Record the raw invocation. Assert on what you see, not what you are told.

Which of your agent's tool calls have you never actually witnessed?