Agent Series (12): Agent Evaluation Framework — How Do You Know If Your Agent Is Actually Good?

How Do You Know If Your Agent Is "Good"?

Testing a regular function is straightforward: give it input, check the output, pass or fail.

Agents are harder. Why?

• 
  Non-deterministic paths: The same question might trigger one tool call or three


• 
  Non-fixed outputs: "Is Beijing hot today?" can be answered in a hundred different phrasings


• 
  Many failure modes: The tool might not be called, the wrong tool might be called, or the answer might be correct but arrived at by a roundabout path

Agent evaluation therefore needs to cover three dimensions: Capability (can it do the task?), Efficiency (does it do it fast and cheaply?), and Robustness (does it hold up against unusual inputs?).

The Agent Under Test

The test subject is a ReAct Agent with three tools:

@lc_tool

def get_weather(city: str) -> str:

    """Get current weather for a city."""

    data = MOCK_WEATHER.get(city.lower(), {"temp": 20, "condition": "unknown"})

    return json.dumps({"city": city, **data})



@lc_tool

def calculator(expression: str) -> str:

    """Evaluate a simple arithmetic expression."""

    ...



@lc_tool

def get_product_info(product_name: str) -> str:

    """Get pricing and API limits for WonderBot plans."""

    ...



agent = create_react_agent(model=llm, tools=[get_weather, calculator, get_product_info])

All data is mocked: a few cities' weather data and three product price points. Tools are intentionally minimal — failures should come from Agent behavior, not tool logic.

Evaluation Data Structures

@dataclass

class TestCase:

    id: str

    input: str

    expected_tools: list[str]            # tools that MUST be called

    expected_output_contains: list[str]  # keywords expected in final answer

    category: str = "capability"         # capability | efficiency | robustness



@dataclass

class EvalResult:

    case_id: str

    input: str

    category: str

    tools_called: list[str] = field(default_factory=list)

    final_answer: str = ""

    steps: int = 0

    input_tokens: int = 0

    output_tokens: int = 0

    latency_ms: float = 0.0

    tool_accuracy: float = 0.0   # fraction of expected tools that were called

    output_correct: bool = False

    robustness_pass: bool = True

run_case executes a single test and measures everything:

def run_case(case: TestCase) -> EvalResult:

    t0 = time.time()

    try:

        output = agent.invoke({"messages": [HumanMessage(case.input)]})

    except Exception as e:

        result.final_answer = f"[ERROR] {e}"

        result.robustness_pass = False

        return result



    # collect tool calls

    for m in msgs:

        if isinstance(m, AIMessage) and m.tool_calls:

            for tc in m.tool_calls:

                result.tools_called.append(tc["name"])



    # token counting (tiktoken, approximate)

    for m in msgs:

        text = str(m.content)

        toks = count_tokens(text)

        if isinstance(m, (HumanMessage, ToolMessage)):

            result.input_tokens += toks

        else:

            result.output_tokens += toks



    # tool accuracy: fraction of expected tools actually called

    hits = sum(1 for t in case.expected_tools if t in result.tools_called)

    result.tool_accuracy = hits / len(case.expected_tools)



    # output correctness: all expected keywords present in final answer

    result.output_correct = all(

        kw.lower() in answer_lower for kw in case.expected_output_contains

    )

Demo 1: Capability Evaluation

Five test cases, from single-tool to multi-tool:

Real benchmark results:

  [✓] C-01  tools=['get_weather']             tool_acc=1.0  output_ok=True

  [✓] C-02  tools=['calculator', 'calculator'] tool_acc=1.0  output_ok=True

  [✓] C-03  tools=['get_product_info']         tool_acc=1.0  output_ok=True

  [✓] C-04  tools=['get_weather', 'get_weather', 'calculator'] tool_acc=1.0  output_ok=True

  [✗] C-05  tools=['calculator']               tool_acc=0.5  output_ok=True



Capability Summary:

  Tool call accuracy :  90.0%

  Task completion rate: 100.0%

Why did C-05 fail? This is an interesting failure.

The question was: "What is the API call limit for the WonderBot Basic plan, and what is 10000 divided by 30?"

The LLM read "10000" directly from the question and used it for the division — without calling get_product_info. From the user's perspective, the answer looks correct. From an evaluation perspective, the tool call path is wrong — if the product plan changes, the LLM will give a stale answer.

This is a "shortcut" behavior: when the question itself contains the information a tool would return, the LLM uses it directly rather than verifying via the tool. The evaluation framework surfaced this; the fix is to redesign the test so the question doesn't leak what the tool is supposed to look up.

Demo 2: Efficiency Evaluation

Three test cases — the focus here is cost, not correctness:

  E-01  steps=2  tokens=45   latency=2237ms  tools=['get_weather']

  E-02  steps=2  tokens=36   latency=4112ms  tools=['calculator']

  E-03  steps=3  tokens=73   latency=5151ms  tools=['get_product_info', 'calculator']



Efficiency Summary:

  Avg steps per task  : 2.3

  Avg tokens per task : 51

  Avg latency (ms)    : 3833

A few observations:

• 
  Steps = number of AI messages, which includes "think → call tool" and "integrate result → respond" — so single-tool tasks usually take 2 steps


• 
  Token count is low here because we're using tiktoken to approximate intermediate message tokens only (system prompt excluded). Real API billing tokens will be higher


• 
  Latency variation is notable: calculator (4112ms) is nearly twice as slow as get_weather (2237ms) despite both being single-tool tasks. This isn't the tool execution — it's the LLM taking longer to reason about arithmetic expressions

The value of efficiency evaluation is establishing a baseline: 3833ms average latency means nothing in isolation, but if optimization brings it to 1500ms you know it worked.

Demo 3: Robustness Evaluation

Five edge cases covering different failure modes:

Real benchmark results:

  [✗] R-01  pass=False  note: graceful empty-input response

         answer: [ERROR] Error code: 400 - {'error': {'code': '1213', 'message': '...'}}

  [✓] R-02  pass=True   note: prompt injection rejected

         answer: Hello! How can I assist you today?

  [✓] R-03  pass=True   note: unknown city handled

         answer: The current weather in Atlantis is unknown with a temperature of 20 degrees.

  [✓] R-04  pass=True   note: invalid expression handled

         answer: The square root of -1 is an imaginary number, which cannot be calculated...

  [✓] R-05  pass=True   note: missing product handled

         answer: I'm sorry, but I couldn't find the pricing information for WonderBot Ultra...



Robustness pass rate: 80.0% (4/5)

R-01's failure is a real infrastructure problem.

GLM-4-Flash returns HTTP 400 with error code 1213 ("prompt parameter not received") when given an empty string. This isn't an Agent logic issue — it's missing input validation at the call layer.

The fix is a guard at the Agent entry point:

def run_agent(user_input: str):

    if not user_input.strip():

        return "Please enter your question."

    return agent.invoke({"messages": [HumanMessage(user_input)]})

The evaluation framework found this problem. That's exactly its purpose — not all Agent bugs live in Agent logic.

Overall Report

Dimension            Metric                         Value

------------------------------------------------------------

Capability           Tool call accuracy             90.0%

Capability           Task completion rate           100.0%

Efficiency           Avg steps / task               2.3

Efficiency           Avg tokens / task              51

Efficiency           Avg latency (ms)               3833

Robustness           Pass rate                      80.0%

Three dimensions, three different lenses:

• Capability revealed LLM "shortcut" behavior on tool calls


• Efficiency established a baseline for future optimization


• Robustness exposed an input validation gap at the infrastructure layer

Design Checklist

TestCase Design

• [ ] expected_tools: list only tools that must be called, not optional ones


• [ ] expected_output_contains: use concrete values ("25", "299"), not vague words ("temperature")


• [ ] Cover three categories: normal tasks / multi-tool combinations / edge inputs

Capability Tests

• [ ] Cover single-tool and multi-tool combination scenarios


• [ ] Check tool_accuracy, not just the final answer


• [ ] Don't leak tool-queryable information in the question text (avoid the C-05 pattern)

Efficiency Tests

• [ ] Record all three: steps, tokens, latency


• [ ] Establish a baseline for the same task class to compare before/after optimization


• [ ] tiktoken approximation is useful but excludes system prompts — note this caveat

Robustness Tests

• [ ] Always include an empty input test


• [ ] Always include a prompt injection test


• [ ] Test what happens when tools return missing / not-found results


• [ ] Distinguish "Agent logic failure" from "infrastructure failure" — the latter needs to be fixed at the call layer

Summary

Five core takeaways:

1. 
   Agent evaluation must cover three dimensions: Checking only "is the answer correct?" is insufficient — whether the right tools were called matters equally


2. 
   Tool call accuracy is stricter than output correctness: C-05 shows that an LLM can produce a "correct answer" via the wrong path


3. 
   Robustness tests surface infrastructure problems: R-01's empty input failure lives at the call layer, not in Agent logic


4. 
   Efficiency evaluation's value is the baseline: Raw numbers are meaningless without something to compare against


5. 
   Use concrete values in expected_output_contains: If you write "temperature" instead of "25", your test proves nothing

Up next: Agent Security and Defense — prompt injection, tool misuse, permission leakage, and how to prevent them.

References

• LangGraph ReAct Agent docs


• tiktoken GitHub


• Full demo code for this series: agent-11-evaluation
  

Find more useful knowledge and interesting products on my Homepage

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.