Tool Definition Drift: When Your Agent's Toolset Outgrows Its Prompt

Set an alert: any tool with zero calls over the last 1,000 requests gets flagged. A human reviews it and decides whether to delete it from the registry or rewrite its description so the model has a reason to pick it. Either path works; ignoring the orphan does not.

Why descriptions stop matching after a while

Tool descriptions rot the same way comments rot. The first version was hand-written by the engineer who added the tool. They knew the call sites, the failure modes, and the right vocabulary, so they wrote a clear paragraph. The second version was added in a hurry by someone fixing a different bug. They copy-pasted the first description and changed a word. By the tenth tool, the descriptions are inconsistent in tense, length, and terminology. Two tools have a <example> block, six have none, the rest have a half-finished one.

The model reads all of them. It treats the inconsistency as signal. A long, detailed description outranks a short one, regardless of which tool is actually right for the request. A description that uses the same vocabulary as the user's question outranks a description that does not, regardless of which tool is right. The ranking has shifted away from tool capability and onto the prose quality of whoever wrote the description on a Tuesday.

Two fixes work. The first is a description style guide that every new tool has to match (length range, required sections, vocabulary). The second is to skip the prose and generate the description from the schema directly.

Auto-generating descriptions from JSON Schema

If the tool's input schema is rich enough, the description writes itself. You walk the JSON Schema, pull the title, the parameter names and descriptions, and any examples, and template them into a deterministic format. Every tool ends up with the same shape, which kills the prose-quality bias.

def describe_tool(tool: dict) -> str:

    schema = tool["input_schema"]

    name = tool["name"]

    purpose = schema.get("description", "").strip()

    props = schema.get("properties", {})

    required = set(schema.get("required", []))



    param_lines = []

    for key, spec in props.items():

        marker = "required" if key in required else "optional"

        desc = spec.get("description", "").strip()

        param_lines.append(

            f"  - {key} ({marker}): {desc}"

        )



    examples = schema.get("examples", [])

    example_block = ""

    if examples:

        example_block = "\nExample input:\n" + "\n".join(

            f"  {e}" for e in examples[:1]

        )



    return (

        f"Tool: {name}\n"

        f"Purpose: {purpose}\n"

        f"Parameters:\n"

        + "\n".join(param_lines)

        + example_block

    )

Now your tool registry stores the schema. The system prompt is generated at request time from the schemas, with a one-line preamble per tool. If you add a parameter to search_customers, the description regenerates. The prompt and the tools cannot drift, because there is only one source.

You pay a small cost: the auto-generated descriptions are blander than a hand-written one. A hand-written description can say "use this for cross-team docs only, not the legal corpus", which the schema cannot express. The trade is consistency for craft. Hand-writing wins under ten tools. Somewhere between ten and twenty, the consistency dividend starts beating individual craft, and past twenty, generation pulls ahead by a wide margin.

The tool-router pattern

If your toolset is large enough that the schema-vs-prompt diff keeps growing, the next move is to stop showing every tool to the model. A tool router is a small upstream classifier (an embedding-similarity match, or a cheap-model classification) that picks 3 to 5 candidate tools per request and only those go into the prompt.

The shape:

def select_tools(query: str, registry):

    # Pre-computed at registry load time.

    embeddings = [t["embedding"] for t in registry]

    query_emb = embed(query)

    scored = [

        (cosine(query_emb, e), t)

        for e, t in zip(embeddings, registry)

    ]

    scored.sort(reverse=True)

    return [t for _, t in scored[:5]]





def call_agent(query: str, registry):

    selected = select_tools(query, registry)

    return client.messages.create(

        model="claude-opus-4-5-20250929",

        max_tokens=2048,

        tools=selected,

        messages=[{"role": "user", "content": query}],

    )

The model now sees the 5 tools that look most relevant to the query, not all 28. Hallucination drops because the prompt is shorter and more focused, and orphan tools stop paying token rent on every request. Ambiguous selection drops too: the router does the first cut, leaving only tools that pass a similarity threshold.

The cost is misroute risk. The router excludes the right tool from the candidate set and the model has no way to recover. Mitigate it two ways. First, evaluate the router with a held-out set of (query, expected_tool) pairs and watch top-5 recall. Second, keep a "fallback" tool always in the candidate set that lets the model say "none of these match, escalate." A span attribute carrying the selected tool names per request lets you spot the misroutes in production.

What to do with this on Monday

A short sequence to run before the week ends.

Run the schema-vs-prompt diff on your current agent in CI. If the diff returns anything in in_prompt_only, ship a fix today; those names are actively pulling the model toward a tool that does not exist. Then build the coverage histogram off your last 1,000 traces. Anything in the registry with zero calls is either dead code or has a description the model never picks, so decide today which it is, and delete or rewrite accordingly.

Once the current state is clean, put the next checkpoint on the calendar. Tool registries grow whether you watch them or not, and the drift is silent until a user complains. A two-hour audit every quarter beats an outage when the model invents a tool it heard about three releases ago.

The prompt is part of the toolset. Keep them in the same source, regenerate one from the other where you can, and watch the drift number rather than waiting for a hallucinated tool name to surface it for you.

If this was useful