Tool Search for Agents: Dynamic Tool Loading When You Have 50+ Tools
Stuffing every tool definition into the prompt burns tokens and makes models pick the wrong tool. Here is how deferred tool loading and tool search work, how to build your own retrieval layer, and how to measure whether it helps.
You connect three MCP servers to your agent. Each exposes 15 to 30 tools. Suddenly every request carries 60+ JSON schemas before the user has said a word. Two things happen: the prompt gets expensive, and the model starts picking search_issues when it should call search_docs. Tool definitions are the quiet tax on every agent that grows past a handful of capabilities.
The fix gaining traction is tool search: don't load every tool up front, load a small set and let the model (or a retriever) find the rest on demand. Anthropic's API and Claude Code ship this natively (check the tool search docs for current model support and syntax), and Spring AI and others have similar patterns. This post explains why it works, how to build it yourself, and how to know if it helped.
The problem: tool definitions are not free
Every tool you pass is serialised into the context: name, description, and a JSON schema for the parameters. A modest tool costs 200 to 500 tokens. Well-documented tools with enums and nested objects cost over 1,000. Do the arithmetic:
| Tools loaded | Avg tokens per tool | Prompt overhead per request |
|---|---|---|
| 10 | 350 | ~3.5k |
| 40 | 350 | ~14k |
| 100 | 450 | ~45k |
That overhead is paid on every turn of every conversation, and it competes with the context you actually care about. Prompt caching softens the cost but does not remove the second problem: accuracy. With dozens of similarly named tools, selection errors rise. Models confuse near-duplicates (create_ticket vs create_task), and long tool lists dilute attention. Teams routinely report tool-selection accuracy dropping as catalogs grow; measure it on your own set rather than trusting a number from a blog post, including this one.
How tool search works
The pattern has three parts:
- A small always-loaded set. The handful of tools used in most conversations (say, 3 to 5), plus a search tool.
- A deferred catalog. Everything else is registered by name and description, but full schemas are not sent to the model.
- A discovery step. The model calls the search tool with a query (or a retriever runs on the user message). Matching tool definitions are then injected into context, and the model calls them normally.
In Anthropic's implementation, you mark tools as deferred (a defer_loading flag on the tool definition) and include a tool search tool, either regex/BM25 based or a custom one that you implement. Because the exact field names and beta status change, verify them against the official docs before copying anything. The concept is portable, so the rest of this post uses a provider-agnostic implementation.
Build it yourself: a minimal tool registry with retrieval
You can implement the same pattern for any model that supports function calling. The core is a registry that stores tool specs and returns the top-k for a query.
from dataclasses import dataclass, field
import math, re
from collections import Counter
@dataclass
class Tool:
name: str
description: str
schema: dict # JSON schema for parameters
always_load: bool = False
tags: list[str] = field(default_factory=list)
def tokenize(text: str) -> list[str]:
return re.findall(r"[a-z0-9]+", text.lower().replace("_", " "))
class ToolRegistry:
def __init__(self, tools: list[Tool]):
self.tools = {t.name: t for t in tools}
self.docs = {t.name: tokenize(f"{t.name} {t.description} {' '.join(t.tags)}") for t in tools}
n = len(self.docs)
df = Counter(tok for doc in self.docs.values() for tok in set(doc))
self.idf = {tok: math.log(1 + n / c) for tok, c in df.items()}
def search(self, query: str, k: int = 5) -> list[Tool]:
q = tokenize(query)
scores = {}
for name, doc in self.docs.items():
tf = Counter(doc)
scores[name] = sum(self.idf.get(t, 0) * tf[t] / (tf[t] + 1.2) for t in q)
ranked = sorted(scores, key=scores.get, reverse=True)
return [self.tools[n] for n in ranked[:k] if scores[n] > 0]
def base_tools(self) -> list[Tool]:
return [t for t in self.tools.values() if t.always_load]
This is plain BM25-ish lexical scoring. It is fast, needs no embeddings, and works surprisingly well because tool names and descriptions are keyword-dense. Upgrade to embeddings (or hybrid) only if your eval shows lexical misses on paraphrased queries.
Wire it into the agent loop
The search tool itself is just another function. When called, it returns tool definitions, and the loop adds them to the tools list for the next turn.
import anthropic
client = anthropic.Anthropic()
registry = build_registry() # your Tool list
SEARCH_TOOL = {
"name": "search_tools",
"description": "Find tools by capability. Call this before saying you cannot do something. "
"Describe the action, e.g. 'create a calendar event' or 'query billing invoices'.",
"input_schema": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
}
def to_api(tool: Tool) -> dict:
return {"name": tool.name, "description": tool.description, "input_schema": tool.schema}
def run(user_msg: str, model: str, max_turns: int = 8):
active = {t.name: t for t in registry.base_tools()}
messages = [{"role": "user", "content": user_msg}]
for _ in range(max_turns):
resp = client.messages.create(
model=model,
max_tokens=2048,
tools=[SEARCH_TOOL] + [to_api(t) for t in active.values()],
messages=messages,
)
messages.append({"role": "assistant", "content": resp.content})
calls = [b for b in resp.content if b.type == "tool_use"]
if not calls:
return resp
results = []
for call in calls:
if call.name == "search_tools":
found = registry.search(call.input["query"], k=5)
for t in found:
active[t.name] = t # becomes callable next turn
text = "\n".join(f"- {t.name}: {t.description}" for t in found) or "No matching tools."
results.append({"type": "tool_result", "tool_use_id": call.id, "content": text})
else:
out = execute_tool(call.name, call.input) # your dispatcher
results.append({"type": "tool_result", "tool_use_id": call.id, "content": out})
messages.append({"role": "user", "content": results})
Pass a current model ID from the docs for model. Two details matter here.
Tool-list changes invalidate your prompt cache. If tools is the first part of the prefix and you mutate it every turn, you lose caching on everything after it. Native implementations avoid this by appending discovered definitions in the message stream rather than rewriting the top-level tool list. If you roll your own, keep the base tool list stable and cap how often the active set changes.
Cap the active set. Without a ceiling, a long conversation re-accumulates every tool and you are back where you started. Evict least-recently-used tools beyond, say, 15.
Choosing an approach
| Approach | Best for | Weakness |
|---|---|---|
| Load everything | Under ~15 tools, short sessions | Cost and accuracy degrade as you grow |
| Native tool search (provider feature) | Large MCP catalogs, least code | Provider-specific, check model support |
| Custom retriever in your loop | Multi-provider, custom ranking, auth-aware filtering | You own the eval and the cache behaviour |
| Router/sub-agents by domain | Clear domain boundaries (billing, support, infra) | More orchestration, handoff losses |
| Retrieve per user message, before the model | Predictable intents, latency-sensitive | Misses tools needed mid-task |
A useful rule: if your tool count fits comfortably under 15 and descriptions are distinct, don't bother. Complexity you don't need is its own failure mode.
Write tool descriptions for retrieval
Search quality is capped by description quality. Treat descriptions as search documents:
- Lead with the verb and object: "Create a Jira issue in a project" beats "Issue creator".
- Include the synonyms users actually say: "ticket, bug, task".
- Name the system: "GitHub", "Stripe", "internal HR portal". Namespaced names (
github_create_issue) help both retrieval and disambiguation. - State when not to use it: "For read-only lookup use
get_issue." - Keep schemas lean. Long enums are the usual token hogs, so move them into a lookup tool if they exceed ~20 values.
Measure it: the eval that tells you if this worked
Don't ship tool search on vibes. Build a small eval set of 50 to 100 tasks, each with the expected tool (or tool sequence), and track:
| Metric | What it tells you |
|---|---|
| Tool recall@k | Did search surface the right tool at all? |
| Selection accuracy | Given the loaded tools, did the model pick correctly? |
| Discovery turns | Extra round trips added by searching |
| Input tokens per task | The savings you were after |
| Task success rate | The only number that ultimately matters |
Run the baseline (load everything) against the dynamic version on the same tasks. Typical outcomes reported by teams: large input-token savings, a small latency cost from the extra search turn, and accuracy that is flat or better on big catalogs but sometimes worse on small ones. If recall@5 is below roughly 95%, fix descriptions or retrieval before blaming the model.
Common pitfalls
- The model never searches. It confidently says "I can't do that." Put an explicit instruction in the system prompt and the search tool description: search before declaring a capability missing.
- Searching for the wrong thing. Queries like "tools" return noise. Show examples of good queries in the search tool description.
- Cache thrash. Rewriting the tools array every turn silently multiplies cost. Check cache hit rates before and after.
- Authorization leaks. Search results must be filtered by the caller's permissions. A tool the user can't run shouldn't be discoverable, and it must still be enforced at execution time.
- Deferred tools with side effects and weak descriptions. A model picking a "close" action from a vague match is worse than a model failing to find it. Pair destructive tools with approval gates.
- No observability. Log every search query, results returned, and whether the model used them. Misses show up here first.
Checklist before you ship
- Fewer than ~5 tools always loaded; the rest deferred
- Descriptions rewritten as search documents, with synonyms and namespaces
- Eval set with expected tools; baseline vs dynamic compared
- Prompt cache hit rate measured after the change
- Active tool set capped with eviction
- Permission filtering applied at search and execution
- Search queries and outcomes logged
Next steps
Start by counting: dump your current tool definitions, measure their token total, and compare against your median user request. If tools are more than 20% of your input tokens, tool search is worth a day of work. Begin with your provider's native option if it supports your model, and fall back to the registry above if you need portability.
Building production agents with large tool catalogs is a core module in our engineering cohort. If you want to go from prototype to reliable agent systems with feedback on your own code, take a look at the Joinloop AI Engineering cohort.
Want to build this yourself?
Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.