Embed, retrieve, let the LLM decide works for about 100 tools. At 1,000 it needs a control plane: hybrid retrieval, hard filters, a learned reranker, progressive disclosure and per-stage evals.
The claim is that a reliable classifier is a pipeline, not a prompt. Embeddings find the neighborhood. Retrieval returns definitions. The LLM compares them. A validation step chooses between a category and an abstention. I agree with every stage.
The ceiling rule in that write-up deserves one extension. If the correct category is not retrieved, the LLM cannot pick it. True. It is equally true of every stage after retrieval, and that is where scale starts to hurt.
At around 100 items the plain pipeline is enough. A 100-tool catalog at roughly 400 tokens per schema is 40k tokens. That figure is the one I used in my Tool Attention paper, and real servers run higher: the public audits I cited put the full GitHub server at about 55,000 tokens for 93 tools, close to 590 each. Forty thousand tokens is wasteful but it fits, and a single top-10 embedding lookup rarely misses.
At 1,000 tools the same arithmetic gives 400k tokens. Retrieval stops being an optimization and becomes the only way in. That is the point where embed, retrieve, decide and validate is necessary and no longer sufficient. The rest of this post is about the parts I would add.
It is tempting to treat tool selection as classification with a bigger label set. The analogy holds for the retrieval half and breaks for everything around it.
| Property | Category classification | Tool selection |
|---|---|---|
| Wrong answer | A wrong value in a row. Fixable later. | A side effect: a sent email, a deleted record, a charged card. |
| Ground truth | A taxonomy that changes slowly. | The catalog, plus per-user permissions, plus live connection state. It changes per session. |
| Decisions | One per input. | A sequence. Each observation changes which tools are relevant next. |
| Near-duplicates | Adjacent categories. | Overlapping tools from different servers, often three different search tools. |
| Abstention | One exit: insufficient evidence. | Three exits: ask the user, widen the search, or report the missing capability. |
| Trust | The input text is untrusted. | The definitions are untrusted too. A tool description is text the model obeys. |
Row one is the one that changes the architecture. Safety cannot live in the prompt when a wrong choice writes to a production system. It has to live in code that runs regardless of what the model says.
The baseline write-up frames retrieval as supplying information the model lacks. That is half of it. The other half is that a model with all the information in a crowded context still does worse, even when the right answer is technically in there.
These are four different benchmarks, so I am not comparing them to each other. They agree on the direction, and that is enough for the design rule: retrieval is a context-control mechanism. Expose the smallest decision space that still contains the answer.
Once you add a reranker, the ceiling is no longer one number. Every stage can drop the right tool, and the drops compound. The precise form is a product of conditional recalls:
P(right tool used) = R_retrieve # gold is in the candidate set * R_filter | retrieved # gold survives policy filters * R_menu | filtered # gold survives the reranker cut * R_select | menu # the LLM picks it from the menu
The word "conditional" matters. Multiplying raw retrieval recall by raw reranker recall double counts, because the reranker only ever sees what retrieval returned. Here is an illustration with numbers I made up to show the shape, not measurements:
| Stage (illustrative) | Conditional recall | Cumulative |
|---|---|---|
| Retrieve 40 | 0.95 | 0.950 |
| Hard filters | 0.98 | 0.931 |
| Rerank to 6 | 0.92 | 0.857 |
| LLM selects | 0.90 | 0.771 |
Raising retrieval from 0.95 to 0.99 lifts the end result by about three points. Raising selection from 0.90 to 0.95 lifts it by about four. The weak stage is rarely the one people tune first, because retrieval is the stage everyone knows how to tune. The fix is to log four sets per request and compute the numbers per stage:
STAGES = ["retrieved", "filtered", "menu", "chosen"] # each is a set of tool_ids on the trace def stage_recall(traces): alive, out = traces, {} for stage in STAGES: kept = [t for t in alive if t.gold in getattr(t, stage)] out[stage] = len(kept) / max(len(alive), 1) # conditional on the previous stage alive = kept return out # multiply the values for end-to-end
Evaluating only the LLM is misleading, and the baseline write-up says so. At scale two numbers are not enough. You need four, and you need a labeled set where the gold tool is one the user could actually call.
The first thing that breaks at scale is not the model or the index. It is the catalog. In my simulation 29 of 500 tasks failed with tool gating. Fourteen (48%) came from a query matching several semantically similar tools. Seven (24%) came from cryptic legacy descriptions. That is 72% of failures caused by what the catalog says, and the model was not the problem. It is a simulated workload, so read the proportions as a hypothesis worth testing on your own traces.
So keep the registry outside the prompt and give every tool a card written for retrieval, not for the implementer:
{
"tool_id": "github.search_code",
"domain": "source_control",
"summary": "Find where something is implemented across connected repositories",
"capabilities": ["code_search", "repository_read"],
"entities": ["repository", "file", "commit"],
"input_concepts": ["query", "repository", "organization"],
"prefer_over": ["web.search"],
"not_for": "Reading one known file. Use github.read_file.",
"examples": ["Find where login sessions are validated in the checkout-service repository"],
// not indexed: fetched only after the tool is selected
"requires": { "connection": "github", "scopes": ["repo:read"] },
"risk": "read_only",
"cost": "low", "p50_ms": 900,
"owner": "platform-team", "schema_version": 7,
"input_schema": { /* full JSON Schema */ }
}
Three fields do more work than they look. summary is written in the voice of the user's intent. In my paper, regenerating summaries that way cut their average length by 63% and raised retrieval F1 by 8 points, on the same synthetic benchmark. not_for and prefer_over are the disambiguation fields. They give the reranker and the model a reason to choose between near-duplicates, and near-duplicates were the largest failure bucket. owner and schema_version exist because at 1,000 tools somebody has to be paged when a description rots.
One structural change belongs here too. A flat top-k over thousands of tools tends to fill the shortlist with ten variants of one family and crowd out the second family you needed. Group tools into domains and route in two hops, domain first and tool second. This is my rule from building these systems, not a measured result. Test it against a flat index on your own catalog before you trust it.
The baseline pipeline embeds the input once. A tool request is rarely one intent. "Find where sessions are validated in checkout-service and check whether the fix is deployed" is two subgoals with different tools. Embedding the whole sentence returns a blurred neighborhood that serves neither. I extract structured intent first and run discovery per subgoal:
{
"subgoals": [
{ "goal": "locate session validation code",
"operations": ["repository_search", "code_read"],
"entities": ["checkout-service repository"],
"required_access": ["GitHub"], "mutation": false },
{ "goal": "check deployment status",
"operations": ["rollout_inspect"],
"entities": ["checkout-service"],
"required_access": ["Kubernetes"], "mutation": false }
]
}
Retrieval then runs three ways and fuses the ranks. Vectors catch paraphrase. BM25 catches the exact tokens embeddings blur: tool ids, product names, error codes. Capability matching uses the structured operations field. Reciprocal rank fusion (Cormack et al., SIGIR 2009, constant 60) merges them without needing comparable scores.
Two more signals feed the same stage: historical success on similar tasks, and whether the connector is available in this environment. I use history as a ranker feature and availability as a pre-filter. False positives are acceptable here and a miss is not. A missing tool costs the whole task, while an irrelevant one costs a few tokens that the reranker removes.
The detail people miss is the pre-filter. If you filter for permissions after retrieval, the top 40 can be full of tools this user cannot call, and your recall of usable tools collapses while the recall you measure looks fine. Push the static predicates into the index query.
def rrf(rankings, k=60): score = defaultdict(float) for ranking in rankings: for rank, tool_id in enumerate(ranking, start=1): score[tool_id] += 1.0 / (k + rank) return sorted(score, key=score.get, reverse=True) def discover(sub, ctx, index, n=40, menu=6): where = {"connection": ctx.connections, "env": ctx.env} # static, pushed into the index fused = rrf([ index.vector(sub.text, n=n, where=where), index.bm25(sub.text, n=n, where=where), index.by_capability(sub.operations, n=n, where=where), ])[:n] allowed = [t for t in fused if request_ok(t, sub, ctx)] # depends on this request ranked = reranker.score(sub, allowed) # learned, relevance only return prefer_cheap_within(ranked, margin=0.05)[:menu] def request_ok(tool, sub, ctx): return (tool.scopes <= ctx.granted_scopes and (tool.risk == "read_only" or sub.mutation))
Hard filters are deterministic policy, not LLM judgment. They remove a missing connection or scope, incompatible data types, a write tool for a read-only request (and a read-only tool for a write request), an action that policy disallows or that is riskier than the request needs, and any tool unavailable in this environment. They run before any model sees a candidate, and they are cheap to test.
Each stage optimizes something different, and it helps to say so out loud:
The reranker then cuts roughly 30 to 50 candidates to a menu of 5 to 8. I keep the cut generous. A reranker error deletes the right tool for good, so this is the stage where I trade a few extra tokens for recall. In my paper I recommended k of 8 to 12 with the threshold doing the precision work.
A common way to specify the reranker is a weighted sum over the candidate tool t:
S(t) = w1*I + w2*C + w3*E + w4*P + w5*H - w6*R - w7*L - w8*K
I intent match, C capability match, E entity or data-source match.P permission and availability, H historical success on similar tasks.R operational risk, L expected latency, K execution cost.The weights should move with the task. For a sensitive action, permission and risk count for more. For an interactive request, relevance and latency count for more. I like the inputs and I like that instinct. I would change how the terms combine:
| Term | Problem inside a weighted sum | Where I put it instead |
|---|---|---|
| Permission | A missing scope is not a small penalty. It is a no. It also double counts with the hard filter. | Filter only. |
| Risk | Subtracting risk lets a slightly more relevant destructive tool beat a safe adequate one, or the reverse, and nobody audits the weights. | A risk policy after selection: confirm, dry-run or deny. |
| Cost, latency | They should never outvote relevance. | A tiebreaker among tools within a small relevance margin. |
| History | Popular tools collect more history and crowd out better new ones. | A learned feature, with deliberate exploration of new tools. |
| Hand-set weights | They do not survive a catalog change. | Train the relevance ranker on (subgoal, tool used, success) traces. |
The advice to weight permission and risk more heavily for sensitive actions points the same way. Past some weight they stop being weights and become gates.
Anything that must never happen belongs in a filter or a policy. Anything that is a preference belongs in the ranker. A weighted sum blurs the two, and the blur is where incidents come from.
Retrieval narrows which tools are visible. Progressive disclosure narrows how much the model sees about each one. The token figures below are rough, and they follow from the 400-token schema assumption above.
| Stage | The model sees | ~Tokens per tool | Trigger |
|---|---|---|---|
| Discovery | Name and one-line capability | 30 to 40 | Every subgoal |
| Selection | Constraints, risk, cost, one example | 100 to 150 | Only when two menu items are close |
| Invocation | Full input schema | 400+ | Only for the chosen tool |
| Recovery | The error, alternatives, a wider menu | varies | After a failure or a rejected call |
A six-tool menu at discovery level is about 200 tokens. The same six with full schemas is about 2,400. The saving is real, and so is the risk: the model chooses from one-liners, so the one-liners must be good. That takes you straight back to the catalog. This is all the model sees at the discovery level:
1. github.search_code Search code inside connected repositories 2. github.read_file Read a specific repository file 3. jira.search_issues Search issues and tickets 4. web.search Search public internet sources
Only after it picks github.search_code does the system reveal that tool's full input schema. Two operational details. First, keep the summary pool in the stable prefix of the prompt and put per-turn schemas right before the user message. In my simulation this gave an 84% prompt-cache hit rate over a 30-turn session against 22% for naive full-schema injection, because the tool list stopped invalidating the cache. Second, give the model an escape hatch: a search_tools meta-tool it can call when the menu is wrong. Anthropic's tool search is the same idea shipped as a platform feature.
Pair that with a gate: if the model calls a tool that is not in the active set, reject it with a structured error listing what is available. In my simulation the gate fired on 2.3% of turns. The model recovered on the next turn 78% of the time and asked the user a clarifying question the other 22%.
Validate before executing. The model may pick one tool or a short sequence from the menu, and each call gets deterministic checks on required parameters, types, permissions, read versus write intent, confirmation requirements and target scope. For writes I add a dry-run where the tool supports one. Feed validator errors back to the model verbatim. They are the cheapest correction signal in the system.
Return observations, not payloads. The executor should cap and normalize the result, and keep a handle to the full output outside the context. A 200 KB response dumped into the prompt undoes everything the earlier stages saved.
Rediscover on every subgoal. A tool's output can reveal the next need: a config reference that points to a deployment system you had no tools for. In my simulation 5 of the 29 failures (17%) were multi-hop cases where the right tool only became relevant after an intermediate result. Re-embedding the query after each observation partly fixed them. The agent should not carry every plausibly useful tool for the whole session.
search the repository # round 1: github.search_code -> found a config reference config says it ships as a Kubernetes rollout # new subgoal retrieve Kubernetes tools # round 2: k8s.get_rollout_status read the failing pod's logs # round 3: k8s.get_pod_logs
Name the failure class. "The agent used the wrong tool" is not a diagnosis. There are at least six distinct failures, each with a different detector and a different fix:
| Failure class | How you see it | Where the fix lives |
|---|---|---|
| No suitable tool exists | Gold label is "none". Abstention rate on those cases. | Report the missing capability. Roadmap input. |
| Exists, not retrieved | Low R_retrieve |
Catalog text, hybrid retrieval, intent decomposition. |
| Retrieved, cut by reranker | Low R_menu |
Reranker training data, a larger menu, not_for fields. |
| Shown, not chosen | Low R_select |
Better one-liners, a selection-level disclosure step. |
| Chosen, invoked wrongly | Validator rejection rate per tool | Examples in the schema, clearer parameter names. |
| Ran, returned too little | Observation check fails; the subgoal is still open | Next tool in the candidate list, or a new discovery round. |
Recovery paths follow from the table. Low discovery confidence: widen retrieval, reformulate the intent, or ask the user. A selected tool fails: return to the candidate list and take the next compatible one. Nothing viable: say which capability or connection is missing. Do not let the model invent an answer to fill the gap.
Everything above is easier to judge on a single request. This is an illustration, not a benchmark: the catalog size, the counts and the tool names are mine, chosen to be realistic. The catalog has 1,200 tools across 30 servers, and the request is:
"The checkout-service deploy failed last night. Find what changed, check the rollout, and tell the payments channel on Slack."
| Stage | What happens | The model sees |
|---|---|---|
| Intent extraction | Three subgoals: find what changed (GitHub, read), check the rollout (Kubernetes, read), post a summary (Slack, write). | The request only. |
| Retrieval, per subgoal | 40 candidates each, out of 1,200. For the rollout subgoal, BM25 catches the literal word "rollout" in k8s.get_rollout_status, which the vector search had ranked ninth. | Nothing yet. |
| Hard filters | Rollout subgoal: 9 candidates removed because this user has no Argo CD connection, and k8s.rollout_restart removed because the subgoal is read-only. 30 remain. | Nothing yet. |
| Rerank | 30 become a menu of 6. The not_for field on logs.search ("for Kubernetes pod logs use k8s.get_pod_logs") settles a near-duplicate. | Nothing yet. |
| Disclosure and selection | The menu is shown at discovery level. The model picks k8s.get_rollout_status, and only then is its full schema revealed. | Six one-liners (about 200 tokens), then one schema (about 400). |
| Validation | The required parameter namespace is missing. The validator returns an error, the model supplies payments, and the second attempt passes. | The error string. |
| Execution | 200 KB of pod events is capped to a 1.2 KB observation, with a handle to the full output kept outside the context. | 1.2 KB. |
| Write subgoal | Posting to Slack is a write. The risk policy shows the user the draft message and waits for a confirmation before slack.post_message runs. | The draft message. |
Count the tokens with the 400-token schema assumption from earlier. Showing the whole catalog would cost about 480k tokens on every turn, more than most context windows hold. This run used three menus of about 200 tokens and three schemas of about 400, roughly 1,800 tokens, a cut of more than 99%. Those are assumptions multiplied out, not a measurement.
The more useful observation is where the LLM acted. It made two kinds of decision: splitting the request, and choosing from a menu of six. Every other step was an index, a rule or a learned ranker. That is the point of the control plane. If this run had gone wrong, it would have gone wrong in one identifiable place. Had k8s.get_rollout_status never been retrieved, that is a retrieval miss. Had it been retrieved and cut, that is the reranker. Had it been chosen and called badly, the validator catches it before anything runs.
Here is the whole flow in one place, in three stages. It starts where the user does, with the request. The tag on each step says whether it is rules, a learned model, or the LLM. Two of the eight steps are the LLM, and every loop is decided by code.
You do not need all of it. The thresholds below are my judgment from building these systems, not measured boundaries. In my simulation the full-schema baseline crossed the 70% context-utilization line at about 50 tools, which is why the first row is short.
| Tools | What breaks first | Add |
|---|---|---|
| Up to ~50 | Nothing yet. | Nothing. Send them all. |
| 50 to 200 | Cost and mid-list misses. | The baseline pipeline: one embedding, top-k, definitions, validation. |
| 200 to 1,000 | Near-duplicates, multi-step tasks, unmeasurable failures. | Hybrid retrieval, hard filters, progressive disclosure, per-stage recall. |
| 1,000+ | Shortlists crowded by one family, catalog rot, stage latency. | Domain routing, a trained reranker, per-subgoal discovery, catalog ownership and linting. |
Each stage adds latency, and I have not measured a full stack end to end, so budget it before you commit. Indexes go stale when a server ships a new schema, which is why schema_version exists. And tool descriptions are attacker-controlled text. In my simulation, gating cut the success rate of 50 poisoned descriptions from 38% to 6%. That was a side effect of never showing irrelevant tools, not a defense, and it needs a real one.
For classification, the rule is to stop asking the LLM to remember the taxonomy and give it the taxonomy when it needs it. I would keep that and add its counterpart for tools.
Do not ask the LLM to be the only thing standing between a request and a side effect. Give it a small, filtered, well-described menu. Run the rules around it in code. Measure every stage, so that when the agent is wrong you know which part to fix.
My design rule fits in one line. The LLM reasons over the actions that are viable right now and does not carry the platform's whole capability surface in every prompt. Retrieve broadly, filter deterministically, rerank intelligently, and reveal full tool details only when the model needs them.
At 100 tools, the model does the work. At 1,000, the control plane does most of it, and the model makes the last call.