Tool search: Finding the right tool at the right time - Command Line
Skip to main content
Skip to main content
Source
Signal blog<br>Official Microsoft Blog<br>Command Line<br>Microsoft On The Issues<br>Asia<br>Canada<br>Europe, Middle East and Africa<br>Latin America<br>The Code of Us
What's new today
AI
Innovation
Digital Transformation
Sustainability
Security
Work & Life
Diversity & Inclusion
Unlocked
Microsoft 365<br>Azure<br>Copilot<br>Windows<br>Surface<br>XBOX<br>Deals<br>Small Business<br>Support
Windows Apps<br>Outlook<br>OneDrive<br>Microsoft Teams<br>OneNote<br>Microsoft Edge<br>Moving from Skype to Teams
Computers<br>Shop XBOX<br>Accessories<br>VR & mixed reality<br>Certified Refurbished<br>Trade-in for cash
XBOX Game Pass Ultimate<br>PC Game Pass<br>XBOX games<br>PC games
Microsoft AI<br>Microsoft Security<br>Dynamics 365<br>Microsoft 365 for business<br>Microsoft Power Platform<br>Windows 365<br>Small Business<br>Digital Sovereignty
Azure<br>Microsoft Developer<br>Microsoft Learn<br>Support for AI marketplace apps<br>Microsoft Tech Community<br>Microsoft Marketplace<br>Software companies<br>Visual Studio
Microsoft Rewards<br>Free downloads & security<br>Education<br>Gift cards<br>Licensing<br>Unlocked stories
View Sitemap
Search articles
Deep Dive
Share
in
Tool search: Finding the right tool at the right time
Tool search: Finding the right tool at the right time
As you add more tools to AI agents, the advantage comes from making the right tools discoverable at the right moment without paying the full cost of exposing everything upfront.
By Viswajeeet BalajiPrincipal Software Engineer, Microsoft and Saket SathePrincipal Applied Scientist, Microsoft
2026.07.29
Every tool you give an agent is both a capability and a distraction. Five tools make an agent feel capable. Fifty tools make it feel prepared. A hundred tools can make every turn start with thousands of tokens of names, descriptions, JSON schemas, argument definitions, and nested parameters before you’ve asked anything useful. The agent looks more powerful, but first it has to read a menu it may not need.
This is one of the tensions Toolboxes in Microsoft Foundry is designed to solve at enterprise scale. A single toolbox can front Microsoft IQ, Work IQ, OpenAPI tools, A2A integrations, remote MCP servers, and several native Azure capabilities. Agent builders should be able to scale heterogeneous tool catalogs without rebuilding integrations or loading every tool on every turn. The old line about great power going hand-in-hand with great responsibility applies here, but the responsibility lives at the context layer: if the platform makes it easy to connect everything, it also needs a way to keep the model focused on what matters now.
Tool search capability in Toolbox emerged as we worked backwards from customer experience. Large tool catalogs were becoming too expensive to send to the model on every turn, and the model didn’t need most of them for most tasks. Our initial experiment of deferring all the tools and letting the model search for tools based on user query did what we hoped: it made tool-using agents cheaper, made the prompt smaller, and kept the system prompt stable enough to work well with prompt caching.
Then the real story emerged. We thought we were solving a token-cost problem; we were also building a search product. At small scale, a catalog can look like schema management: register the tool, validate the JSON, expose it to the model, dispatch the call. At larger scale, tool names and descriptions become ranking features. That’s the shift from tool-maxxing to tool relevance-maxxing.
The default agent tax
The problem isn’t just cost, though cost is the easiest part to measure. A full manifest also fills the context window with definitions unrelated to the current task. The model has to choose from an overcrowded menu, and the prompt prefix becomes larger and more fragile.
Prompt caching is enabled by default in Azure OpenAI and is the recommended behavior, so our baseline had to include it. But caching isn’t the same as not loading. Cached tokens are roughly 90% cheaper than regular input tokens, not free, and cached context still competes for the model’s attention. The obvious move was to stop loading everything upfront.
Two tools instead of a hundred
Tool search changes the initial contract between the toolbox and the model. Instead of exposing every toolbox tool in the first tools/list, Foundry can expose two meta-tools: tool_search(query, limit) and call_tool(name, arguments). The model describes the capability it needs, Foundry searches the toolbox, and the model receives a small set of matching tool definitions before calling the chosen tool with name and arguments.
The rest of the catalog stays hidden from the initial tool list. That’s why the second proxy exists: if a tool wasn’t registered in the original tools/list, many runtimes will guard against the model calling it directly as an unknown tool. call_tool gives the framework a registered, policy-aware dispatch path.
That sounds almost too small to be an...