The Discovery Problem: When AI Agents Build Tools No One Can Find

The Discovery Problem: When AI Agents Build Tools No One Can Find

We installed 15 AI skills in a single session. Eight were invisible to future agents — not because they were broken, but because they lacked three lines of metadata.

Next AI Labs · Human-Agent Research · 9 min read

A case study in capability discoverability — the gap between what tools exist in an AI system and what agents can actually find. We document how 53% of newly installed skills failed a discoverability audit, the five surfaces agents use to find tools, and why self-auditing discovery is the missing step in multi-agent toolchains.

The most dangerous failure mode in a multi-agent system isn't a tool that breaks. It's a tool that works but can't be found.

We discovered this the hard way. In a single session, we installed fifteen AI skills — security auditors from Trail of Bits, OWASP vulnerability checkers, Playwright browser automation, custom test fixture generators, error handling standards, and more. Every one worked. Every one passed its verification test. And eight of them — more than half — were invisible to any future agent that might need them.

Not broken. Not misconfigured. Just unfindable.

The Risk: Invisible Capability Accumulates Silently

When human teams lose institutional knowledge, someone eventually notices. A new hire asks a question no one can answer. A process breaks and nobody remembers why it existed. The knowledge gap becomes visible through its consequences.

AI agents don't have this feedback loop. When an agent builds a tool that other agents can't discover, nothing visibly breaks. The system doesn't error. No alert fires. The tool sits in its directory, fully functional, while future agents — facing the exact problem it solves — build their own solution from scratch. Or worse, they proceed without it and ship vulnerable code that the undiscoverable security auditor would have caught.

This is [[concept:invisible-capability]] in action — the AI equivalent of institutional knowledge trapped in someone's head, except no one even knows to ask them about it.

The compounding effect is what makes this dangerous. Every session that builds tools without verifying discoverability adds to the invisible pile. Over time, the gap between what tools exist and what agents can find — the [[concept:capability-discoverability-gap]] — grows wider. The system becomes simultaneously more capable and less effective.

What We Tried: Building Fifteen Skills in Parallel

The session that revealed this problem was ambitious by design. We needed to close gaps in security, code quality, and system observability for a production coaching platform's API. The plan called for three phases of parallel installation:

Phase 1 — Four external security skills: OWASP Top 10 checks, six Trail of Bits audit methodologies (insecure defaults detection, variant analysis, differential review, sharp edges identification, static analysis, and Semgrep rule creation), a destructive-command safety net, and Anthropic's official skill library.

Phase 2 — Three infrastructure tools: a read-only MongoDB connector for live debugging, Playwright browser automation for end-to-end testing, and a configuration linter for agent setup files.

Phase 3 — Seven custom skills built from codebase analysis: a controller extraction workflow for decomposing 2,500-line files, React component patterns specific to the codebase, a silent failure detector for swallowed errors, system observability standards, API error handling conventions, per-endpoint security checklists, and canonical test fixtures.

Each phase ran as parallel subagents — four agents installing external tools simultaneously while the main context wrote custom skills. The entire installation completed in a single session. Every tool passed its functional test.

Then we ran the [[concept:discovery-audit]].

The Discovery Audit: 53% Invisible

The audit checked each skill against a simple question: can a future agent, facing a relevant problem, find this tool through automated discovery?

The results were stark:

| Skill Source | Count | Discoverable | Invisible |<br>|-------------|-------|-------------|-----------|<br>| Custom-built | 7 | 7 (100%) | 0 |<br>| External | 8 | 0 (0%) | 8 (100%) |<br>| Total | 15 | 7 (47%) | 8 (53%) |

Every custom-built skill was discoverable. Every externally-sourced skill was invisible.

The pattern was consistent: the seven custom skills had been written with discovery metadata built in — keyword trigger phrases that the matching system uses to connect agent queries to relevant skills. The eight external skills, sourced from respected security research firms and open-source repositories, had everything except that metadata.

The fix was absurdly small. Three to five lines of YAML per skill:

```yaml<br>triggers:<br>"security vulnerability"<br>"injection prevention"<br>"XSS prevention"<br>"security review"<br>```

That's it. Without...