Web Bot Auth in the Wild — 79 directories, linted

Web Bot Auth · Cloudflare Radar verified-bot registry · 2026-07-10

We lint every published bot key directory. 1 in 6 is broken or down.

79 of the 684 bots in Cloudflare's verified-bot<br>registry publish a Web Bot Auth key directory — the<br>/.well-known/http-message-signatures-directory<br>a verifier reads to confirm a crawler is who it claims. We fetch and grade all of them with<br>@sitedex/web-bot-auth-lint.

13/79

directories fail verification or are unreachable.<br>Check yours before a verifier does: npx web-bot-auth-lint your-domain.com

All 79 directories, one square each

✓26 verify, zero findings<br>!40 verify, with warnings<br>✕9 fail verification<br>~4 unreachable

The most common hygiene problems

Warning-grade findings across all 79 directories. A directory can<br>carry several; warnings don't block verification today, but each is a mistake a stricter<br>verifier could reject.

The 9 failures

BotOperatorWhy a verifier rejects it

Cloudflare Radar URL ScannerCloudflareReturned HTTP 404, not 200. The directory has no `keys` array.<br>ConsentCheck BotConsentCheckReturned HTTP 404, not 200. The response body is not valid JSON.<br>DNSnifferBotDNSnifferNo usable key in this directory — Invalid public-key value: key #1 (x is invalid bytes, expected 32).<br>FlowpaneFlowpane LtdNo key is usable yet — every key's not-before time is in the future: QZdoanWPKy-cHIy6B080FKtxV0Zky1tSDQ6WIslvi7E.<br>NostraNostraNo key is usable yet — every key's not-before time is in the future: 3vKgyEtYsNMyY8M-CDk8_HzkuZO22aLy_LlHEjtT3WI, Pav1sqEdDlQ9qYyCUqV7J5oT3Eh2aTabW5-Ii6OCHT0.<br>Novellum AI CrawlNovellumReturned HTTP 404, not 200. The response body is not valid JSON.<br>PureConsentBotPureConsentThe response body is not valid JSON.<br>TrustLink-VerificationAgentPearl Industry Network LLCThe signature tag is "web-bot-auth"; a directory-response signature MUST carry tag="http-message-signatures-directory".<br>YouBotYou.comReturned HTTP 403, not 200. The response body is not valid JSON.

Unreachable after 3 attempts with backoff: Amazon Bedrock AgentCore Browser (AP South) (attempt 3: ETIMEDOUT: fetch failed); Quartr Crawler (attempt 3: UND_ERR_CONNECT_TIMEOUT: fetch failed); sourcedash (attempt 3: EAI_AGAIN: fetch failed); TwinAgent (attempt 3: ENOTFOUND: fetch failed).