Snowmen, Recruiters, and Terry Pratchett: The Web's HTTP Header Junk Drawer | JonLuca's Blog Skip to content HTML is what a site shows you. JavaScript is what it does. Headers are what it can’t help telling you.
They leak the habits of the machinery underneath: CDNs, frameworks, caches, security controls, dead browser workarounds, migration scars, and bits of infrastructure that were meant to be temporary and never left. I thought it would be interesting to see what unique or non-standard headers the most popular sites on the internet were serving, so I crawled the top 1,000 domains by traffic and saved their response headers.
For each domain I hit the root page, then one same-origin internal page when I could find one safely. I used a browser-like HTTP client and retried likely challenge pages once with Playwright. If a response still looked like a WAF, a bot wall, or an access-denied page, I logged it and dropped it from the stats so they wouldn’t skew the header counts.
# The dataset
Of the 1,000 domains I attempted, 417 returned clean pages I could analyze; the rest were WAFs, bot walls, or challenge pages I dropped. Those clean sites sent 677 distinct header names, and 619 of them aren’t in the IANA HTTP Field Name Registry.
All “not in IANA” means is that the name isn’t registered. The web runs on standards, but it also runs on convention, vendor prefixes, CDN metadata, and whatever someone shipped five years ago that still works. Learn only the standardized headers and you’ll have the formal grammar of the web while missing the dialect anyone actually speaks.
# Browser fossils
The crawl turned up a whole fossil bed:
headersitesnotex-xss-protection178Chrome’s old XSS auditor switchpragma84deprecated, but still used for cache controlp3p24compact privacy policies for old Internet Explorer cookie behaviorx-ua-compatible13Internet Explorer document-mode hintexpect-ct6certificate transparency enforcement, now deprecatedfeature-policy6predecessor to Permissions-Policycontent-md52obsoleted integrity header<br>P3P is my favorite. It’s a privacy-policy header from the early 2000s, remembered mostly because setting any plausible-looking value could talk old IE into accepting third-party cookies. One value in the crawl is exactly the kind of thing you hope to dig up:
CP="This is not a P3P policy! See g.co/p3phelp for more info."<br>The header is obsolete. The scar tissue stays.
# Security headers are uneven
Among the 417 eligible sites, adoption of common browser security headers ranged widely:
headersitessharestrict-transport-security27064.7%x-frame-options21451.3%content-security-policy19446.5%referrer-policy10525.2%permissions-policy6315.1%cross-origin-opener-policy4510.8%cross-origin-resource-policy286.7%cross-origin-embedder-policy20.5%clear-site-data00.0%<br>Plenty of sites have good reasons to skip some of these, so read the table as a map rather than a scorecard. COEP breaks the moment you embed a third-party resource. Clear-Site-Data is a sharp tool.
The spread still tells a story. HSTS is now normal. CSP is common but not yet universal. The newer cross-origin isolation headers remain rare. Browser security is a stack of migrations, and most migrations never finish.
# Infrastructure leaks through
Some headers act as status lights on the machines behind the page.
headersiteswhat it revealsserver303server or gateway familyx-cache147cache statevia146proxy/CDN pathx-served-by53edge node or cache layerx-powered-by47framework or runtimex-request-id31request tracingx-generator7CMS or static-site generator<br>Some Server values are dull: nginx, cloudflare, gws, AkamaiGHost. Others say more. The crawl found Express, Next.js, ASP.NET, and Drupal generator strings scattered around.
On its own, most of this is harmless operational metadata. But an outsider can use it to cluster sites by stack, host, CDN, framework, and sometimes deployment shape. The public web ships a lot of public implementation detail.
# The largest headers were genuinely large
The biggest header block I saw belonged to state.gov, around 15.6 KB. The runners-up were big enough to notice:
domainpageapproximate header bytesstate.govroot15,622state.govinternal15,620eset.cominternal12,819mixpanel.cominternal12,554cursor.shroot11,702<br>These are approximate, since the crawler redacts sensitive-looking values before analysis. The point holds: headers can grow into a real chunk of the response. The bulk usually comes from reporting endpoints, CSP directives, cookies, or CDN metadata. The body gets blamed for web bloat, but the prelude packs on weight too.
# Some headers are just for fun
A few headers in the crawl weren’t metadata at all. Someone wrote them by hand.
headervaluesitex-clacks-overheadGNU Terry Pratchettmozilla.org, debian.orgx-hackera recruiting pitch (full text below)wordpress.comx-recruitinga recruiting pitch (full text below)otto.dex-launch-statusGo...