Informing HTTP Extension Design with Data

Mark Nottingham

recent entries<br>all entries<br>feed

Hi, I’m Mark Nottingham.<br>I write about the Web, protocol design, HTTP, Internet governance, and more. This is a personal<br>blog, it does not represent anyone else.<br>Find out more.

Comments? Let's talk on Mastodon.<br>@mnot@techpolicy.social

other Internet and Web posts

The Nature of Internet Standards (series)

No One Should Have That Much Power<br>Monday, 29 April 2024

RFC 9518 - What Can Internet Standards Do About Centralisation?<br>Tuesday, 19 December 2023

Moving Control to the Endpoints<br>Tuesday, 11 June 2019

What is the Web?<br>Thursday, 4 December 2014

Five Favourite Protocol Design Papers<br>Thursday, 15 April 2004

Informing HTTP Extension Design with Data

Friday, 7 August 2026

Internet and Web

A while back, I refactored REDbot to separate out a HTTP message linting library. httplint checks about 325 aspects of HTTP messages, covering the syntax and semantics of almost 200 HTTP header fields.

Then, I got curious: what happens when you point it at Common Crawl? Its availability on AWS brings the opportunity to do a large-scale survey of how HTTP is used and abused. With the help of Claude Code, the result is cc-lint.

Pointed at over 120 million responses from nearly 50,000 sites, the resulting report is a lot to digest. For me, the most interesting aspects are reflections of how we design new protocol elements – and often, how they come up short. Collectively, what I observe below is in some ways very obvious, but also we keep on forgetting it when we design new protocol elements – i.e., “hope-based standardisation.”

So, this post serves as a reminder or wake-up call for protocol designers (depending on your perspective). It covers the tool and those observations; followups are likely to dive into specifics.

This is based upon the lightning talk I gave at the 2026 HTTP Workshop.

Background and Caveats

First, the caveats.

Common Crawl is not the whole Web; they are blocked from crawling some sites, for example; others sit behind paywalls, WAFs or geofences. Furthermore, cc-lint focuses on the Tranco top 100,000 sites, so that behaviour of very small, idiosyncratic sites (especially spam sites) doesn’t skew the results.

The corpus is overwhelmingly 200 (OK) responses to GET requests: almost no redirects, errors, conditional or range requests. And it’s response metadata only — headers, no bodies, and nothing from the request side.

Put together, what you’re looking at is the successful responses of popular sites, seen from one side. A lot of the most interesting HTTP — error handling, negotiation, cache validation — occurs in the responses that aren’t here. But it’s still over 120 million real responses, which turns out to be plenty to see some patterns very clearly.

1. Set It and Forget It

It’s very noticeable that a common deployment pattern for a successful protocol extension (like a new header field) is “fire and forget” – someone decides they need a particular behaviour, looks up the relevant protocol element, adds it to their configuration, and never revisits that decision.

This means that adoption often depends more on how much effort is involved than how important it is to get things right.

Headers that can be copied from a HOWTO and then forgotten have great deployment; HSTS is on 51% of sites, X-Frame-Options on 44%, and even X-XSS-Protection – which browsers stopped paying attention to seven years ago – is still on 29% of sites.

A header that requires crafting a site-specific policy after careful consideration fares less well; Content-Security-Policy is on 32% of sites, but there are major caveats on that which I’ll cover in a separate post. Permissions-Policy is only present on 9% of sites; Cross-Origin-Opener-Policy on 4% and Cross-Origin-Embedder-Policy on less than 1%.

So, if you want your protocol extension to be adopted and deployed – and even zombie on long after its useful life has ended – keep it simple and repeatable.

2. Make It Break

Another pattern: an extension gets deployed correctly in proportion to how loudly it fails when you get it wrong. The visible failure is its own feedback loop: operators fix what they’d otherwise never notice.

The clearest evidence is a natural experiment that the browsers have unknowingly run: the *-Report-Only headers. A policy header and its -Report-Only twin share identical syntax and are set by the same kinds of operators — but the plain header changes browser behaviour (so a mistake is visible), while the Report-Only version only sends a report somewhere (so a mistake usually isn’t). Turn enforcement off and the error rate jumps:

Cross-Origin-Opener-Policy is malformed in 3.0% of responses; its Report-Only variant, 12.7% — about four times worse.

Cross-Origin-Embedder-Policy climbs from 13.1% to 20.6% the same way.

The syntax is the same; only the consequence of getting it wrong differs.

So, if you want your header to be emitted correctly,...