HTTP Integrity Digests: Past, Present, and Future | Lucas Pardue - Bits && Pieces
The core HTTP specifications don’t define their own means of providing<br>integrity. Although “hop-by-hop” protocols underneath HTTP, such as TLS or QUIC,<br>provide integrity for HTTP message transactions, it isn’t enough. HTTP powers<br>massively distributed systems where intermediaries (aka proxies) are first-class<br>citizens. Coupled with that are the powerful semantics of HTTP resources and<br>representations.
This post explores the past, present, and future of HTTP extensions that allow<br>for HTTP Integrity. We’ll cover core concepts, fields such as Content-MD5,<br>Digest, Content-Digest, Repr-Digest, Unencoded-Digest, and more.<br>However, some foundational HTTP understanding needs to be established before we<br>can start. If you just want to jump into an interactive demo playground see my<br>HTTP Integrity Demos page.
What Do We Mean By Integrity?
This is one of the most important things to define before we get into harder<br>topics. I really like Wikipedia’s opening sentence in the Data Integrity article:
Data integrity is the maintenance of, and the assurance of, data accuracy and<br>consistency over its entire<br>life-cycle[1]<br>and is a critical aspect to the design, implementation, and usage of any<br>system that stores, processes, or retrieves data.
For the purposes of this post, we can assume data integrity relates to ensuring<br>the bytes of a $thing are exactly the bytes that are expected. Issues such<br>as data corruption, software program errors, or malicious modification could<br>cause unexpected changes that damage integrity.
When it comes to HTTP, what the $thing is is often more complicated than your<br>first intuition might suggest. This complexity arises from HTTP Semantics<br>defined in RFC 9110.
In essence, HTTP semantics define concepts like actors (clients, servers,<br>proxies), request/response messages (and their sub components like methods,<br>status codes, data, and metadata), and resources and representations. If you are<br>not familiar with this, I suggest reading both the RFCs (RFC<br>9110 et al) and related<br>resources such as MDN’s HTTP<br>pages (caveat: MDN can be<br>incorrect, more on that later in the post. If ever in doubt defer to the RFCs).
For the remainder of this blog, I’ll use “HTTP” as shorthand for “HTTP Semantics<br>defined in RFC 9110”.
Finally, deciding what to do if an integrity check fails is often something the<br>upper-layers need to decide. It needs to factor in the threat model. A text file<br>that is corrupted by one-byte may still be readable by a human and pose no<br>security threat. A binary file that has been maliciously swapped out could be a<br>major security threat. HTTP specifications for integrity tend to punt on these<br>matters. Instead, we expect an “application profile” to define the nature of<br>interactions between endpoints, what is acceptable or not, and how to deal with<br>integrity validation failures.
Differences Between Resources and Representations
Section 3.1 of HTTP<br>defines the term “resource”:
The target of an HTTP request is called a “resource”. HTTP does not limit the<br>nature of a resource; it merely defines an interface that might be used to<br>interact with resources. Most resources are identified by a Uniform Resource<br>Identifier (URI), as described in Section 4.
Section 3.2 of HTTP<br>defines the term “representation”.
A “representation” is information that is intended to reflect a past, current,<br>or desired state of a given resource, in a format that can be readily<br>communicated via the protocol. A representation consists of a set of<br>representation metadata and a potentially unbounded stream of representation<br>data (Section 8).
One of HTTP’s super powers is the ability for a single resource, identified by<br>some URI like https://lucaspardue.com/example-resource, to generate multiple<br>different representations. For example, different user agents could send<br>different requests with different language preferences all to the same URI, and<br>my server could return an appropriate translation to them all. This concept is<br>formally summarized by the statement in Paragraph 4 of Section 3.2 of HTTP:
A target resource might be provided with, or be capable of generating,<br>multiple representations that are each intended to reflect the resource’s<br>current state. An algorithm, usually based on content negotiation (Section 12),<br>would be used to select one of those representations as being most<br>applicable to a given request. This “selected representation” provides the<br>data and metadata for evaluating conditional requests (Section 13) and<br>constructing the content for 200 (OK), 206 (Partial Content), and 304 (Not<br>Modified) responses to GET (Section 9.3.1).
Metadata, Data … HTTP Fields and Content
Representation data and metadata are defined Section 8 of HTTP.<br>Metadata is sent in the form of HTTP...