Video — thirty years of breaking the web's assumptions — talking.video<br>← All postsFor a long time, video on the web didn’t really have anything to do with the web itself. The O.G. solutions available around the turn of the millennium were the RealPlayer and QuickTime plugins, both of which are largely best forgotten to avoid triggering any old traumas. (If you remember how the and tags worked, then you are officially old, congratulations — may your knees outlive the QuickTime brand which still lingers as a zombie in the macOS video player app.)

Browser plugins were effectively independent applications that could render into a frame within a web page. Like a blank zone on a medieval map that reads “Here Be Dragons”, they could contain absolutely anything. As native programs with full access to the operating system, plugins could freely mess around behind the browser’s back and without its knowledge: create sockets, read and write files, send the contents of your hard drive to an IP address in Belarus. Not coincidentally these were also the golden years of malware.

The “sockets” and “render into a frame” parts were useful for video, though. At that time, browsers didn’t have any APIs that could render even basic 2D graphics in realtime (Canvas wouldn’t be introduced by Apple’s Safari until some years later). The only networking interface was making a HTTP request. This assumed a single immediate response, but it could tortuously be used for a realtime connection in “long polling” mode where the server left the response open and kept writing to it. Some early webcams even used this to send a video feed as a series of JPEG frames, but not all browsers supported this kind of live-updating image.

A browser plugin didn’t have to play any of those games. It could open a proper socket to receive encoded video and audio data, decode those in native C code, write pixels into the frame it had within the browser window, and play sound to the user’s audio device. Presto, streaming video playback. And for the socket part, a crucial implementation detail was that a native program could use UDP instead of TCP.

Most Internet protocols including HTTP are built on TCP. It’s highly resilient to lost packets: if/when some data goes missing as it bounces around the Internet’s series of beautiful shiny pipes, the server will eventually find out and will retransmit those TCP packets. This is generally the right thing to do when you’re loading search results or viewing flight details or receiving a WhatsApp message from your long-lost teenage love — you want the data to arrive regardless of whether it takes 30 milliseconds or three seconds.

But for live video streaming, you mostly don’t care about frames that should have been displayed three seconds ago. That’s ancient history, there’s no point in retransmitting that data. UDP is a more basic kind of Internet protocol where senders don’t retransmit automatically, but instead the receiver has to make its own decisions about what to do about data that might be incomplete. It’s more complicated, but it lets you make efficient use of bandwidth and latency.

The browser plugin that put all this together in just the right way to enable novel audiovisual experiences on the web was Macromedia Flash. Precursors like QuickTime were basically just player applications that happened to be running inside a web page, but Flash let web developers create complete interactive experiences where video was just one element. It also helped that the Flash plugin shipped pre-installed on an astonishing 96% of all computers in 2003 (if there were such a thing as Tech Oscars, Macromedia would honestly deserve a posthumous Lifetime Achievement Award for “Best Partnerships Management”).

Flash shipped a video codec called Sorenson Spark already in 2002, but nobody liked it very much — it was inefficient, slow, and had an expensive closed source license. The video floodgates opened when Flash added a codec called VP6 in 2005. This was also the year when YouTube launched. (Indeed, YouTube’s video player would default to Flash for a full decade until 2015.)

For the next five years, everybody was mostly happy to keep web video in the desktop-centric Flash plugin sandbox. Flash soon added support for the H.264 video codec which remains incredibly popular for all kinds of video applications today. In 2009, a Russian teenager named Andrey Ternovskiy launched an app called ChatRoulette that proved Flash can do real-time two-way video calls in a browser. No extra install, just open a link to talk on video with a complete stranger. This capability opened the door to completely new browser-based experiences (as well as completely new ways to involuntarily see someone’s genitals).

Meanwhile in Cupertino, California, a clock was ticking down. Apple’s iPhone was a runaway success and its browser intentionally didn’t have Flash. In 2010, Steve Jobs wrote a public memo with the polite title “Thoughts on Flash” but...