Behind the scenes : watching media on the browser | Async Impulse

January 9, 2021

13 min to read

Behind the scenes : watching media on the browser

Go through the whole journey of how media are being play on the internet.

Have you ever thought about what it’s actually happening when you click on the play button on your favorite media websites, maybe Netflix, Youtube, Twitch or what ever websites serving media streaming. How are those media data being transferred over the internet? How do they package them? And how do browsers handle those data and represent them to users? Let’s take a deeper look at that to see what’s going on.

Downloading Media Data

The content providers, such as Youtube, would put media content on their servers and then distribute them onto worldwide CDN, which referes to a geographically distributed group of servers which work together to provide fast delivery of internet content because users won’t need to go back to the actual data server which might be far from users in geographically.

When browsing a website containing a media element used to play media, it would start requesting data from the server. The process of downloading media data from servers is called “Progressive Download”. Between servers and your devices, a media file would be splitted to many small chunks and be packaged into HTTP response packets, being transferred over the internet bit by bit.

How many data do browsers need before media starts?

First thing you need to know is that the media data transferred over the internet are all compressed data, which is also called encoded data . That helps to reduce the file size by ten to twenty times. In audio compression, each audio file are independent, so decompressing one single audio frame is possible. However, in video compression, they’re usually not all independent. Imagine that if two video frames have almost 90% similarity in their scene, in this situation, instead of saving two frames in the original size, we could save only one original frame, and record the difference between two frames for another video frame, which help reducing the amount of data we need to store.

That is a basic concept of video compression, and the real compression algorithm is definitely way more complicated than that. Therefore, for video data, we need to get enough amount encoded data before we can output one uncompressed frame, which is also called decoded data . Thus, the answer is that having partial data is enough, but the amount of data that browsers need varies on different media files.

In the HTML specification for media element, it defines different levels of state indicating if media is ready for playing or not, which helps websites to know when they can starts playing media. After media starts playing, browsers would keep downloading remaining media data and store them beforehand, which we call caching data .

The reasons of caching data are (1) to keep playback smooth all the time. If we don’t have enough data to decode, then we have to wait for more data again, which would be not pleasant for users. (2) to improve the speed of jumping to other position in the file, which is called seeking. Once we have a whole file, we can jump to the arbitrary position in that file without requesting data from the server again. Requesting data from servers is an expensive operation, which would be done by using HTTP range request and we want to avoid that as possible as we can.

How many data would browsers cache?

It depends on each browser’s implementation, in Firefox, we cache at most 500 MB per process. That raises another question, can websites determine how many data would browser download ahead? Because they might want to buffer less data to save the bandwidth. And yes, that can be done by using Media Source Extension API that gives websites more ability to control the data they want to play, such as controlling the amount of buffered data and adaptive bitrate streaming.

Wait? “adaptive bitrate streaming”? It sounds like a cool stuff, what is it and why it does matter in terms of streaming media on internet?

Adaptive Bitrate Streaming (ABS)

Imagine that you’re in a moving train where the internet accessing is not constantly stable, sometime you get LTE, but sometime only 3G. If you want to watch a video under this circumstances, you might probably not be able to watch same solution video smoothly all the time because the internet speed varies, which would lead to a frequent buffering. That is why we need the adaptive bitrate streaming that would determine what quality of video is the best choise for user’s watching experience by considering user’s internet bandwidth and the CPU usage.

The basic idea is that the content providers would prepare videos with different resolutions on the servers, and give a list that defines the corresponding relationship betweens the bandwidth and video files, the timestamp for vides, media-content availability and other encoding details....