As a consumer, when an Internet application doesn’t deliver to your expectations, you voice your frustration to your supplier, a consumer ISP. But it’s interesting to know how that data gets to your house, and who’s hands and decisions it flows through along the way.
In a simplified form, your ISP connects to a set of ‘peers’ (which are usually a type of ‘middleman’ ISP such as Level 3 or Cogent), and these peers in turn connect to the server you are accessing. The server is usually arrayed in a Content Distribution Network (CDN), such as LimeLight Networks and Akamai.
When more than one path between you and a CDN exists, as it usually does, the sender chooses which one they will use. In the diagram below, CDN1 makes two key decisions:
- They have chosen to connect to Peer1 and Peer2, but not Peer3 or Peer4. This constrains their choices.
- They choose whether to send traffic on Peer1 or Peer2 at any given time.
That simple example works if the content you want is served by only one CDN; you pick the content, the CDN picks the path, and we are done.
But what if the content is served by more than one CDN, as is the case for Netflix, served by LimeLight Networks and their own CDN (Open Connect)? What happens here when you open Netflix and select your House of Cards episode?
In this case, the first choice has unwittingly been made by you! Yes you. It turns out the hardware device you have bought constrains some of the choice of CDN. Netflix chooses to serve you differently depending whether you use a Blu-ray player, tablet, PC, AppleTV, PS3, etc. This includes which content (SuperHD, HD, …) is licensed for your device and whether it can do 1080p, but also how Netflix chooses to stream the traffic to you.
The second decision is made by Netflix for you, inside your home. The client you run to access Netflix makes a decision based on time, load, and other factors, and, based on this, selects CDN1 or CDN2 from above. Assuming they pick CDN2 which is interconnected in many cities, they also pick a city to stream from (though they may not always pick the closest).
Once the CDN is selected, the CDN operator chooses which peer to send your flow down. In the case of Netflix and Open Connect, this is also their choice. Once the data arrives inside your ISP’s network, they decide how to route it to your house, and then you watch and enjoy.
But you wonder, is this all academic? Wouldn’t all players always make the ‘best’ decision (and thus the same decision for the same inputs?). It turns out the answer is ‘no’ for a complex set of business and technical reasons.
CDN1 may always prefer Peer1 because they are cheaper. But cheaper normally has some reason, such as lower quality. CDN1 may be aware of this and turn a blind eye in order to reduce costs, or may have a difficult time diagnosing this problem or knowing when it occurs.
Still not convinced this isn’t all academic? Here’s an example. In this case, we had a Sandvine employee take two near identical devices; an AppleTV and an Apple iPad. They then used each of them to watch the same show at the same time, and then repeated this at several peak and off-peak times of the day. Interestingly, the AppleTV always chooses Netflix’s own CDN, Open Connect, at all times of day but my Apple iPad uses the LimeLight CDN at off-peak times and Open Connect during peak times.
At off-peak times, the AppleTV device used 18.104.22.168, which is ipv4_1.lagg0.c087.dfw001.ix.nflxvideo.net. The way you read this is ‘dfw’ == Dallas, and ‘nflxvideo.net’ means it is hosted by Netflix. At off-peak time the iPad used 22.214.171.124, which is cds846.dal.llnw.net. The way you read this is ‘dal’ == Dallas, and llnw.net means the content is hosted by LimeLight Networks. Keep in mind this is the same content, and near identical Cupertino devices, in the same home, at the same time. Given that a network’s cost is a function of peak usage, this means that there will be under-utilized capacity on LimeLight (and their Peers) at peak times, and there may be over-utilized capacity on Open Connect (and their Peers) at peak times.
So, did this matter? The Sandvine employee have an 105Mbps downstream connection (thanks Comcast) so they have the luxury of running two streams side-by-each. In each case, they were watching the same episode of House of Cards (poor man’s 3D!). Here, in the first two charts, we see that the iPad achieved roughly twice the quality (measured via bitrate) as the AppleTV (at the same time in the same house!). The only difference was the CDN that Netflix chose: the Netflix App chose LimeLight on the iPad, and Open Connect on the AppleTV. And could they see the difference? The most obvious difference was the stall on the AppleTV (shown roughly 3 minutes in, after which Netflix’s adaptive streaming switched to a lower quality stream and the stalls disappeared).
And, to double check, they ran this again later (during peak hours) and found that the iPad and the AppleTV delivered more or less the same (when they both chose the same CDN), shown below.
So, conclusions? Choices matter. It’s just not always obvious whose choice.