When I first got my iPhone 5 last October it was my first LTE phone, so the very first application I ran on it was the Speedtest.net. When looking at some 1H 2013 Phenomena data I noticed I may not be the only one to do this, because on one LTE network in Europe, Speedtest.net was the eighth ranked upstream application on the network accounting for over 2% of traffic at peak period.
Looking at this behavior, I think a lot of consumers may believe that speedtest.net is the one true way to measure the speed of your internet connection, and that is simply not the case.
Below are two tests I ran from my home. My results are pretty consistent, no matter what time of day the tests are run, with one speed test server always under-reporting the other by a large margin.
As you can see, one server gives me 45Mbps of downstream, the other 9Mbps. I currently pay for 35Mbps service, so depending on my world view (and server selection), I may want to head down to my ISP’s head office with either a box of chocolates or pitchfork in my hands in order to praise or complain about getting more or less than what am I paying for.
So what is the cause of this discrepancy? It can be related to server performance: it takes a big machine to drive this amount of bandwidth to all the users testing it, or, it could be the upstream ISP paths. I happen to know (using traceroute) that both Speedtest.net servers I used are in the same building (277 Lancaster W, Kitchener, ON: one via ‘Megawire‘ and one via ‘Netflash‘), and that those servers are connected to me in quite a different fashion.
If we look at the map below, we can visualize my traceroute results to see the very different paths taken for each test (server A in black, server B in red).
What other well-known performance metrics might not tell the whole Internet speed story? Netflix’s ISP Speed Index which ranks network performance may be one. It measures average speed for Netflix streams in Mbps, and not the capacity available to the subscriber. This means that much like with Speedtest.net, these results can be influenced by a number of factors include the devices used by subscribers, but also the various routing relationships ISPs have with the CDNs used to store and serve Netflix’s video content.
So what is the moral of the story? Don’t believe every Speedtest you take. The access speeds offered by your ISP may be good, bad or ugly, but Speedtest.net is not necessarily the best or only measure. Quality is an end to end chain, if any component (server or interconnecting network) is not working optimally, it won’t matter how fast your access is.
For those wanting more detail, below are additional tests we had Sandvine employees from around the globe run, which resulted in similar findings.
Sao Paulo, Brazil
We can see there is one server which is an outlier (~110ms and ~20% slower) for the same time and same consumer location.
We see there is also one server which is an outlier, ~30% slower in bandwidth than the others (interestingly its latency is lower than one of the fast servers suggesting this might be a server issue more than a congestion issue).
Denver, Colorado, USA
We also see very different results for the same time period.
Raleigh, North Carolina, USA