In a recent blog posted on Level 3 Communications’ own website, Mark Taylor VP of Content and Media at Level 3, spoke about how Level 3 Communications’ own networks work and even made a small plug about how the company has spent over $40 billion on building up their 180,000 miles of high speed fiber to help interconnect the internet. However, where their connections terminate with their peers is ultimately where they are the most vulnerable. However, Level 3 obviously does not cover the entire globe and their customers need to be able to connect to the whole globe, so there are peering agreements that allow for the sharing of data across these networks to enable fewer, better and faster connections from point A to point B. To understand how huge Level 3 communications really is, they actually provided a map of all the places where their fiber lies and where they may be leasing fiber to complete connections that they themselves haven’t laid or own.
Now, looking at the map above, the orange lines are built and owned by Level 3 Communications while the yellow lines are either owned by multiple carriers or are leased. But this map represents the entire global network that Level 3 has access to and can connect with other carriers or ISPs. Now, these guys, Level 3 Communications are sort of a ‘middle man’ in the internet’s grand scheme. They aren’t a middle man in the sense that they exist for no purpose other than to skim money off the existence of the internet, they are a middle man because they quite literally sit between your ISP and the destination of where you are trying to go. Or, they sit between the servers of the company that is serving you data and your ISP. Your ISP connects you to the internet and Level 3 Communications and other carriers are responsible for helping you get there. Now, Level 3 Communications themselves has tens of thousands of customers that they help connect to the internet with 51 peers in 45 different physical locations, listed below are a total of 66 actual connections (since some locations are repeated).
In all of Level 3 Communications connections globally, only 12 of those are actually experiencing any sort of bottlenecking or packet loss. And of those 12, 6 of them are already in the process of being upgraded to remove these bandwidth issues. Since after all, Level 3 communications has a total interconnection capacity of over 13,600 Gbps (1,360 10 Gigabit ethernet ports, plus a few smaller ones) they have distributed capacity with individual peers ranging from a single 10 GigE port to 148 ports, depending on the peer. The average number of interconnection cities per peer is five, but ranges from 1 to 20, so there is quite a bit of variance between peers and between cities. But what’s most important is that the average utilization of those interconnected ports is 36 percent, while congested ones near 90 percent and start to experience packet loss and connectivity issues.
However, these 6 peers that are problematic have been problematic for the past year on a consistent base and have chosen to do nothing about it. They maintain their interconnect utilization at nearly 90% all the time and as a result are dropping packets like crazy. 5 of those peers (companies/ISPs) are US based and 1 is in Europe. Level 3 communications did a pretty good job of explaining exactly how this problem manifests itself by showing one of their 100 Gbps connections to Dallas on April 3rd with a problematic peer and then one located in Washington D.C. that isn’t problematic. While Level 3 Communications has chosen not to name and shame these peers, I can tell you with 99% certainty that the Dallas peer is none other than SBC aka AT&T. So, while Level 3 didn’t directly name and shame their problematic peers, we already know that one of them is AT&T and it isn’t going to be much of an assumption to think Comcast is going to be among that list as well. However, we have pretty concrete evidence from Level 3 that AT&T is doing a piss poor job of maintaining their interconnections with Level 3.
First, you can see the congested peer… (AT&T)
Followed by the uncongested peer….
Notice, they are both getting close to 100% utilization, but the uncongested one is not dropping any packets or having any sort of errors because it isn’t plateauing and getting overutilized. But even in this scenario, Level 3 communications is working with their peer to augment the connection to improve the performance so that there isn’t a chance of packet loss in the future.
Overall, Level 3 Communications is further making clear as to why providers like Netflix are crying out against companies like Comcast and speaking out against the Comcast – Time Warner Cable merger (both probably on Level 3’s list), as well as how such behavior is affecting Net Neutrality and how certain companies are using neglect to force companies like Netflix to pay for dedicated lines into Comcast’s network even though companies like Level 3 are already doing so and their interconnects are merely being neglected.
If you’re interested, here’s a list of the best and worst ‘middlemen’ like Level3, which according to these claims, is the best.