Merry Christmas, Twitch viewers! Or, at least, that’s how it’s supposed to go, right?
For those who are users of Twitch, the company posted on their Blog earlier today about changes to the company’s operating status in South Korea. Now, previously, there were announcements from the company about discontinuing VOD services, dropping the maximum resolution of streams to 720p, and other measures such as exploring Peer-to-Peer delivery methods, and moving CDN resources out of the country in order to reduce their cost of delivering video to website visitors inside of Korea. The reason for these actions has to do with a controversial law in Korea called the “Network Usage Fee Law” which operates in a “Sending Party Pays” methodology.
Sound familiar?
It sure is. For those who have been following the Broadband Communications industry, time and time again, companies decide to make attempts at further monetizing their networks, sometimes in an excessive manner. I like to call it “Network As A Billing System (NAABS).” A more familiar term that this is bringing up is Net Neutrality, the concept where all traffic on a network is treated equally, regardless of the contents or the origin, so long as the connectivity has been paid for.
The major proponents for Net Neutrality tend to be the every day Internet users; you and I. People who use the Internet for entertainment, communication, as a means to make a living, and for improving society. People who don’t want to understand what a Megabyte means, how much a Megabyte eats into a data cap and when, and just want something that is dependable, fast, and easy to understand (flat rate pricing!). Major companies who are built around a neutral and stable Internet such as Google, Yahoo!, Twitch, and well, virtually any company on the Internet today, are also (hopefully) major proponents for net neutrality, since practicing neutrality allows them to continue to develop services, and meet the demands of their end users (you and I). While a well managed network isn’t necessarily 100% neutral in how traffic is treated, we can count on traffic being treated fairly and not singled out harshly.
The opponents to Net Neutrality are often Internet providers with a conflict of Interest. In the US, companies who have regularly argued against and have lobbied against Net Neutrality bills include AT&T, Verizon, Comcast, and networking equipment vendors such as Cisco, Panasonic, and Ericsson (WTF? They don’t want to sell faster hardware and more licenses?). By conflict of interest, I mean a few things. These are often incumbent providers who will go through great lengths to create red tape for competition attempting to bring faster and cheaper service to customers [Link 1] [Link 2]. These are the providers who will go through great lengths to mislead customers from knowing the exact technology being used to deliver a service (Is that coaxial cable magically Fiber? Wireless Fiber? No way!), the exact speeds one can expect (What are upload speeds?), and what the actual month to month pricing is [Also link]. The biggest culprits are those who sell TV Service over the same infrastructure, too. Guess who sells TV Service AND is an Internet provider caught up in these net neutrality related debates? Verizon, AT&T, Comcast… and SK Broadband. The TV Part gets me, as the TV Service from all of these providers utilizes their IP and last mile Infrastructure in some way, shape or form. They even use the same public CDN infrastructure (such as Akamai, Amazon CloudFront, etc) that companies such as Twitch use to deliver content to end users. Sites like Twitch are merely video alternatives to traditional TV.
Why is this whole CDN thing important?
Twitch is a massive streaming platform. They push a lot of Internet traffic as demanded by their visitors, and likewise, they should pay a lot into Internet infrastructure in order to ensure their viewers and streamers are receiving a high quality, lag-free stream. This is exactly what Twitch does, by constructing what is called a CDN (or Content Distribution Network) with servers close to key geographic locations of their visitors. These CDN servers are often established at major Internet exchanges where major and even minor ISPs all come together to connect their networks together, so the Internet is… well, the Internet. At these exchanges, Twitch rents and pays for space, electricity, and support for their servers. They pay for their DNS infrastructure, for IP address space [here’s their AS!], and their network operations teams. They may operate their own facility, or co-locate space in a shared data center. They also pay for their connectivity and bandwidth, which at their scale, may be tens to hundreds of Gigabits a second per installation, per region. Twitch may be present at dozens of key exchanges around the world. When you are paying for 10Gbps, 40Gbps, or 100Gbps at an exchange, you are often doing so across multiple providers. For example in Los Angeles, Twitch may buy 40Gbps of connectivity to Verizon, 100Gbps of connectivity to AT&T, 40Gbps of connectivity to Comcast, and then establish peering with their networks directly. Twitch has a DIRECT shot to customers on these ISPs. For ISPs where peering and transit isn’t directly purchased with, Twitch may then purchase or establish generalized peering at an Internet exchange… let’s say 80Gbps, for a Tier 1 or Tier 2 provider like Level3 to carry their data to smaller networks that cannot afford to give Twitch and other content networks direct connectivity.
Now, this is an extremely simple view of the entire situation. What matters here is the fact that a CDN is constructed to support the scale of the operation without causing harm to other Internet traffic by taking up capacity where it doesn’t need to be. Efficiency is key, and saves EVERYONE including the Internet providers money by distributing the work across infrastructure. Without CDNs, the Internet would be very slow. CDNs may use centralized servers in some cases, such as for stream ingest and encoding, but that data will be copied and distributed quickly and efficiently to the CDN severs located closest to where it is being demanded, using as little bandwidth as possible in between (since again, bandwidth costs money).
The ISPs such as AT&T, Comcast, Verizon, and SK Broadband use a similar architecture for their (streaming) TV Services. AT&T TV (now DirecTV Stream) relies on Akamai’s public CDN Services. Comcast X1/Xumo similarly relies on a CDN of some sorts, likely from Akamai or Amazon Web Services. SK Broadband has similar setups for their IPTV service (did I say IPTV?), and Verizon has an IP CDN set up for their FiOS TV On-Demand, and out of home streaming. The end goal is to ensure their TV Services work reliably, are fast, and are efficiently delivered by not congesting the middle mile transit in between cities, with thousands of customers trying to watch bandwidth hogging TV. The architecture and design of content delivery you might say, is standardized by years of experience and mistakes (more on this later).
One little tidbit I will also emphasize here is on the usage of public CDNs. Above I had mentioned AT&T / DirecTV Stream utilize Akamai, a public CDN provider for their live streams. For DVR Storage, DirecTV also utilizes Akamai for the front-end but may store VODs in their own data centers. For VOD Storage and serving, Twitch utilizes Amazon’s CloudFront service, which is also a public CDN (Twitch is owned by Amazon). Twitch however, operates their own CDN for live streams. Sometimes CDN services are outsourced to specialized providers where the expertise, connectivity with other providers, and costs, are managed. Other times, whether due to flexibility, cost, or technical requirements such as with Twitch’s bandwidth usage and low latency streaming technology, services are in-housed and managed internally. TV providers like AT&T/DirecTV Stream may utilize public resources because their expertise is not in content delivery. Twitch using a public CDN is the same as AT&T using a public cloud, is the same as SK Broadband using a public cloud, etc. Bandwidth is bandwidth as long as it is paid for.
This is where the conflict of interest is now established, and helps to explain the flawed basis of Korea’s “Network Use Fee” and why consumers and certain companies alike are against it.
Who pays for all this data?
Anyone who connects to the Internet. For example, a paying home Internet customer will pay $60/m for their 300Mbps Internet connection, with an agreement that the data is unmetered within reason, but the bandwidth is shared with many others and not guaranteed. Twitch will pay for their bandwidth as I established previously. They pay into dedicated bandwidth through complex agreements (Gigabits/second with or without per-bit metering) to various points (different from what a consumer might pay), and from there, they ride shared connectivity to the customer. If traffic to/from Twitch or a customer has to cross multiple Internet providers, the telecom providers in between will have their established connectivity agreements to ensure they are able to exchange traffic between each other, and will pay for upgrades to capacity when needed using the money received from their customers. Where there are not multiple Internet providers involved, such as what is common in a CDN architecture, Twitch and the end user pay for the entire path through the ISP, and capacity is managed based on usage on both ends. No matter what, the infrastructure is shared, and as demand increases for bandwidth, each end of the connection will pay for more capacity into the Telecom provider naturally to increase capacity. There is never a need for a “Usage Fee” since existing capacity pricing already accounts for this.
Unfortunately, in a “Sending Party Pays” environment, the end consumer is the ultimate payer. The network operator enjoys their double dipping rewards.
Has ISP vs. Content Providers vs. Consumer happened before?
Yes. In fact, an example I can think of is in a case which occurred long ago, during the rise of Netflix in the United States. When Netflix’s streaming services were first established, Netflix relied on Level3 (now Lumen Networks) as their primary bandwidth provider for their Content Distribution network. As Netflix usage exploded, network capacity between Level3 and other providers began to choke. Video began to buffer. General Internet traffic crossing between Level3 and other networks began to slow down. One prominent case I felt first hand was in Level3 vs. Verizon.
Level3 and Verizon are both classified as “Tier 1” providers. Both providers carry significant portions of Internet traffic between themselves, other Tier 1 networks, and to and from smaller ISPs without national or global backbones. Tier 1 providers typically establish what is known as “Settlement Free Peering” where connectivity between the two providers are established at a low to minimum cost, and with a basic agreement that traffic to and from each other remains balanced. Balanced meaning, Verizon sends as much traffic to Level3 as they receive, and vice versa. This structure helps to ensure that everyone can reach everyone on the Internet, and incentivizes Internet service providers to be neutral towards the types of applications and customers which are running over their networks.
When Netflix began sending video to the masses, traffic between Level3 and Verizon was sent out of balance, with Level3 sending far more traffic to Verizon than they were receiving. Connectivity between the carriers became congested. In order to resolve the imbalance, one or more things had to happen. Either Level3 needed to pay Verizon to obtain additional connectivity, or Verizon needed to have enough traffic being generated to send back to Level3, so upgrades could be performed in a “settlement free” manner. With Verizon being an “eyeballs” Internet provider that end users subscribe to Internet access with, they also hold an obligation to their customers to deliver fast and reliable service, and that meant ensuring Netflix did not buffer to any fault of their own, but this also means it is hard for them to generate enough return traffic to remain in balance.
However, customer complaints began to rise. Providers such as Verizon began to become accused of throttling Netflix due to extreme buffering, while Level3 accused Verizon of withholding upgrades needed to serve their paying customers. Verizon replied back in response to explain why capacity issues were occurring, with end users ultimately being at the crosshairs of everyone for requesting a bandwidth heavy service. At the time as a Verizon customer, I was upset at Verizon and Level3, accusing them of being unfriendly because the congestion was impacting other services that traveled through Level3. However, at the time I did not realize that Netflix had constructed their content network in an inefficient manner which led to the problem.
The long term solution ended up being the construction of a Content Distribution Network, placing servers close to customers at key points. Netflix established the OpenConnect program, which allows ISPs to host Netflix content servers within their network in order to improve streaming performance. Additionally, where OpenConnect appliances were seen as undesirable and as an ask for free colocation and bandwidth, Netflix needed to diversify their transit providers to spread traffic across multiple providers. The results of which can be seen here and here.
Similar efforts have also been performed by Google/YouTube (another company who has experienced significant congestion/throttling backlash), Facebook/Meta, and likewise by established CDN providers such as Akamai, Cachefly CloudFlare, Edgecast, and Amazon CloudFront, who are instrumental in making the Internet work as fast as it does today.
Let’s talk about the issue with Twitch Korea and Net Neutrality. What’s going on?
Now that we have established some history and knowledge, we can talk about the situation with Twitch Korea. In essence, Twitch and Twitch’s users already pay the network providers in Korea for access to the Internet. The carriers, such as SK Broadband, and KT (Korea Telecom) are responsible for delivering traffic in a neutral manner. As in, regardless of the traffic being video, peer-to-peer, file transfers, gaming, conferencing, or what be it, they deliver the traffic from the sender to the recipient within all reasonable capacity limitations, and with a level of quality of service. As Twitch and consumers pay for faster connections, and as technology improves, the carrier improves delivery with faster, more efficient hardware, and by implementing network management (such as active queue management) where needed to carry all traffic as fairly as possible. Twitch has built their content network to efficiently deliver video, without bringing Korean internet to a halt.
The Network Use Tax is effectively a targeted double dipping. For content networks which deliver video, additional access fees are charged despite an access agreement for a specific amount of bandwidth being established by a provider’s customer on both ends. This is primarily on the basis of video. To a video network, this is like saying, despite ordering 10Gbps of connectivity, if you send video over it, you now have to pay $10/TB. But if you send web pages, you simply need to pay for 10Gbps of connectivity.
Likewise, let’s say Twitch were to switch to a Peer to Peer delivery method. Twitch sends video to a few ISP customers, and those customers seed and distribute streams to other Twitch viewers on ISP networks. Twitch has paid for their connectivity and has all of the bandwidth they need. Customers have paid for all of the connectivity they need. But Twitch now has to eat a $10 per 1TB of video charge despite Twitch users distributing video to others over Peer to Peer. Maybe Twitch viewers will end up paying extra as well, just because they distributed video over Peer to Peer while visiting the site.
This sort of practice is generally inexcusable. The cost of upgrading networks should already be paid for by the purchasing of transit services from an Internet provider. If it’s not being covered, then that is a failure in basic business practice.
Remember earlier when I mentioned that SK Broadband is a TV Provider? Yes, their TV Service is IP-based. Just like Twitch’s services. It rides the same exact infrastructure. It uses a CDN architecture just like Twitch does. Both have paid into capacity, except one is being favored due to being the provider’s own service. Also remember earlier when I mentioned an “eyeballs” network? Likewise, SK Broadband is an eyeballs network, because their customers are those with eyeballs which consume content primarily. It is impossible for an eyeballs network to return traffic back to Twitch to balance the equation. But yet, Twitch and the eyeball customer pay the Internet provider in order to access the content.
What can Twitch do now?
That is still being established. At this point, it’s looking to me like Twitch may have to block all Korean IP addresses from streaming to and from the site. This of course will depend on Korean law and how it covers entities such as Twitch under the Network Use Fee. Twitch will likely deliver news to us when they can.
Another plan explored by Twitch was serving all streams from outside of Korea and removing their Korean CDN presence, relying on the limited capacity entering and leaving Korea by means of Tier 1 bandwidth providers. Emphasis on limited connectivity here, as forcing Twitch to send their traffic over Tier 1 providers will create massive inefficiencies, significantly decrease stream performance in the country, as well as result in the same issue that I discussed earlier in Level3 vs. Verizon vs. Netflix. Connectivity as it is to and from Korea is already heavy congested in my own observations. Hey, forcing inefficiencies with these “Internet fees” between Internet providers could become bad enough to create a peering dispute, which can lead to Korea becoming disconnected from portions of the Internet. I’ll cite Sprint vs. Cogent for this one, since I got to experience multiple services I use going offline and unreachable for several weeks unless I used a web proxy or VPN when that dispute happened.
Twitch isn’t the only company experiencing hostile behavior by Korean Internet providers. Netflix, Meta, and Google have also been in the crosshairs as major producers of Internet traffic, but producers by demand of Internet providers’ own customers. Coincidentally, these all seem to be foreign companies being targeted, and I have not heard of similar feuds with domestic companies such as Naver and AfreecaTV. Coincidentally, stock prices for those companies have gone up following the Twitch announcement. Are these companies getting specialized treatment to the bandwidth in Korea?
Are they simply folding and paying up, and perhaps Twitch should fold and pay up? If you want my opinion… nope.
