I’ll open with this; chances are if you are reading this from a Sky Cable Philippines Broadband connection then it probably took well over a minute to load the page, that’s if the page even opened at all! The reason for this is that this website is hosted in the UK and to connect from your Sky Cable connection to this server you would have likely been routed via ae-13.r01.tkokhk01.hk.bb.gin.ntt.net (220.127.116.11) which amongst a growing number of other hops seems to grind to a halt. You would be forgiven in thinking “If this is happening outside of the Philippines then why are you singling out Sky Cable?”, the answer is because this ONLY happens with Sky Cable and not any other Philippines ISP, and this issue is starting to affect many other websites including Apple’s download servers, Amazon S3 Hosting, and even Sky Cable’s own website hosted in Singapore. All of these sites when served or routed through certain CDNs/Hops in Hong Kong or Singapore will not be able to serve you traffic above 20~100KB/sec, or in the case of Apple a slightly better max speed of about 300 KB/sec. That’ll take you well over 5 hours to download an iOS update!
Let me go into more details, and let me get some things out the way first, because I am sure there will be things you are thinking before we get to the end of this post.
- My current line speed with Sky Cable in the Philippines is 55Mbps which works out at around 6.875MB/sec when downloading files from the internet. If you do not know the difference between Mbps and MB/Sec then please go read this.
- I also have a 5Mbps PLDT line which maxes out with a file download speed of about 600 KB/Sec (Just over half a megabyte a second).
- That means the PLDT line is 11 times slower than the Sky Cable line, yet as you will see later on, the PLDT line can reach download speeds on problematic servers well above what Sky Cable can.
- All my tests are fully repeatable at ANY time of the day, there is zero argument for network traffic being the cause of these issues.
- These issues have been confirmed by two other people using Sky Cable lines.
- I had been trying to get Sky Cable to acknowledge this issue for nearly a year without anyone understanding what I’m talking about.
- Finally a month ago I started speaking with one of the head engineers for Sky Broadband via Facebook who initially acknowledged and confirmed the issues. I’ll be referencing some of my conversations with him later in the post.
- NOTHING has been done to fix these issues and I am sure they are affecting the majority of Sky Broadband customers in some way, the only reason why this has gone unnoticed for so long is because the median speed of Philippines internet users is so slow that everyone is used to things taking an age to download.
- I’m certain that if this was a UK or USA ISP that it would be all over Social Media, and in the USA you’d probably see a class action lawsuit or something due to the lack of coming anywhere close to providing a service to these servers that could be deemed reasonable.
Where did all this start? Why are Sky Broadband downloads from Apple so slow?
About a year ago I had noticed that downloads from Apple servers were going really slow at about 50 KB/Sec, I took to Twitter to see if anyone else had the same issue and sure enough there were a couple of posts talking about the same problem, so I responded…
— Dean Chew (@DeanChew) September 16, 2016
Side note: Let me get one thing off my chest now. Sky Cable’s technical support is diabolical. They are very nice, they take the time to respond but they literally have no technical training whatsoever. It doesn’t stop at the support guys on Twitter though, even if you call them and have them send out a network technician you will end up with a guy coming round your house who has had about 2 hours on-boarding training and has been told that if things don’t work to “reset the router”. Reset the router will forever be burned into my eyes and ears as it is Sky Cable’s only solution to ANY problem. Having relationship issues? Reset the router. Got diarrhea? Reset the router. The devil has appeared in front of you and is about to cast you down to the depths of hell? Reset the router my friend 😉
Right, so back to Apple. As I mentioned I saw that the issue was only happening on my Sky Cable line but it was happening to ANY Apple download server. So that would be my Macbook updating, my iPhone updating whilst connected to my home Wifi, etc. Obviously when I took my iPhone off the home network and just using Globe’s LTE the downloads would fly down, how odd.
From there I thought that perhaps Sky Cable were being sneaky and were throttling traffic for some reason, so I loaded up my VPN and connected to a LA server. Sure enough the download speed from Apple servers ON THE VERY SAME LINE jumped up from around 50KB/Sec to about 3MB/Sec – over 60 times faster!
.@SKYserves Have 2 use VPN to LA 2 get acceptable speeds to Apple updates. <50kbps without VPN, >3.3MB/Secwith VPN. What r u doing 2 traffic?
— Dean Chew (@DeanChew) September 16, 2016
There is also a YouTube video that shows this very thing happening. Slow speeds to Apple, VPN on, Slow speeds go away.
I showed this to Sky Cable, I showed this to the technicians they sent to my house, and we went round and round in circles for about another month before I just gave up and hoped that somehow it would get fixed. How wrong I was, the issue was just going to get worse.
More websites start getting slower and slower.
So between late last year and now I noticed that more and more websites were getting slower, not just page loads but actual downloads. I posted in the Philippines Internet Facebook group but unfortunately my post was met with confusion and the majority of discussions are currently around Mobile Broadband, however it did catch the eye of a Sky Cable head Engineer (who we’ll just call “*Joe”) and we got talking about the problem and trying to troubleshoot it. The next thing that I found out about the problem was pretty confusing for me, it would seems that changing the DNS server used would have a direct impact on increasing download speeds, but some of these DNS servers would work for a little while and then stop working. I decided to test these DNS servers from my Sky Cable line and also test them against my PLDT line, which if you remember has a max speed which is 11 times slow than my Sky Line.
*Just a note about Joe, I’ve got to say that the guy has been more than nice in entertaining me when asking these questions. He could have easily just kept quite but he is at least reaching out to aide in some way. So if you ever read this, Joe, I do say thank you for your help despite my frustrations with Sky Cable as a company.
Now the image above is pretty in your face, but it’s actually quite easy to understand, a few points…
- I tested the same set of servers on both my SkyCable line and my PLDT line 10 times
- Each test set was tested with the DNS settings changed. I used the 10 fastest DNS servers from my location as per a DNS tester.
- These test were done late at night around midnight, network traffic congestion at this time is almost none existent.
- The servers I tested were…
- Fast.com (This was a test control server to show full speed line capability)
- NVIDIA Driver download (This was a test control server to show full speed line capability)
- Apple update server (which seems to be hosted in Singapore)
- Put.io Hong Kong (a simple hop from the Philippines)
- I also included a couple of lines of the TraceRoute to see if there was anything obvious that was going on.
So what does this table actually show us? Well there are two things that stand out straight away.
- With the problematic servers my PLDT line (11 times slower than the Sky Cable line remember) almost ALWAYS maxed out at around 500 KB/Sec whilst the Sky Cable line could barely get above 90KB/Sec in some cases.
- Simply changing the DNS server used on the Sky Cable line can result in much faster downloads if you manage to get the right DNS server. However I can tell you from my own experience that this solution doesn’t tend to last very long.
- The DNS Server switch might just be a red herring for Apple servers. I think there is perhaps two separate issues which is everything Apple (~200KB/Sec), and then everything not Apple (< 20KB/Sec)
So with the above data I went back to Joe (the Sky Cable engineer) and showed him what was going on. I asked a couple of questions…
Why are Apple Downloads so slow with Sky Cable? and How come PLDT aren’t having the same issues? This was his response…
We’ve escalated the issue with Apple. we can get more bandwidth through multiple connection. seems that they are limiting our connection.
I followed up on this and asked for some clarification and was told…
For Apple, seems that they have limited the speed per stream. And the updates are http. We can go as high as 2.5mbps per tcp connection. Or about 10mbps for 4 tcp connections. We also think that PLDT has a transparent caching for http traffic that’s why you can have the full bw. For other sites we are looking at direct peering with thier respective CDNs in HK or SG.
Ok, so a response at least and Sky are looking into it. However here is a question, why on earth would Apple limit the download speed to a national ISP? Especially when Apple make billions of dollars through their App store and iTunes stores. What possible sense does that make?
Secondly, if PLDT have this so called transparent cache then why don’t Globe and Smart have the same issue that Sky Cable have? The issue is only with Sky Cable, not with any other Philippines ISP.
When asked about the DNS, i was told…
Pointing to our DNS ensures that the CDN will point to the nearest node. if that node is congested, then we may experience low bandwidth.
Unfortunately my further questions went unanswered so I decide to conduct some more tests and collect a little more data to more websites.
Now what I have done here is expand the test to 13 total servers, and as I mentioned in the intro we have some heavy hitters here like Amazon S3, Reddit, and Sourceforge. I’ve not bothered to test these against the PLDT line because you can take my word on it and see from everything I have written above that there is no issue with any other ISP.
So what does the above tell us?
- My two control servers, Fast.com and Nvidia are still pumping out max download speeds.
- More and more websites are being pulled into this Sky Cable issue.
- The average download speed from these servers is 91.27 KB/Sec or a line speed of 0.72MBPS, these speeds are reminiscent of dial up internet from 25 years ago!
- The slowest connection which is 5.5 KB/Sec is 1,250 times slower than the line is capable of, and must be hundreds of times slower than the average delivery speed described in Sky Cables’ contract.
- That 5.5KB/Sec speed means it takes 30 minutes to download a 10.8MB file. It’d take you 3x longer to download three Spotify songs than it would to listen to them!
- A 73MB file (probably about the average iOS app store App) takes over 2 hours!
- At this speed it would take you around 101 hours to download an iOS update from Apple.
- Doing traceroutes to each of these servers shows that they are CDNs serving from Hong Kong or Singapore.
Whilst not in the list Sky Cables own website is hosted in Singapore and can be affected by this issue, let’s have a look at the Traceroute for their own site from their own line…
traceroute to mysky.com.ph (18.104.22.168), 64 hops max, 52 byte packets
1 192.168.0.1 (192.168.0.1) 3.056 ms 2.510 ms 2.353 ms
2 22.214.171.124 (126.96.36.199) 14.490 ms 18.795 ms 19.823 ms
3 ge-1-5.sj1.skybroadband.com.ph (188.8.131.52) 18.360 ms 15.155 ms 23.655 ms
4 184.108.40.206 (220.127.116.11) 8.130 ms
18.104.22.168 (22.214.171.124) 18.952 ms
126.96.36.199 (188.8.131.52) 14.599 ms
5 184.108.40.206 (220.127.116.11) 34.346 ms 15.106 ms 16.821 ms
6 ix-xe-2-1-0-3-0.e453.net (18.104.22.168) 43.291 ms 41.665 ms 42.000 ms
7 if-ae-16-2.tcore2.svw-singapore.as6453.net (22.214.171.124) 72.434 ms
if-ae-32-2.tcore2.svw-singapore.as6453.net (126.96.36.199) 92.321 ms 78.678 ms
8 188.8.131.52 (184.108.40.206) 77.894 ms 74.093 ms 72.200 ms
9 * * *
10 * * *
11 220.127.116.11 (18.104.22.168) 65.758 ms 100.979 ms
22.214.171.124 (126.96.36.199) 70.910 ms
12 * * *
It all looks to be ok with hops times that should be fine, but I suspect the issue is around ix-xe-2-1-0-3-0.tcore2.hk2-hong-kong.as6453.net (188.8.131.52) because it seems that tcore1 or tcore2 for HK or SG always causes issues. For Singapore I have also found that Equinix also has major issues, 13335.sgw.equinix.com (184.108.40.206) 41.833 ms 54.802 ms 49.804 ms.
So I gave all the info to Joe, and basically what I got back was this…
We are now correlating the destination IP. there must be a common choke point somewhere.
We will also be connecting to a major IX in the US and EU this 3rd Qtr. Lets see how will this improve our connectivity.
Yes Joe there must be a choke point somewhere, why have Sky been unable to fix this despite it being an issue for over a year? Are we really just supposed to cross our fingers and hope that some new IX will solve this issue? Why do you not know what is causing it right now?
Joe also had this to say about Equinix…
We are looking at our Equinix connection. we just activated our 2nd 10G in Singapore early this year and it’s not balanced.
Why has this not been addressed yet? It surely couldn’t have been connected correctly in the first place?
After writing this entire post I did ask Joe again about the ix-xe-2-1-0-3-0.tcore2.hk2-hong-kong.as6453.net (220.127.116.11) hop which is NTT owned.
Our NTT link are down since last July, most of the traffic are routed to pccw now.
So we are starting to get somewhere now, it is all still a bit of a mystery but here are some facts that have been confirmed by Joe and are obviously having an impact on Sky Cable’s ability to deliver international network traffic into the Philippines…
- Sky cable know there is an issue connecting to Apple servers but have been unable to resolve it.
- They have an issue with a line to SG Equinix “earlier this year”, but again have been unable to resolve it.
- They have a completely down line link to NTT since July and claim that traffic to Hong Kong is being routed to PCCW, but obviously this isn’t the case else the hops to this very website wouldn’t be showing in the traceroute. Again, they have been unable to resolve this.
Over to you for your help.
I have tried to get this issue resolved and I have tried to get other people to understand that there is an issue but I have hit a brick wall and I need your help.
- If you are using Sky Cable could you please open this Google Spreadsheet.
- Enter your name on a new line, then click each of the links in the header row.
- These links will open downloads, save the downloads and then open your download manager which is usually CTRL(CMD)+J for Firefox or CTRL(CMD)+Shift+J on Chrome
- You can then see the download speed that is shown in the download manager. Simply enter this into the spreadsheet.
Hopefully with enough data we can get some exposure on this. Maybe it turns out that it is just my line that has an issue, maybe it is just a set of customers in a certain area, but maybe it is an issue which is affecting a lot of people and instead of us all moaning about Philippines internet speed, we might be able to do something to fix an issue that a 8.7billion revenue per year company just hasn’t been bothered with.
Feel free to comment below, and if you have knowledge in the network field then I am happy to try any other tests. I appreciate you reading such a long post 🙂
There was a little bit of traction from a Reddit submission in /r/Philippines which meant that some users added their line test speeds to the Google sheet above. Around the same time I got confirmation from a Sky Cable service rep that Sky Cable Guarantee a delivery speed that is at least 40% of the line speed. So in my case on my 55Mbps line, all traffic should be delivered to me at 22Mbps or 2.7MB/Sec. We can quite clearly see that this isn’t the case for both myself, but also for the vast majority of others who added in their test speeds. So Sky Cable aren’t just delivering a crappy service, they are in breach of their own contracts!
I also decided to document a “day on the life of Sky Cable” for a day, this honestly is no hyperbole.
Check video recap from last night at http://optionalpha.com/ – 20KB/Sec
Login to internet banking with HSBC – 20KB/Sec
Check Facebook – 200 KB/Sec (woo!)
Check Twitter – 20 KB/Sec
Download iOS app update – 250 KB/Sec (Woo!)
Load Reddit – 500 KB/Sec (Woo!)
Load images from imgur – 500 KB/Sec (Woo!)
Load images from i.reddit.com – 200 KB/Sec
Check this blog post – 9KB/Sec – Unloadable, have to switch to PLDT line
Do some work which involves looking at a selection of about 100 websites or so, at least 30% will have the 20~200 KB/Sec issue.
Download some podcasts – 100 KB/Sec
Try to listen Calm Radio – 20 KB/Sec – unusable, switch to Glove or PLDT line.
Watch YouTube 2MB/Sec – works for the most part with 1080p
Watch a streaming soccer game – Almost all streams buffer due to Sky issues, switch to PLDT.
So in a typical day there are 8 websites that I visit all the time, and all 8 of those have issues with Sky Cable, issues which basically make them unusable. How is that remotely close to 40% speed, and how is it justifiable paying 4,000PHP per month for broadband?
I knew it was a waste of time doing this, I know it would just get me angry, but I did it anyway. I sent an email to SkyServes@mysky.com.ph…
Here is is below, images have been removed as they are just copies of the ones above.
Date: 8/31/2017 6:33:00 PM
Subject: Letter of complaint due to breach of service contract by Sky Cable
Please let me preface this long letter of complaint by saying that it should find its way to the correct person within Sky Cable who 1) Has the technical knowledge to understand the points being made, and 2) Can process complaints and action remedies to them. What I do not expect is a front line customer service rep to email me back asking to restart my modem or to book a engineer to come to my house, and nor do I expect to be ignored.
Account Number: XXXXXX
Line Speed: 55Mbps
I am writing to you today to voice my continued issue with Sky Cables broadband service which is in breach of your own service contract which states that you will guarantee 40% of the line speed. This breach of contract is not only affecting me personally but is also affecting a wide range of Sky Cable customers and is due to international routing problems that your network has when connecting to CDNs in both Singapore and Hong Kong. Large websites such as Apple, Amazon S3, HSBC, and even your own SkyCable.com.ph website are all affected by this network issue and in cases will NOT load at speeds higher than around 20KB/sec. The 20KB/sec speeds that these websites are consistently loading at are around 0.8% of my line speed, well below the supposed guarantee of 40%.
Not only have these speeds been recorded by multiple customers, I also have quotes from one of your own engineers that admits there are fundamental problems with the Sky Cable network and how it connects to Singapore and Hong Kong.
A lot of the data I am going to be referencing has already been covered in a blog post that can be found here – http://www.chewie.co.uk/general/sky-cable-philippines-slow-broken-internet-service/
So as I have mentioned at the start, the problem seems to be that when Sky Cable connects to a web server which uses a certain CDN in Hong Kong or Singapore then the speeds at which it connects at become unusable. You can see a sample of websites which uses one of these CDNs in the table below…
Inline images 1
Now what this table shows us is that with a 55Mbps line we are getting 4% of the line speed or less when connecting to 11 of 13 test servers. These speeds are so low that it takes over 2 hours to download a 75mb file. This speed is 0.14% of the line speed and is nowhere near acceptable. When connecting to these servers the issue can be replicated 24 hours a day, any day of the week, this is NOT regular network congestion, its an issue with your traffic delivery.
I have also tested connections to these severs using different DNS settings alongside connecting to them on a 5Mbps PLDT line….
Inline images 2
As per the image above you can quite clearly see that the PLDT line which is 10x slower than the Sky Cable line has no problem whatsoever in reaching full speed to these test servers.
At this point I want to let you know that I have been doing everything I can to bring this to Sky’s attention and get the issue fixed. I started by connecting you about this issue with Apple over a year ago, and the past three months I have been sending data on this problem to one of your engineers, NOTHING is being done to rectify the issue despite my good faith.
I am sure that you will be assuming that this is just an issue with my line, and is localised, but this is simply not the case, multiple users are documenting the same problem…
Inline images 3
The above image clearly shows that out of 84 tests to these servers only 5 of them will connecting at a guaranteed speed of 40% or higher. The other 79 test conducted are speeds that are so slow they are unusable. This issue is replicated time and time again with your service in cases such as having to switch to Globe LTE to stream music on a cell phone, or having to use the PLDT line to download an Apple update, the issue is affecting every aspect of internet browsing and you are simply not delivering on what you say you will in your contract.
Now I have quotes from a Sky Engineer that state you know of issues…
“We are looking at our Equinix connection. we just activated our 2nd 10G in Singapore early this year and it’s not balanced.”
“Our NTT link are down since last July, most of the traffic are routed to pccw now.”
But these issues are not fixed, and data that is being sent to you by your customers is being ignored.
So to recap here are some facts about the situation…
1) My line speed of 55Mbps costs me 4,000PHP per month
2) The guaranteed speed that you have in your contract to provide for that 55Mbps line is 22Mbps, this is a transfer speed of 2.75 MB/Sec
3) The charts above show connections to large companies like Apple and Amazon are hundreds of times slower than that speed, and in cases are over forty times slower than your supposed guaranteed speed
4) There has been no effort to fix this issue despite information being given to you
5) There has been no compensation or act of goodwill from Sky Cable for breaching the contract and not delivering a service that you say you will
As such, I do expect a response from you about how this is going to be fixed. I have been paying for a premium internet service for the past year and a half and you simply have not been delivering it. The speeds shown in the test above are so ridiculously slow that they would not even be acceptable 20 years ago in dial up!
If I do not receive a satisfactory response from you then I will be seeking legal advice, and I shall also be contacting local digital and print publications to gain more exposure on this issue.
Now a reply from the most incompetent company in the world…
Date: 9/01/2017 2:56:00 PM
Subject: Re: Letter of complaint due to breach of service contract by Sky Cable
Our apologies for the inconvenience this has caused you.
Please be advised that your request for a service visit is already facilitated and we will send to you our serivce technician within 48 to 72 hours.
Kindly keep your lines active to accommodate a call from our team to informed you regarding the schedule.
Thank you and best regards,
SKY Office Support
Customer Care Division
Nice one Sky Cable, nice one!