cancel
Showing results for 
Search instead for 
Did you mean: 

Rogers outage affects millions

Korth
Mayor / Maire

The entire Rogers network went offline for most of the day on July 8.

 

So roughly one-third of all Canadians had no mobile service for the day. I've heard a lot of people complain about that, omg no social media for a whole day I'm astonished there weren't angry mobs and riots and suicides.

 

But the outage affected millions of Canadians in the real world, too. Debit machines and ATMs unable to connect to the network, fuel pumps unable to connect to the network - if you happen to use a bank which happens to be a Rogers customer then you were broke all day.

Most fuel pumps unable to connect to the network. The media claims that "transport and logistics in Canada ground to a halt", which is exaggerated but still based on some truth.

And most government services, government offices shut down for the day. It seems that the government really likes subscribing to Rogers. So do the police, emergency, and 911 lines.

Likewise, many non-government things - hospitals, schools, supermarkets, Tim Hortons - had to cease operations for the day, or could only continue by handling filthy untraceable cash.

 

These things happen, I get it.

 

But what I don't understand is that nobody's really talking about it. And Rogers hasn't really made any statements about it. And nobody really expects them to, we'll just accept whatever they do (or do not) say.

 

Communications (and banking, transport, government, etc) disrupted for a day. Why? Evil hackers? Broken machines? Sabotage, incompetence, or corruption? Cold war and terrorism?

Some kind of explanation is needed. To assure people that this was a freak incident which won't be repeated, no?

 

To me this entire thing is just another example of our nation's over-reliance on the Big Three. A real demonstration of how backwards and counterproductive our contrived oligarchy really is. One third of a county offline because a single company failed? What if it took them a week to fix it instead of a day? What if you were one of Canada's ten thousand 911 callers that day? Why is nobody accountable? What guarantee do we have that it won't happen again?

27 REPLIES 27

bluejaywpg
Model Citizen / Citoyen Modèle

@sheytoon OK I reread your post & got lost in the explanations thinking they were something different. I see where you did explain the 2 things (core & RAN) - they are referred to as E-UTRAN & EPC. But again, it confused me a bit at 1st.

I tried to do that, but maybe I didn't do a good job of it.

 

RAN = tower

Core = back-end equipment in data centres. Most of the important decision making is done at the core.

 

The goal is for the RAN is to connect your phone to the core network.

bluejaywpg
Model Citizen / Citoyen Modèle

@sheytoon thanks, that is an interesting read, though some of it is a bit tricky to understand.

I sometimes forget that Telus was previously AGT & BCTel, which I assume figures into the history & reasoning into why the 2 companies share.

I read this & although it is somewhat technical (& so am I in supporting end-user devices etc.), I had to look up stuff to understand this better. A good update would be to briefly explain some of the terminology (core, RAN, etc.).

@bluejaywpg Bell and Telus share pretty equally. They each own towers in approximately half of their coverage area.

 

More info on how it works if you're interested:

https://productioncommunity.publicmobile.ca/t5/Get-Support/Network-sharing-explained/td-p/129092

bluejaywpg
Model Citizen / Citoyen Modèle

Yes if you're talking about the big one in 2003 that affected several states + Ontario. It showed how although private utilities have failed in the past to provide stable power at all times, even some public utilities didn't exactly take precautions against such catastrophic events at the time, which is why Ontario was affected but not Quebec which had a safety mechanism installed. 

 

In Canada's case, unlike the numerous utilities that supply electricity across the country, we are limited to 2 truly national sets of cell towers (Bell & Rogers), as I believe Telus doesn't quite have a fully national grid like the other 2 (correct me if I'm wrong).

 

It could've happened to us, but we're lucky it didn't, & it happened to the other guys. Still, that indeed shows how vulnerable we really are.

Community_QA
Model Citizen / Citoyen Modèle

Canadian put all eggs in 1 basket.. actually 2 (Rogers vs Bell/Telus), it's just a matter of time these things happen

 

Rogers outage remind me about the Big power outage 20+ years ago.  i wonder what will happen if we have another one.  

 

CFPartDeux
Town Hero / Héro de la Ville

I read a blurb that said someone was looking into starting a class action lawsuit, with the intention of getting a $400 credit for every Rogers, Fido, & Chatr customer..... will wait to see if that ever happens. 🤔


@bluejaywpg wrote:

 

Anything is possible. Hard to know from the outside.

Easier to know things which have been observed.

 

Like how so many critical and non-critical government services apparently rely exclusively on Rogers - and apparently don't have some sort of contingency or fallback they can activate in a timely manner. When nation-wide financial, logistics, government, medical, and police services are crippled on city-wide scales because their internet provider goes offline ... that shows an alarming picture of how (un)prepared Canada will be if/when a real disaster knocks out communications.

 

What will Canada do if, for example, a tsunami or a cyberattack threatens to do real damage (which costs real money, real suffering, real lives) but the first sign of danger is a Big Three communication network itself being disrupted? Indeed, I think that if any deviants, criminals, terrorists, or enemies wanted to inflict harm on Canada then they've just been given a great example of a vulnerability they can exploit.

bluejaywpg
Model Citizen / Citoyen Modèle

@softech wrote:

@bluejaywpg wrote:

A maintenance update caused the outage, according to Rogers.

theglobeandmail.com/canada/article-rogers-ceo-says-maintenance-update-caused-fridays-daylong-national


I wonder how many fellows in the maintenance team  will loss their jobs


It's such a potentially complex thing. We don't know the nature of the update or what equipment the update affected, & why. Was a contractor involved? Was it fully in-house? Was it their own software, or another company's software? Why couldn't they restore to the old configuration quickly? Did any financial / hiring decisions affect Rogers' ability to respond quickly? Did Rogers recently lay off or lose their best IT staff that knew about the affected software / equipment?

 

Anything is possible. Hard to know from the outside.

bluejaywpg
Model Citizen / Citoyen Modèle

I've seen plenty of chatter about that aspect of redundancy amongst my tech friends. I'm sure that many smaller organizations have little in the way of DR (disaster recovery) plans. But when it comes to large organizations, there are so many points of potential failure that a company may only want to cover the cheapest / easiest methods, along with the most important aspects that would cause the company massive financial loss if failure were to occur. Some of those include:

- Loss of internet connectivity due to provider - since this can happen at anytime & often for short periods, a backup connection can come in very handy, & may not need the full power of the main connection in order for core business to continue normally, like at retail stores. The provider should be able to quickly replace a failed part if for example a modem fails & it belongs to, or was supplied by the provider.

 

- Network equipment failure - onsite IT staff should be able to purchase plenty of backup equipment in the event that a network switch or other important piece of network infrastructure should fail.

 

- Computer server failure - redundant servers are not necessarily super-expensive, & servers are less prone to failure now than in the past. Even the servers themselves contain redundant parts like dual power supplies & redundant hard drives.

 

- Virus / malware / network security - it takes a lot of measures to ensure these are not compromised by requiring strong passwords, using good security apps, & other measures.

 

- Bad software updates - it's necessary to have proper backups & to be able to restore old software configurations if a bad update renders important servers & equipment inoperational.

 

- Physical disaster like fire, flood, etc. - many large organizations have multiple sites already, so why not implement redundant networking / server equipment at another site?

 

I find it unconscionable that such an important organization like Interac would rely on a single provider for its internet services. It makes me wonder what else they didn't do because they were unwilling to dedicate money & resources?

Kristowhy
Model Citizen / Citoyen Modèle

Good discussion.

 

Nobody is talking about Interac and their lack of diversification which lead to debit, ATM and e-transfer transactions being unavailable.  I wonder how many other organizations have no/little redundancy let alone DR plans.  

 

interac.jpg

 

 

 


@softech wrote:

@bluejaywpg wrote:

A maintenance update caused the outage, according to Rogers.

theglobeandmail.com/canada/article-rogers-ceo-says-maintenance-update-caused-fridays-daylong-national


I wonder how many fellows in the maintenance team  will loss their jobs


It is a family business.  Group hugs all around.  

Will they be proactively crediting me as well? I don't subscribe to Rogers. But it turns out a job site, all the debit and credit terminals, and both ATMs I tried to use that day didn't work. That's three hours of my time, driving around, some personal reasons (I was hungry) and some work reasons (no networks = no machining).

 

Will Rogers be issuing me some sort of payment for my time?

softech
Oracle
Oracle

"We will proactively credit your account for Friday’s outage. This credit will be automatically applied and no action is required from you."

 

Sound like the credit will be just one day.  So, on average, around  $2 for Mobile or $3 for Home internet for a crazy day like that?

 

I guess ok.  Many people, esp those WFH people, earned a free long weekend (they just call back office and tell them internet down , they cannot work and then they headed over to the beach 🤣


@bluejaywpg wrote:

A maintenance update caused the outage, according to Rogers.

theglobeandmail.com/canada/article-rogers-ceo-says-maintenance-update-caused-fridays-daylong-national


I wonder how many fellows in the maintenance team  will loss their jobs

darlicious
Mayor / Maire

@Korth 

Forbearance.....the bain of our existence. Make your voice known and maybe, just maybe something might change.....

 

https://ised-isde.canada.ca/site/mobile-plans/en/proposed-policy-direction-crtc-competition-affordab...

bluejaywpg
Model Citizen / Citoyen Modèle

A maintenance update caused the outage, according to Rogers.

theglobeandmail.com/canada/article-rogers-ceo-says-maintenance-update-caused-fridays-daylong-national

bluejaywpg
Model Citizen / Citoyen Modèle

Why would government involvement be automatically bad? It was government initiatives to encourage more competition & also the threat of government intervention that created Public Mobile & then established it as a low-cost tier of service along with Lucky & chatr.

 

If there was no government intervention, we would probably still be paying no less than $55 per month for at best average cell service compared to other countries. I believe there needs to be way more intervention to control prices for consumers & create more competition, meanwhile ensuring service doesn't go down.

 

"I confess that I was thinking some sort of regulation or oversight - along with penalties for noncompliance - would be a solution. Since this situation happened once and we are not being given any believable assurances that it won't happen again, the industry apparently isn't regulating itself very well."

 

@Korth I believe this would work to some degree. Companies have a very poor history of self-regulating. Their mandate is profits & nothing else. If bad service increases profits then companies will very often do it.


@will13am wrote:

 

 

I hope you are not suggesting the government take on the responsibility.  That would simply increase costs.  

I confess that I was thinking some sort of regulation or oversight - along with penalties for noncompliance - would be a solution. Since this situation happened once and we are not being given any believable assurances that it won't happen again, the industry apparently isn't regulating itself very well.

 

I understand nines of service. It's simply impossible to keep every component of a nation-wide network online and uninterrupted all of the time. At any given moment there's going to be many broken things all over such a large network, along with dozens of disconnected subscribers. But for the entire thing to all fail suddenly and catastrophically, to affect millions of subscribers and to affect millions of non-subscribers?

 

Perhaps I was just angry that I'm very consciously not a Rogers subscriber, yet I am still entirely subjugated by when Rogers services fail.

 

Governmental interference would suck. It would add cost, create paper and politics and accomplish nothing useful. But when the Big Three (and their CRTC minions) are so firmly entrenched within the system - they are the system - then who else can Canadians turn to?


@will13am wrote:


I hope you are not suggesting the government take on the responsibility.  That would simply increase costs.  


"increase costs"? oh no, I see another petition brewing then.

will13am
Oracle
Oracle

@Korth wrote:

The entire Rogers network went offline for most of the day on July 8.

 

So roughly one-third of all Canadians had no mobile service for the day. I've heard a lot of people complain about that, omg no social media for a whole day I'm astonished there weren't angry mobs and riots and suicides.

 

But the outage affected millions of Canadians in the real world, too. Debit machines and ATMs unable to connect to the network, fuel pumps unable to connect to the network - if you happen to use a bank which happens to be a Rogers customer then you were broke all day.

Most fuel pumps unable to connect to the network. The media claims that "transport and logistics in Canada ground to a halt", which is exaggerated but still based on some truth.

And most government services, government offices shut down for the day. It seems that the government really likes subscribing to Rogers. So do the police, emergency, and 911 lines.

Likewise, many non-government things - hospitals, schools, supermarkets, Tim Hortons - had to cease operations for the day, or could only continue by handling filthy untraceable cash.

 

These things happen, I get it.

 

But what I don't understand is that nobody's really talking about it. And Rogers hasn't really made any statements about it. And nobody really expects them to, we'll just accept whatever they do (or do not) say.

 

Communications (and banking, transport, government, etc) disrupted for a day. Why? Evil hackers? Broken machines? Sabotage, incompetence, or corruption? Cold war and terrorism?

Some kind of explanation is needed. To assure people that this was a freak incident which won't be repeated, no?

 

To me this entire thing is just another example of our nation's over-reliance on the Big Three. A real demonstration of how backwards and counterproductive our contrived oligarchy really is. One third of a county offline because a single company failed? What if it took them a week to fix it instead of a day? What if you were one of Canada's ten thousand 911 callers that day? Why is nobody accountable? What guarantee do we have that it won't happen again?


I hope you are not suggesting the government take on the responsibility.  That would simply increase costs.  

bluejaywpg
Model Citizen / Citoyen Modèle

I mostly agree, but I disagree with:

"But what I don't understand is that nobody's really talking about it."

Did you listen to ever conversation by every Canadian in the last 24 hours? Obviously not. And we're barely past this event. Yes, some people are already talking about these aspects. My tech friends were already talking about certain technical aspects like redundancy.

"It seems that the government really likes subscribing to Rogers. So do the police, emergency, and 911 lines."

A few government services did go down. But NOT 911 in the sense that other customers could still call 911 from a phone that was not with Rogers, Fido, or chatr. I think we would've heard about a lot more services going down than just passport services & the odd other thing. I suspect many subscribe to Bell.

Your last paragraph is spot on. For the rates we have to pay for 5G (or even 4G) service, we should have gold-plated service that is the best in the world. No dead zones - not even in Nunavut! No downtime ever. Perfect service.

CountyDownIeUk
Mayor / Maire

If it was a DDOS attack...they won't tell!!!

Did they pay a ransom....they won't tell!!

I recently went through a DDOS attack at voip.ms. They blamed it all on the attack. But kept updating us with all of the improvements they were making. So I had to ask myself....if they did all these improvements before hand....would an attack been successful?  I ported my DID to PM. But now I have all my eggs in one basket and in hindsight is not a good idea. 

 

 A retailer, a financial institution, a customer should all be asking them what are you going to do to prevent this from happening again? Some customers will find alternatives. 

 

To me it is one of:

Preventative measures not up to date. 

An actual equipment failure

 

A poorly done, over the mid night hours upgrade

TA DA!! (That is....if can be trusted ....I remain skeptical.)

Rogers CEO apologizes for outage, says it came after network maintenance update

The fallout from a massive network outage at Rogers Communications that shut down mobile and internet services across much of Canada continued to come into focus on Saturday, even as the company restored most services and began offering an explanation as to what happened

 

A complete screw up by one or more

 

hTideGnow
Mayor / Maire

HI @Korth   I don't think no body talked about it.  In fact it has been a hot topic in Canada  and even in the world yesterday.

 

some "smart" expert suggested  it is time to think of a backup solution.  But I was thinking: REALLY?   do they know how much we are paying now for one "primary" service?    So, they suggest the solution is to pay  another $50 for a backup phone and backup internet service.?  

 

Maybe the real backup is Rogers and Bell should have a easy flip over to each other network when disaster like this happens.  They should be the one with a backup plan, not consumers

 

 

softech
Oracle
Oracle

for some reason, I think Rogers definitely has a higher commercial market share , at least with the retails sector, than Bell/Telus.  So, a Rogers outage like this would affect more than if it is a Bell outage.

 

And yes, it affects everything.  Simple thing like even getting food.  Many restaurant cannot take credit card and debit card, but you cannot get cash from ATM.  You cannot order via UberEat because restaurant, and because many Uber Drivers are using Rogers phones.

 

I am surprise someone here said it affected RBC , too.  RBC card already setup in PM system but RBC cannot process the charge because they are using Rogers

 

more and more interesting examples like that.

 

 

sheytoon
Mayor / Maire

Nothing definitive from Rogers yet, but according to Cloudflare, it looks like a major routing mistake (BGP) by Rogers that was preventing traffic from reaching other networks.

https://blog.cloudflare.com/cloudflares-view-of-the-rogers-communications-outage-in-canada/

MrSpock
Deputy Mayor / Adjoint au Maire

@Korth Nobody is talking about? Really?

Need Help? Let's chat.