Lessons Learned: Flashback to Summer’s Great Rogers Outage (Part 2)

In our last post, we looked at the Great Rogers Outage of 2022.

Millions of Canadians experienced life without mobile and Internet service – a necessity in our pandemic life. The cause was traced back to a system-change gone wrong. It appears that though Rogers had tested some parts of the planned change, it was insufficient to identify all the issues. The result was that the network got flooded with traffic and then the systems went down.

 

What are some lessons we can learn from this outage?

Major Controls Frameworks, like COBIT and ISO27001, and audit standards, like SOC2, require that management implement change management controls. Consequently, the outage presents a unique opportunity to understand what can go wrong when it comes to change management. Moreover, it highlights what types of controls are relevant from a real-live scenario - as Rogers documented in its submission to the CRTC. 

With that in mind, let’s look at four lessons from the Great Rogers Outage of 2022. 

Lesson #1: The Importance of Redundancy

When commenting on the impact of the outage on governments within Canada, Rogers noted: “It is important to note that in most of the cases, we provide a portion of the telecommunications solution, but not all underlying services. Many institutional customers have redundant services” [emphasis added].

Also, as previously noted that they had “established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity.”

The implication of this lesson is that we should try to diversify the telecom providers within our professional and personal lives. For example, my personal device is provisioned through Fido (a Rogers sub-brand), while my work cell is provisioned through Bell.  

Lesson #2: Test, Test, Test

They say in real-estate it’s about location, location, location. In change management it’s test, test, test. In the aftermath of the outage, Rogers doesn’t deny that they need to review their change implementation process:

“Most importantly, Rogers is examining its “change, planning and implementation” process to identify improvements to eliminate risk of further service interruptions.”

To be fair, it’s not like there was no testing done. Instead, Rogers had used a phased approach to rolling out the change:

“Concerning the July 8^th outage, the proposed activities were very carefully reviewed, as we normally do with all network changes. We validated all aspects of this change.  In fact, we had begun introducing this change weeks ago, on February 8^th and had already implemented successfully the first five (5) phases in our core network.” [emphasis added]

It’s a good reminder that in the world of IT General Controls, and IT Risk Management more broadly, it’s not about what goes right but what goes wrong. Consequently, companies should ensure that the scenarios tested are comprehensive enough to identify hidden assumptions or dependencies. For example, Rogers had a procedure that relied on “alternate carrier SIM Cards”. Hypothetically, testing whether this worked ahead of time could help identify whether the employee could find their SIM cards or how they activated such SIM cards when they have no Internet.

Lesson #3: Planning Crisis Communications from Content to Channels

According to the Rogers submission, the company conducted the following communications:

“During the outage, Rogers communicated with customers across several different channels, including social media, media outlets, Rogers Sports & Media properties, website banners, virtual assistants, interactive voice responses (“IVR”), public service announcements and community forums. In addition, Rogers’ CEO conducted broadcast interviews with CP24, Global News, CTV News, BNN, and CityNews. Rogers SVP of Access Networks & Operations also conducted broadcast interviews on CBC and CityNews.”

The following CBC news clip illustrates what was communicated and how:

As can be seen, the reporter was a little surprised that they got message from the IT team – instead of Rogers themselves. However, Rogers did admit that they “will be updating [their] plans and procedures”. Specifically, they plan to:

• Equip the communications team with “back-up devices on [an] alternate network”
• Be more timely “in posting details to customer care channels, web properties, social media, as well as public service announcements (“PSAs”) across media properties”
• Provide more frequent updates “even if there is limited or no additional information to share”
• Determine an alternative way for the communications team to authenticate themselves, when the second-factor registered with the social media service is reliant on “a device on the Rogers network”
• Provide specific “status of critical services (such as 9-1-1), how they may be impacted by the outage, and advice for customers”

The outage is a good illustration of how critical crisis communications can be. Maintaining effective communications with customers or other stakeholders is key to minimizing the reputational damage that such incidents can potentially have.

Lesson #4: Monitoring

The final takeaway is the importance of having resources and tools to monitor the restoration efforts. That is, the fixes deployed may not resolve all the issues. Rogers reported the following results with respect to bringing things back online:

“Once the technology team confirmed stability of our core network, and that traffic volumes were returning to normal level across the network, we proceeded to inform customers that our network and systems were returning to fully operational service for the vast majority of our customers. We also notified them that some customers may experience intermittent issues, and that our technology teams are monitoring and would work to resolve any issue as quickly as possible.” [emphasis added]

As can be seen, Rogers was able to restore the service for the vast majority of customers. However, there were a few that still experienced lingering issues. Consequently, it’s important to have continuous monitoring in place to ensure that the service is restored fully before returning to business as usual.

 

Closing thoughts

The incident highlights how dependent society has become on the wireless carriers for the day-to-day transactions and functioning of society. Vass Bednar (also interviewed in the above CBC newsclip) summarized the situation in an op-ed in the Globe and Mail as follows: 

“Enormous advances in mobile tech have made Canada's telecoms enormously powerful, and that power has consolidated in just five major players. That number threatens to get smaller, too, with the proposed Rogers-Shaw merger currently under review by Canada's Competition Bureau. If the deal goes through, the company that caused so many Canadians to lose connection with each other would serve roughly 40 per cent of all households in English Canada… it reinforced the idea that our telecommunication networks are vital public infrastructure that is controlled by private corporations. We've lost sight of that balance, despite the ways we rely on those networks.”

As discussed in the first takeaway, the issue of redundancy is paramount when it comes to ensuring ongoing access. Ironically, the lack of sufficient alternatives in the mobile carrier space amplifies the availability risk for us all.

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

Fiona’s Fury: Flashback to Summer’s Great Rogers Outage (Part 1)

Canadians continue to pick up the pieces after tropical storm Fiona battered the maritime provinces. Although estimates of the damage are yet to be calculated, the “Nova Scotia Premier Tim Houston announced over C$40 million ($29.1 million) in support to help those who were impacted by Fiona” (link). In terms of cellphone outages, CBC reported that “there are still areas of the province without cellphone service Monday although companies declined to say exactly  how many customers have been affected.”

 

The Canadian Radio-television and Telecommunications Commission (CRTC) has asked for estimates on how many people were affected by the outage, but the telecom companies are reticent to share this information. As CBC reported: “Bell and Telus asked for some of their submissions to be redacted, while Eastlink and Rogers demanded their entire reports be kept confidential.”

Photo by Pixabay: link

 

Rogers Outage in Review: What happened last summer?

When looking at the outage that hit the Maritimes, it reminds us of the situation that unfolded over the summer. In July 2022, the Rogers outage was not limited to the East Coast. Instead, it affected the entire country. When Rogers was requested to explain what happened, it appears that they had a more conciliatory tone:

“Rogers Communications Canada Inc. (“Rogers”) is in receipt of a letter containing Requests for Information (“RFIs”) from the Canadian Radio-television and Telecommunications Commission (“CRTC” or the “Commission”), dated July 12, 2022, concerning the above-mentioned subject. Attached, please find our Response to that letter… At the outset, Rogers appreciates the opportunity to explain to the Commission, the Government of Canada and all Canadians what transpired on July 8^th, 2022.  The network outage experienced by Rogers was simply not acceptable. We failed in our commitment to be Canada’s most reliable network. We know how much our customers rely on our networks and we sincerely apologize.” [Emphasis added]

 

Though the documented was redacted, it did provide some background as to what happened. For this post, we will take a look at the outage itself. For the next post, we will look at the lessons learned.  

 

Cause of the outage

Rogers explained the cause of the outage as follows:

“Given the magnitude of the outage, it appears that Rogers had to be more forthcoming with what happened and were “Maintenance and update windows always take place in the very early morning hours when network traffic is at its quietest. At 4:43AM EDT, a specific coding was introduced in our Distribution Routers which triggered the failure of the Rogers IP core network starting at 4:45AM… The configuration change deleted a routing filter and allowed for all possible routes to the Internet to pass through the routers. As a result, the routers immediately began propagating abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their capacity levels and were then unable to route traffic, causing the common core network to stop processing traffic. As a result, the Rogers network lost connectivity to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers.” [Emphasis added]

In other words, the change inadvertently resulted in an attack pattern similar to a denial-of-service attack – where the network shutdown because it became overwhelmed with traffic.

They also go on to explain that the company “uses a common core network, essentially one IP network infrastructure, that supports all wireless, wireline and enterprise services. The common core is the brain of the network that receives, processes, transmits and connects all Internet, voice, data and TV traffic for our customers… Certain network routing equipment became flooded, exceeded their memory and processing capacity and were then unable to route and process traffic, causing the common core network to shut down.” The implication being that the common core network became a single point of failure.

 

What was and was not impacted

With respect to Rogers Bank (yes, Rogers operates a bank):

“The impact to the Bank’s customers was minimal as the Bank services were available and the Bank’s customers were able to transact on their Rogers Bank credit cards. There was no interruption in the Bank’s core systems (credit card processing, Interactive Voice Response (“IVR”), Call Centre and customer self-serve mobile application) and these core systems remained available to the Bank’s customers. No critical Bank systems were impacted, and all daily processing was completed as required, including by the Bank’s statement printing vendor and its card personalization bureau which received their daily files and were processing them per standard service level agreements and procedures.”

 

This was a different story for those that relied on Rogers phone lines to process payments at their businesses with Interac tweeting:

“There is a nationwide Rogers outage that encompasses all their business and consumer network services. This is impacting INTERAC Debit and INTERAC eTransfer. INTERAC Debit is currently unavailable online and at checkout..”

 

Beyond the millions who had no service, emergency communications were also impacted:

• “Unfortunately, the outage of July 8^th did impact 9-1-1 service across Rogers’ service area, to both wireline and wireless services.
• Wireline impact:  There were approximately [REDACTED] 9-1-1 calls placed successfully across Rogers’ network on July 8th.  The typical daily average of total wireline 9-1-1 calls is [REDACTED] per day. Data is unavailable for unsuccessful wireline 9-1-1 calls.  On July 9^th, there were approximately [REDACTED] 9-1-1 calls placed successfully across Rogers’ network.
• Wireless impact: As can be seen in table below, the outage similarly affected wireless 9-1-1. Total successful calls were [REDACTED] the average daily amount of about [REDACTED] 9-1-1 calls made from Rogers wireless devices.”
•  

Rogers offered service outage credits

The key remedy offered was service credits, but this was not due to breaches in service agreements:

“There was no breach of our service agreements with our retail customers. However, in order to address our customers’ disappointment with the outage, Rogers has already announced it will be crediting 5 days of service fees to its customers. This will be applied automatically to their next invoice.”

 

Cooperation with Bell and Telus

Regardless of the highly-competitive nature of the business, it does appear the Rogers, Bell and Telus were coordinating with each other:

• “On July 17th, 2015, the Canadian Telecom Resiliency Working Group (“CTRWG”), formerly called Canadian Telecom Emergency Preparedness Association, established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity.”
• “As we stated in Rogers(CRTC)11July2022-1.xviii above, our Chief Technology and Information Officer reached out to his counterparts at Bell and TELUS early on July 8th. Assistance was offered by both Bell and TELUS. However, given the nature of the issue, Rogers rapidly assessed and concluded that it was not possible to make the necessary network changes to enable our wireless customers to move to their wireless networks.”
• “Rogers, Bell and TELUS are presently assessing potential options and will report further findings and potential solutions per the creation of the Memorandum of Understanding that will be delivered in September 2022 to the Minister of ISED by CSTAC.”

In closing, the outage comes down to change management. The error was exacerbated by the industry-standard approach to using a single platform to provide the various telecommunication services. Rogers did offer service credits, but were careful to note that this was not due to breach of agreements. Finally, the industry does come together during crisis situation, putting their competitive differences aside. 

In our next post, we’ll take a look at the lessons learned from this outage. Stay tuned!

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.