Lessons Learned: Flashback to Summer’s Great Rogers Outage (Part 2)

In our last post, we looked at the Great Rogers Outage of 2022.

Millions of Canadians experienced life without mobile and Internet service – a necessity in our pandemic life. The cause was traced back to a system-change gone wrong. It appears that though Rogers had tested some parts of the planned change, it was insufficient to identify all the issues. The result was that the network got flooded with traffic and then the systems went down.

 

What are some lessons we can learn from this outage?

Major Controls Frameworks, like COBIT and ISO27001, and audit standards, like SOC2, require that management implement change management controls. Consequently, the outage presents a unique opportunity to understand what can go wrong when it comes to change management. Moreover, it highlights what types of controls are relevant from a real-live scenario - as Rogers documented in its submission to the CRTC. 

With that in mind, let’s look at four lessons from the Great Rogers Outage of 2022. 

Lesson #1: The Importance of Redundancy

When commenting on the impact of the outage on governments within Canada, Rogers noted: “It is important to note that in most of the cases, we provide a portion of the telecommunications solution, but not all underlying services. Many institutional customers have redundant services” [emphasis added].

Also, as previously noted that they had “established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity.”

The implication of this lesson is that we should try to diversify the telecom providers within our professional and personal lives. For example, my personal device is provisioned through Fido (a Rogers sub-brand), while my work cell is provisioned through Bell.  

Lesson #2: Test, Test, Test

They say in real-estate it’s about location, location, location. In change management it’s test, test, test. In the aftermath of the outage, Rogers doesn’t deny that they need to review their change implementation process:

“Most importantly, Rogers is examining its “change, planning and implementation” process to identify improvements to eliminate risk of further service interruptions.”

To be fair, it’s not like there was no testing done. Instead, Rogers had used a phased approach to rolling out the change:

“Concerning the July 8^th outage, the proposed activities were very carefully reviewed, as we normally do with all network changes. We validated all aspects of this change.  In fact, we had begun introducing this change weeks ago, on February 8^th and had already implemented successfully the first five (5) phases in our core network.” [emphasis added]

It’s a good reminder that in the world of IT General Controls, and IT Risk Management more broadly, it’s not about what goes right but what goes wrong. Consequently, companies should ensure that the scenarios tested are comprehensive enough to identify hidden assumptions or dependencies. For example, Rogers had a procedure that relied on “alternate carrier SIM Cards”. Hypothetically, testing whether this worked ahead of time could help identify whether the employee could find their SIM cards or how they activated such SIM cards when they have no Internet.

Lesson #3: Planning Crisis Communications from Content to Channels

According to the Rogers submission, the company conducted the following communications:

“During the outage, Rogers communicated with customers across several different channels, including social media, media outlets, Rogers Sports & Media properties, website banners, virtual assistants, interactive voice responses (“IVR”), public service announcements and community forums. In addition, Rogers’ CEO conducted broadcast interviews with CP24, Global News, CTV News, BNN, and CityNews. Rogers SVP of Access Networks & Operations also conducted broadcast interviews on CBC and CityNews.”

The following CBC news clip illustrates what was communicated and how:

As can be seen, the reporter was a little surprised that they got message from the IT team – instead of Rogers themselves. However, Rogers did admit that they “will be updating [their] plans and procedures”. Specifically, they plan to:

• Equip the communications team with “back-up devices on [an] alternate network”
• Be more timely “in posting details to customer care channels, web properties, social media, as well as public service announcements (“PSAs”) across media properties”
• Provide more frequent updates “even if there is limited or no additional information to share”
• Determine an alternative way for the communications team to authenticate themselves, when the second-factor registered with the social media service is reliant on “a device on the Rogers network”
• Provide specific “status of critical services (such as 9-1-1), how they may be impacted by the outage, and advice for customers”

The outage is a good illustration of how critical crisis communications can be. Maintaining effective communications with customers or other stakeholders is key to minimizing the reputational damage that such incidents can potentially have.

Lesson #4: Monitoring

The final takeaway is the importance of having resources and tools to monitor the restoration efforts. That is, the fixes deployed may not resolve all the issues. Rogers reported the following results with respect to bringing things back online:

“Once the technology team confirmed stability of our core network, and that traffic volumes were returning to normal level across the network, we proceeded to inform customers that our network and systems were returning to fully operational service for the vast majority of our customers. We also notified them that some customers may experience intermittent issues, and that our technology teams are monitoring and would work to resolve any issue as quickly as possible.” [emphasis added]

As can be seen, Rogers was able to restore the service for the vast majority of customers. However, there were a few that still experienced lingering issues. Consequently, it’s important to have continuous monitoring in place to ensure that the service is restored fully before returning to business as usual.

 

Closing thoughts

The incident highlights how dependent society has become on the wireless carriers for the day-to-day transactions and functioning of society. Vass Bednar (also interviewed in the above CBC newsclip) summarized the situation in an op-ed in the Globe and Mail as follows: 

“Enormous advances in mobile tech have made Canada's telecoms enormously powerful, and that power has consolidated in just five major players. That number threatens to get smaller, too, with the proposed Rogers-Shaw merger currently under review by Canada's Competition Bureau. If the deal goes through, the company that caused so many Canadians to lose connection with each other would serve roughly 40 per cent of all households in English Canada… it reinforced the idea that our telecommunication networks are vital public infrastructure that is controlled by private corporations. We've lost sight of that balance, despite the ways we rely on those networks.”

As discussed in the first takeaway, the issue of redundancy is paramount when it comes to ensuring ongoing access. Ironically, the lack of sufficient alternatives in the mobile carrier space amplifies the availability risk for us all.

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

The other DDoS: Denial of Service by DMCA

In information security, the common definition of DDoS is Distributed Denial of Service attack. However, there is a legally sanctioned form of DDoS: DMCA Denial of Service, where a user acting in good faith is 'denied service' because of an alleged infringement of the DMCA. The DMCA (i.e. the Digital Millennium Copyright Act) provides a means to enforce of copyright protections online and was ultimately responsible for killing Napster (who enabled peer-to-peer sharing of  music and other files). Although the Napster case was cut & dry to some (like the Recording Industry), there are some where users are actually acting in good faith, but are taken down through enforcement of such  an Act.

The case that illustrates this issue is the take down of 1.45 million education blogs in October. James Framer, CEO of EduBlogs, noted that "ServerBeach, to whom we pay $6,954.37 every month to host Edublogs, turned off our webservers, without notice, less than 12 hours after issuing us with a DMCA email." He went on to explain what the actual infringement was: "one of our teachers, in 2007, had shared a copy of Beck’s Hopelessness Scale with his class, a 20 question list, totalling some 279 words, published in 1974, that Pearson would like you to pay $120 for." Reading the blog further it turns out that EduBlogs did actually comply with the DMCA request that they received. However, the issue that Pearson had was (a) it was accessible via Google's cache and (b) it was accessible by its Varnish cache. In other words, James Farmer got legally DDoSed: 1.45 million blogs were made unavailable due to ServerBeach rush to comply with the DMCA instead of "calling any of the 3 numbers for us [ServerBeach] have on file".

Edublogs, however, is not the only company to be DDoSed in this manner. Small companies that publish news reports on YouTube or other content sharing sites also face this danger. Take for example Leo Laporte's This Week in Tech (TWIT) new media network, which publishes tech related podcasts and videocasts. The business model of this network resides on him being able to make the video available soon after its airing. Failure to do so will result in the company losing out on ad revenue because the "eyeballs never made it" to the particular show. Consequently, when one of their episodes gets pulled down by Google's robots, or due to request of the copyright holder (as noted here), it jeopardizes the TWIT business model making him another DDoS victim.

From a risk perspective, the risk of such event should be evaluated, especially for businesses that rely on revenues via the distribution of online content. Specifically, the agreement with the third parties that host their content should include provisions that enable them to at least demonstrate compliance prior to be taken down. However, both James Farmer and Leo Laporte have attempted to work with their respective providers to prevent this type of risk. Farmer complied with the request, while Laporte has attempted to contact Google and explain that he is news organization. So this is easier said then done. Laporte hosts the videos on his own servers, however the popularity of YouTube limits the effectiveness of this "backup strategy" (i.e. users won't go to the site to watch the video instead of YouTube). In the end, it may just be an unavoidable cost of relying on such providers.

From a longer-term perspective, it illustrates clash of legacy laws and the capability of the Internet to "network knowledge". This the concept is taken from David Weinbergers's "Too big To Know", who identified how the ability to share, link and debate information on the Internet transforms knowledge into a more fluid state in contrast to the static nature of books. He explains this concept in the following video:

James Farmer implicitly argued this point in his rant against Pearson when he said: "Here’s another idea Pearson, maybe one that you could take from Edublogs, howabout you let this tiny useful list be freely available, and then you sell your study materials / textbooks and other material around that… maybe use  Creative Commons Non Commercial Attribution license or similar to make sure you get some links and business." In other words, Pearson has failed to understand this new world of networked knowledge, where a link to the "offending" list would link to other resources that has Pearson has - enriching both Pearson and those using its publications.

Electrical and cloud outages: Is it time to bring both on premise?

Amazon experienced an outage that affected a number of companies that rely on their cloud service. The company informed its users that its service went down due to the power outage stating: 


"On June 29, 2012 at about 8:33 PM PDT, one of the Availability Zones (AZ) in our US-EAST-1 Region experienced a power issue.  While we were able to restore access to a vast majority of  RDS DB Instances that were impacted by this event, some Single-AZ DB Instances in the affected AZ experienced storage inconsistency issues and access could not be restored despite our recovery efforts.  These affected DB Instances have been moved into the “failed ” state." 


This notice was actually taken from CodeGuard (a start-up that takes snapshots of websites enabling owners to undo unwanted changes) who was one of the companies affected by the outage. 


As can be expected, many will use this as an opportunity to illustrates the danger of moving from on premise to the cloud. A parallel argument would be to highlight the dangers of drawing on electricity from the central grid. One would argue one is more reliant on power than on computing - so why not bring electricity "back" on premise? This is an absurd argument, but that is exactly the point. Companies, as pointed out by Nicholas Carr in the Big Switch, used to produce their own electricity, but eventually moved to rely on the grid for power.  Today hardly anyone produces their own power, but has backup generators in place to provide power should grid go down. And that's the right question to ask: why was there inadequate backup power at Amazon? In other words, society has decided to live with the fact that electricity is delivered centrally - but has built in controls to manage issues that may arise. 


Instead of viewing this as a black mark against cloud computing, it is important to view this discussion in the context of risk. Charles Babcock, InformationWeek published a good article on the reaction to the Outage. He noted that some are leaving AWS in reaction to the service. Specifically,  Whatsyourprice.com (an online dating service) is moving to a hosted solution - away from the cloud. However, he also mentions, Okta (an identity management service) that was unaffected by the outage because they designed their application to be fault tolerant.  

In other words companies need to focus on whether the benefits of cloud computing outweigh its risks. Cloud provide pay-as-you-go computing - giving companies who have uneven workloads the ability to buy compute resources when they need it. It also give start ups, like CodeGuard, a chance to get their offerings into the market.  Here,here and here are the follow-up posts to their outage - they were able to get back online and they are sticking with Amazon. And this should not be a surprise to anyone. Technology startups can leverage the pay-as-you-go model of cloud computing to conserve their capital and instead focus on getting their offering out. For example, the founder of Animoto, points out they went from 50 to 80 compute to 3,500 instances over three-days (they were signing up 25,000 new users per hour at the peak) when their app went viral. So companies will hopefully use the cloud outage to highlight the need for good design and appropriate controls instead of an excuse to stick to the status quo of on-premise computing.