Fiona’s Fury: Flashback to Summer’s Great Rogers Outage (Part 1)

Canadians continue to pick up the pieces after tropical storm Fiona battered the maritime provinces. Although estimates of the damage are yet to be calculated, the “Nova Scotia Premier Tim Houston announced over C$40 million ($29.1 million) in support to help those who were impacted by Fiona” (link). In terms of cellphone outages, CBC reported that “there are still areas of the province without cellphone service Monday although companies declined to say exactly  how many customers have been affected.”

 

The Canadian Radio-television and Telecommunications Commission (CRTC) has asked for estimates on how many people were affected by the outage, but the telecom companies are reticent to share this information. As CBC reported: “Bell and Telus asked for some of their submissions to be redacted, while Eastlink and Rogers demanded their entire reports be kept confidential.”

Photo by Pixabay: link

 

Rogers Outage in Review: What happened last summer?

When looking at the outage that hit the Maritimes, it reminds us of the situation that unfolded over the summer. In July 2022, the Rogers outage was not limited to the East Coast. Instead, it affected the entire country. When Rogers was requested to explain what happened, it appears that they had a more conciliatory tone:

“Rogers Communications Canada Inc. (“Rogers”) is in receipt of a letter containing Requests for Information (“RFIs”) from the Canadian Radio-television and Telecommunications Commission (“CRTC” or the “Commission”), dated July 12, 2022, concerning the above-mentioned subject. Attached, please find our Response to that letter… At the outset, Rogers appreciates the opportunity to explain to the Commission, the Government of Canada and all Canadians what transpired on July 8^th, 2022.  The network outage experienced by Rogers was simply not acceptable. We failed in our commitment to be Canada’s most reliable network. We know how much our customers rely on our networks and we sincerely apologize.” [Emphasis added]

 

Though the documented was redacted, it did provide some background as to what happened. For this post, we will take a look at the outage itself. For the next post, we will look at the lessons learned.  

 

Cause of the outage

Rogers explained the cause of the outage as follows:

“Given the magnitude of the outage, it appears that Rogers had to be more forthcoming with what happened and were “Maintenance and update windows always take place in the very early morning hours when network traffic is at its quietest. At 4:43AM EDT, a specific coding was introduced in our Distribution Routers which triggered the failure of the Rogers IP core network starting at 4:45AM… The configuration change deleted a routing filter and allowed for all possible routes to the Internet to pass through the routers. As a result, the routers immediately began propagating abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their capacity levels and were then unable to route traffic, causing the common core network to stop processing traffic. As a result, the Rogers network lost connectivity to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers.” [Emphasis added]

In other words, the change inadvertently resulted in an attack pattern similar to a denial-of-service attack – where the network shutdown because it became overwhelmed with traffic.

They also go on to explain that the company “uses a common core network, essentially one IP network infrastructure, that supports all wireless, wireline and enterprise services. The common core is the brain of the network that receives, processes, transmits and connects all Internet, voice, data and TV traffic for our customers… Certain network routing equipment became flooded, exceeded their memory and processing capacity and were then unable to route and process traffic, causing the common core network to shut down.” The implication being that the common core network became a single point of failure.

 

What was and was not impacted

With respect to Rogers Bank (yes, Rogers operates a bank):

“The impact to the Bank’s customers was minimal as the Bank services were available and the Bank’s customers were able to transact on their Rogers Bank credit cards. There was no interruption in the Bank’s core systems (credit card processing, Interactive Voice Response (“IVR”), Call Centre and customer self-serve mobile application) and these core systems remained available to the Bank’s customers. No critical Bank systems were impacted, and all daily processing was completed as required, including by the Bank’s statement printing vendor and its card personalization bureau which received their daily files and were processing them per standard service level agreements and procedures.”

 

This was a different story for those that relied on Rogers phone lines to process payments at their businesses with Interac tweeting:

“There is a nationwide Rogers outage that encompasses all their business and consumer network services. This is impacting INTERAC Debit and INTERAC eTransfer. INTERAC Debit is currently unavailable online and at checkout..”

 

Beyond the millions who had no service, emergency communications were also impacted:

• “Unfortunately, the outage of July 8^th did impact 9-1-1 service across Rogers’ service area, to both wireline and wireless services.
• Wireline impact:  There were approximately [REDACTED] 9-1-1 calls placed successfully across Rogers’ network on July 8th.  The typical daily average of total wireline 9-1-1 calls is [REDACTED] per day. Data is unavailable for unsuccessful wireline 9-1-1 calls.  On July 9^th, there were approximately [REDACTED] 9-1-1 calls placed successfully across Rogers’ network.
• Wireless impact: As can be seen in table below, the outage similarly affected wireless 9-1-1. Total successful calls were [REDACTED] the average daily amount of about [REDACTED] 9-1-1 calls made from Rogers wireless devices.”
•  

Rogers offered service outage credits

The key remedy offered was service credits, but this was not due to breaches in service agreements:

“There was no breach of our service agreements with our retail customers. However, in order to address our customers’ disappointment with the outage, Rogers has already announced it will be crediting 5 days of service fees to its customers. This will be applied automatically to their next invoice.”

 

Cooperation with Bell and Telus

Regardless of the highly-competitive nature of the business, it does appear the Rogers, Bell and Telus were coordinating with each other:

• “On July 17th, 2015, the Canadian Telecom Resiliency Working Group (“CTRWG”), formerly called Canadian Telecom Emergency Preparedness Association, established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity.”
• “As we stated in Rogers(CRTC)11July2022-1.xviii above, our Chief Technology and Information Officer reached out to his counterparts at Bell and TELUS early on July 8th. Assistance was offered by both Bell and TELUS. However, given the nature of the issue, Rogers rapidly assessed and concluded that it was not possible to make the necessary network changes to enable our wireless customers to move to their wireless networks.”
• “Rogers, Bell and TELUS are presently assessing potential options and will report further findings and potential solutions per the creation of the Memorandum of Understanding that will be delivered in September 2022 to the Minister of ISED by CSTAC.”

In closing, the outage comes down to change management. The error was exacerbated by the industry-standard approach to using a single platform to provide the various telecommunication services. Rogers did offer service credits, but were careful to note that this was not due to breach of agreements. Finally, the industry does come together during crisis situation, putting their competitive differences aside. 

In our next post, we’ll take a look at the lessons learned from this outage. Stay tuned!

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

COVID-19 & Accounting firms: Will only the agile survive?

The impact of COVID-19 is impacting everyone, including accountants. As discussed in the last post, the crisis has made video calling normal to the point that people are experiencing fatigue. It speaks one of the adjustments people have had to make due to the "new normal"

But what is the wider impact on the profession? How are firms handling the COVID-19 Crisis?

Accounting Today published a survey, "The accounting profession and the coronavirus: The crisis in number" that gives some data as to where the accounting firms are at.

Economic Impacts: Bad News and the Good News

Not surprisingly, nearly three-quarters of the firms surveyed, felt that the pandemic was going to reduce their revenues. In terms of magnitude, 37% of those surveyed are predicting a 10%+ loss in earnings. The good news, however, is that most had not let staff go. Only 7% had laid off staff, while 4% were planning to do so.

The other interesting find is that the most popular service to come about due to this crisis was CARES Act Consulting, with 73% offering this service. The next closest was business continuity consulting at 36%. 

There were also some interesting finds around the tech front. 

• Working Remote: Over 60% of firms had challenges with closing their offices, with nearly half of those having some challenges with the "online approach". The survey found that only 10% had no remote capabilities. See the graphic below for more details
• Closing offices: Closely related to the previous result, only 13% fully shut down their office. The survey did not reveal why this was the case.  But if you can't work remotely, what other choice do you have? 
• Communications: Although more than half used traditional means of communication, 33%were looking at new forms of communication. 

https://www.accountingtoday.com/list/the-accounting-profession-and-the-coronavirus-the-crisis-in-numbers

CPA firms provide COVID-19 services free of charge
Many small businesses have been drastically impacted by the coronavirus shutdown. As reported by the Wall Street Journal, "about 20% of them had enough cash saved to operate normally for only two months if their revenue were to dry up. Among less financially secure companies, only 10% could operate normally on savings alone for two months". The survey found that 1/3rd of CPA firms are stepping up to help by not charging for COVID-19 related services. 

Agility in time of uncertainty

Virtual firms, like Live.ca, seem to have been well prepared for this pandemic. With no offices to speak of, the firm was online from day one. The firm was featured on this CPA Canada promotional video:

Being agile in times of adversity is key to success. Understandably, tech can be daunting for small firms. However, it is also daunting for small businesses. Consequently, the tech-savvy CPA firms are able to offer consulting services like business continuity planning. But before getting there, firms need to ensure that they have the underlying capabilities to be agile. For example, if the firm has limited capability to service clients remotely it not only reduces the ability to service clients but also prevents the firm from being viewed as adaptive by current and prospective clients.

That being said, it's a matter of will. With nearly three-quarters of the firms already offering CARES Consulting, just shows how agile firms can be when the mindset is there.

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else

The other DDoS: Denial of Service by DMCA

In information security, the common definition of DDoS is Distributed Denial of Service attack. However, there is a legally sanctioned form of DDoS: DMCA Denial of Service, where a user acting in good faith is 'denied service' because of an alleged infringement of the DMCA. The DMCA (i.e. the Digital Millennium Copyright Act) provides a means to enforce of copyright protections online and was ultimately responsible for killing Napster (who enabled peer-to-peer sharing of  music and other files). Although the Napster case was cut & dry to some (like the Recording Industry), there are some where users are actually acting in good faith, but are taken down through enforcement of such  an Act.

The case that illustrates this issue is the take down of 1.45 million education blogs in October. James Framer, CEO of EduBlogs, noted that "ServerBeach, to whom we pay $6,954.37 every month to host Edublogs, turned off our webservers, without notice, less than 12 hours after issuing us with a DMCA email." He went on to explain what the actual infringement was: "one of our teachers, in 2007, had shared a copy of Beck’s Hopelessness Scale with his class, a 20 question list, totalling some 279 words, published in 1974, that Pearson would like you to pay $120 for." Reading the blog further it turns out that EduBlogs did actually comply with the DMCA request that they received. However, the issue that Pearson had was (a) it was accessible via Google's cache and (b) it was accessible by its Varnish cache. In other words, James Farmer got legally DDoSed: 1.45 million blogs were made unavailable due to ServerBeach rush to comply with the DMCA instead of "calling any of the 3 numbers for us [ServerBeach] have on file".

Edublogs, however, is not the only company to be DDoSed in this manner. Small companies that publish news reports on YouTube or other content sharing sites also face this danger. Take for example Leo Laporte's This Week in Tech (TWIT) new media network, which publishes tech related podcasts and videocasts. The business model of this network resides on him being able to make the video available soon after its airing. Failure to do so will result in the company losing out on ad revenue because the "eyeballs never made it" to the particular show. Consequently, when one of their episodes gets pulled down by Google's robots, or due to request of the copyright holder (as noted here), it jeopardizes the TWIT business model making him another DDoS victim.

From a risk perspective, the risk of such event should be evaluated, especially for businesses that rely on revenues via the distribution of online content. Specifically, the agreement with the third parties that host their content should include provisions that enable them to at least demonstrate compliance prior to be taken down. However, both James Farmer and Leo Laporte have attempted to work with their respective providers to prevent this type of risk. Farmer complied with the request, while Laporte has attempted to contact Google and explain that he is news organization. So this is easier said then done. Laporte hosts the videos on his own servers, however the popularity of YouTube limits the effectiveness of this "backup strategy" (i.e. users won't go to the site to watch the video instead of YouTube). In the end, it may just be an unavoidable cost of relying on such providers.

From a longer-term perspective, it illustrates clash of legacy laws and the capability of the Internet to "network knowledge". This the concept is taken from David Weinbergers's "Too big To Know", who identified how the ability to share, link and debate information on the Internet transforms knowledge into a more fluid state in contrast to the static nature of books. He explains this concept in the following video:

James Farmer implicitly argued this point in his rant against Pearson when he said: "Here’s another idea Pearson, maybe one that you could take from Edublogs, howabout you let this tiny useful list be freely available, and then you sell your study materials / textbooks and other material around that… maybe use  Creative Commons Non Commercial Attribution license or similar to make sure you get some links and business." In other words, Pearson has failed to understand this new world of networked knowledge, where a link to the "offending" list would link to other resources that has Pearson has - enriching both Pearson and those using its publications.

Hurricane Sandy and Disaster Recovery: Cloud to the rescue?

When looking at the aftermath of hurricane Sandy, the most important aspect of the event is the toll it has had on the people. The Atlantic puts the total impact in terms of dollars at $60 billion, with death toll at 123 people. However, those that survived face the challenges brought about by the flooding and living without power for weeks. For example, 4 million remained without power for extended period of time. This of course challenged individuals to keep their frozen food cold and live without technology for that period of time. As for companies, their disaster recovery plans were put to the test. Perhaps the most poignant example was the New York University Langone Medical Center who had to evacuate patients because their backup generators because they were located in the basements, which got flooded. Hospital officials defended their preparedness  but critics pointed out that the backup power generators "are not state-of-the-art".

Samara Lynn of PC Magazine published an article on how Sandy taught organizations valuable lessons from a Disaster Recovery (DR) perspective (she previously painstakingly put together a 4 part series for small and medium sized businesses on DR planning; see here, here, here, and here). Before I read the article, I was expecting a bulleted list of dos and don'ts when it comes DR planning. But what I was surprised to find is that companies are relying on cloud computing service providers to make up for the unavailability of local processing. Examples include:

• A New York Architectural firm Diller Scofidio + Renfro used Amazon Web Services (AWS) to relocate the company's core applications, enabling users with the proper license configuration to access these applications right from their laptops. Also, the IT Manager, Chris Donnell, used AWS as a remote desktop during the disaster. (I encourage you to read the whole article as it details how Chris was in the middle of an email migration from Outlook to Gmail when Sandy hit; poor guy!). The company also used Panzura to store the data temporarily in the cloud.
• Ring Central, a cloud-based pbx hosting service, (they sponsor TWIET and other podcasts on the TWIT network) was able to relocate their operations away from the storm. More importantly, they offer near instant recovery of phone support by plugging in a piece of hardware they can "bring in a live extension under 10 minutes". Naturally, there is an increased interest in Ring Central by those that were satisfied with the lengthy recovery times of their providers. 

The article also discusses how a service provider made DR as part of IT outsourcing service and how the key to DR is backup power. 

Although not related directly to cloud, one of the most amazing story that I've heard is how SquareSpace (SQS) kept it's platform up and running. Like the hospital, SQS had its back up generator in the basement and that got flooded. It published this blog post to inform customers of what was happening. However, the real interesting story is the lengths that team went to ensure the site stayed up and running. The team physically took fuel from the basement to the generator of the roof going up 17 flights of stair. 

Even more amazing was that the founder and CEO, Anthony Casalena, personally helped in this effort. Talk about Tone at the Top!