Lessons Learned: Flashback to Summer’s Great Rogers Outage (Part 2)

In our last post, we looked at the Great Rogers Outage of 2022.

Millions of Canadians experienced life without mobile and Internet service – a necessity in our pandemic life. The cause was traced back to a system-change gone wrong. It appears that though Rogers had tested some parts of the planned change, it was insufficient to identify all the issues. The result was that the network got flooded with traffic and then the systems went down.

 

What are some lessons we can learn from this outage?

Major Controls Frameworks, like COBIT and ISO27001, and audit standards, like SOC2, require that management implement change management controls. Consequently, the outage presents a unique opportunity to understand what can go wrong when it comes to change management. Moreover, it highlights what types of controls are relevant from a real-live scenario - as Rogers documented in its submission to the CRTC. 

With that in mind, let’s look at four lessons from the Great Rogers Outage of 2022. 

Lesson #1: The Importance of Redundancy

When commenting on the impact of the outage on governments within Canada, Rogers noted: “It is important to note that in most of the cases, we provide a portion of the telecommunications solution, but not all underlying services. Many institutional customers have redundant services” [emphasis added].

Also, as previously noted that they had “established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity.”

The implication of this lesson is that we should try to diversify the telecom providers within our professional and personal lives. For example, my personal device is provisioned through Fido (a Rogers sub-brand), while my work cell is provisioned through Bell.  

Lesson #2: Test, Test, Test

They say in real-estate it’s about location, location, location. In change management it’s test, test, test. In the aftermath of the outage, Rogers doesn’t deny that they need to review their change implementation process:

“Most importantly, Rogers is examining its “change, planning and implementation” process to identify improvements to eliminate risk of further service interruptions.”

To be fair, it’s not like there was no testing done. Instead, Rogers had used a phased approach to rolling out the change:

“Concerning the July 8^th outage, the proposed activities were very carefully reviewed, as we normally do with all network changes. We validated all aspects of this change.  In fact, we had begun introducing this change weeks ago, on February 8^th and had already implemented successfully the first five (5) phases in our core network.” [emphasis added]

It’s a good reminder that in the world of IT General Controls, and IT Risk Management more broadly, it’s not about what goes right but what goes wrong. Consequently, companies should ensure that the scenarios tested are comprehensive enough to identify hidden assumptions or dependencies. For example, Rogers had a procedure that relied on “alternate carrier SIM Cards”. Hypothetically, testing whether this worked ahead of time could help identify whether the employee could find their SIM cards or how they activated such SIM cards when they have no Internet.

Lesson #3: Planning Crisis Communications from Content to Channels

According to the Rogers submission, the company conducted the following communications:

“During the outage, Rogers communicated with customers across several different channels, including social media, media outlets, Rogers Sports & Media properties, website banners, virtual assistants, interactive voice responses (“IVR”), public service announcements and community forums. In addition, Rogers’ CEO conducted broadcast interviews with CP24, Global News, CTV News, BNN, and CityNews. Rogers SVP of Access Networks & Operations also conducted broadcast interviews on CBC and CityNews.”

The following CBC news clip illustrates what was communicated and how:

As can be seen, the reporter was a little surprised that they got message from the IT team – instead of Rogers themselves. However, Rogers did admit that they “will be updating [their] plans and procedures”. Specifically, they plan to:

• Equip the communications team with “back-up devices on [an] alternate network”
• Be more timely “in posting details to customer care channels, web properties, social media, as well as public service announcements (“PSAs”) across media properties”
• Provide more frequent updates “even if there is limited or no additional information to share”
• Determine an alternative way for the communications team to authenticate themselves, when the second-factor registered with the social media service is reliant on “a device on the Rogers network”
• Provide specific “status of critical services (such as 9-1-1), how they may be impacted by the outage, and advice for customers”

The outage is a good illustration of how critical crisis communications can be. Maintaining effective communications with customers or other stakeholders is key to minimizing the reputational damage that such incidents can potentially have.

Lesson #4: Monitoring

The final takeaway is the importance of having resources and tools to monitor the restoration efforts. That is, the fixes deployed may not resolve all the issues. Rogers reported the following results with respect to bringing things back online:

“Once the technology team confirmed stability of our core network, and that traffic volumes were returning to normal level across the network, we proceeded to inform customers that our network and systems were returning to fully operational service for the vast majority of our customers. We also notified them that some customers may experience intermittent issues, and that our technology teams are monitoring and would work to resolve any issue as quickly as possible.” [emphasis added]

As can be seen, Rogers was able to restore the service for the vast majority of customers. However, there were a few that still experienced lingering issues. Consequently, it’s important to have continuous monitoring in place to ensure that the service is restored fully before returning to business as usual.

 

Closing thoughts

The incident highlights how dependent society has become on the wireless carriers for the day-to-day transactions and functioning of society. Vass Bednar (also interviewed in the above CBC newsclip) summarized the situation in an op-ed in the Globe and Mail as follows: 

“Enormous advances in mobile tech have made Canada's telecoms enormously powerful, and that power has consolidated in just five major players. That number threatens to get smaller, too, with the proposed Rogers-Shaw merger currently under review by Canada's Competition Bureau. If the deal goes through, the company that caused so many Canadians to lose connection with each other would serve roughly 40 per cent of all households in English Canada… it reinforced the idea that our telecommunication networks are vital public infrastructure that is controlled by private corporations. We've lost sight of that balance, despite the ways we rely on those networks.”

As discussed in the first takeaway, the issue of redundancy is paramount when it comes to ensuring ongoing access. Ironically, the lack of sufficient alternatives in the mobile carrier space amplifies the availability risk for us all.

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

Larry and Sergei's Exit from Google: How did they get from 'Don't Be Evil' to 'Get Rich or Die Trying'?

Larry Page and Sergey Brin have left the building.

The two founders who built the information empire Alphabet Inc. have left Google. As they noted in their farewell post, Sundar Pichai will now become CEO of both Google and Alphabet Inc. They also pocketed a couple billion or so for their troubles.

I write this post with mixed feelings.

As someone who started at university the year the Internet became commercialized, I witnessed the rise of Google from a number of many search-engine to the only one that you use. And I've written previously about this experience.

But reality is reality: Google doesn't look like the company it used to be.

They began with their motto: "Don't be evil". As noted here the idea, per Paul Buchheit (Googler #23). was not to be evil like "those other companies":

"It just sort of occurred to me that “Don’t be evil” is kind of funny. It’s also a bit of a jab at a lot of the other companies, especially our competitors, who at the time, in our opinion, were kind of exploiting the users to some extent."

And now it's arguable that Google has become one of "those other companies".

In fact, they officially abandoned the "Don't be evil" motto to the less aspirational "do the right thing."

Sure, we could hypothesize that the legal, risk and other compliance experts advised Google to abandon this slogan due to risk aversion. But the problem with that theory is that Google has been raking up the fines, not in the millions but in the billions.  According to The Verge, "Google’s total EU antitrust bill now stands at €8.2 billion ($9.3 billion)". Not sure how that fits in with "doing the right thing". Perhaps it has more to do with "get rich or die trying".  It's no wonder politicians think they can get votes by promising to break up Google and the other tech giants. 

But the fines are just the tip of the iceberg. Google was one of Obama's top campaign contributors in the 2012 election. As noted in this article by the Intercept, the coziness between Google and the Whitehouse went beyond just the election. They visited the Whitehouse 128 times over Obama's tenure. More troubling:

"Most notably, Google has faced questions for years about exercising its market power to squash rivals, infringing on its users’ privacy rights, favoring its own business affiliates in search results, and using patent law to create barriers to competition. Even Republican senators like Orrin Hatch have called out Google for its practices.

In 2012, staff at the Federal Trade Commission recommended filing antitrust charges after determining that Google was engaging in anti-competitive tactics and abusing its monopoly. A staff report that was later leaked said Google’s conduct “has resulted — and will result — in real harm to consumers and to innovation in the online search and advertising markets.”

The Wall Street Journal noted that Google’s White House visits increased right around that time. And in 2013, the presidentially appointed commissioners of the FTC overrode their staff, voting unanimously not to file any charges.

Jeff Chester, executive director of the Center for Digital Democracy, said the administration “has been a huge help” to Google both by protecting it from attempts to limit its market power and by blocking privacy legislation. “Google has been able to thwart regulatory scrutiny in terms of anti-competitive practices, and has played a key role in ensuring that the United States doesn’t protect at all the privacy of its citizens and its consumers,” Chester said."

So now they are using their capital to subvert laws and investigations to maintain their dominance.

What happened? Why did Google take a taxi ride to the dark side? 

Tim Wu, a Columbia law professor, has a theory.

In The Master Switch, he calls this type of thing the Kronos Effect. The idea is that yesterday's scrappy start-up - who defeated the evil ogre's of their day - only to becomes today's evil ogre. For example, he explains how Adolph Zukor and the other avant-garde filmmakers of his day fought the tyrannical Motion Picture Patents Company, which required you to pay royalties for just using a camera. Who did they end up becoming? The major studios of today - who sue people for copyright infringement of their content. Similarly, we can see Google, who was able to defeat Yahoo, Microsoft and others, has become arguably the Microsoft of our times.

But I think Douglas Rushkoff, ironically in Throwing Rocks at the Google Bus: How Growth Became the Enemy of Prosperity, has a better theory. 

When start-ups that emerge from the "operating system of Capitalism" the end-up being defined by that system's code or DNA: growth, profits, market share and shareholder value - these are the only things matter. Capitalism doesn't pay attention to humanitarian values, moral values or spiritual values because, well, they don't add to the GDP. And at the end of the day, that's all that matters in a Capitalist society. 

I would never push anyone to adopt the motto "get rich or die trying". 

But as we can see, Google or otherwise, companies end up with this as their mantra when they want to get to the top. 

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

Audit, Audit, Audit harked Mark: Can CPAs come to Facebook's rescue?

In an investigation by the Guardian and the New York Times, the alleged misdeeds of Cambridge Analytica were revealed.

As noted in the Guardian article:

"Christopher Wylie, who worked with a Cambridge University academic to obtain the data, told the Observer: “We exploited Facebook to harvest millions of people’s profiles. And built models to exploit what we knew about them and target their inner demons. That was the basis the entire company was built on.”... Documents seen by the Observer, and confirmed by a Facebook statement, show that by late 2015 the company had found out that information had been harvested on an unprecedented scale. However, at the time it failed to alert users and took only limited steps to recover and secure the private information of more than 50 million individuals."

The following video from TheVerge sums up the issue:

Although such allegations have received attention (in my opinion due to the association with Trump's campaign), the reality is that these allegations against Facebook are actually not new and reported in both the Intercept in early 2017 and the Guardian way back in 2015. 

There was an ensuing backlash (as noted in the video above and here) that forced Facebook CEO, Mark Zuckerberg to respond. He both had a written response and gave the following interview on CNN:

During the CNN interview, he mentioned the word "audit" 3 times[emphasis added]:

• "So we're going to go now and investigate every app that has access to a large amount of information from before we locked down our platform. And if we detect any suspicious activity, we're going to do a full forensic audit"
• "And we're now not just going to take people's word for it when they give us a legal certification, but if we see anything suspicious, which I think there probably were signs in this case that we could have looked into, we're going to do a full forensic audit."
• "We know how much -- how many people were using those services, and we can look at the patterns of their data requests. And based on that, we think we'll have a pretty clear sense of whether anyone was doing anything abnormal, and we'll be able to do a full audit of anyone who is questionable."

Can CPAs come to Mark's rescue? 

Zuckerberg's repetitive use of the word audit should be read in conjunction with his "welcoming" of regulation:

"I actually am not sure we shouldn't be regulated. You know, I think in general, technology is an increasingly important trend in the world, and I actually think the question is more what is the right regulation rather than yes or no, should it be regulated?"

Zuckerberg would not be the first tech giant to opt for regulation as a business strategy.

In Tim Wu's Master Switch, Theodore Veil also advocated for the concept of a regulated monopoly in the arena of telephones:

"[Theodore] Vail died in 1920 at age 74, shortly after resigning as AT&T's president, but by that time, his life's work was done. The Bell system had uncontested domination of American telephony, and long-distance communication was unified according to his vision. The idea of an open, competitive system had lost out to AT&T's conception of an enlightened, licensed, and regulated monopoly. AT&T would remain in this form until the 1980s, and it would return in not so substantially different form in the 2000s. As historian Milton Mueller writes, Vail had completed the "political and ideological victory of the regulated monopoly paradigm, advanced under the banner of universal service."" [emphasis added]

As Tim points out in his book, the move enabled AT&T didn't always use their monopolistic powers for good. They charged high long distance rates and even stifled innovation suppressing the answering machine due to potential conflict with its main business.

Regardless, it shows that Facebook could be an early advocate for CPAs offering privacy related assurance services around its algorithms.

AlgoTrust: A new service offering for CPAs? 

The concept of AlgoTrust is something I have previously discussed in this post.

The idea actually has support from multiple angles not least of which of comes from information security expert, Bruce Schneier:

"...it is also worth noting that there are other experts who hold that algorithms - from a privacy perspective - need to be regulated. Bruce Schneier, a well-known information security expert who helped review the Snowden documents, in his latest book, Data and Goliath ... also calls for "auditing algorithms for fairness". He also notes that such audits don't need to make the algorithms public, which is it the same way financial statements of public companies are audited today. This keeps a balance between confidentiality and public confidence in the company's use of our data."

Big Data versus Privacy: The monetization paradox
Such an algo-audit could leverage the work done by AICPA and CPA Canada in the realm of privacy, specifically the Generally Accepted Privacy Principles. That being said, privacy audits have been a hard sell in the past. But what distinguishes the service here is that it would be auditing the algorithm for compliance with privacy "regulations".The reason regulations need to be put in quotes is that in substance privacy legislation is effectively eliminated if the consumer consents to use the service.  

The challenge, therefore, is balancing the drive to monetize big data with the privacy needs of the people who use the service. For example, people who identify with the "left" may not want Steve Bannon or Trump accessing their data. Similarly, people who identify with the "right" may not want Obama accessing their social media data. The end result is that no one can access meaningful data due to privacy restrictions - resulting in a standard so restrictive that it eliminates that ability of companies like Facebook to monetize the treasure trove of data that they have collected.

As noted in an earlier post, there is an inherent highlight the conflict between privacy and profiting from big data. The value of big data emerges from the secondary uses of big data. However, privacy policies require the user to consent to a specific use of data at the time they sign up for the service. This means future big data analytics are essentially limited by what uses the user agreed upon sign-up. However, corporations in their drive to maximize profits will ultimately make privacy policies so loose (i.e. to cover secondary uses) that the user essentially has to give up all their privacy in order to use the service.

There is a lot of potential in attempting to create an assurance service to address Facebook's predicament, but as they say, the devil is in the details. 

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else