Wednesday, December 21, 2022

AI and the CPA: OpenAI's ChatGPT will Blow Your Mind! Try it Now!

We previously discussed OpenAI in this post. In that post, I closed with this comment:

"We should be aware of the concept of exponential change. Technology, like AI, improves at an exponential rate and not a linear rate. Consequently, monitoring the space is key for CPAs and other to ensure that they see change coming and adapt accordingly."

And so, my fellow CPAs, you need to check out how much AI has advance in the two years since that post. It's absolutely amazing to see how advanced the technology is and how quickly it has come:


YouTube is one thing. But testing it out is another. To be honest, I was about to review the video and then summarize it. Then I thought, why not ask ChatGPT? 

Question to ChatGPT: "What are 5 key takeaways for CPAs with respect to ChatGPT?"

Answer: 
  1. ChatGPT is a large language model trained by OpenAI.
  2. ChatGPT has a knowledge cutoff of 2021, and it does not have access to the internet or current events.
  3. ChatGPT is able to generate human-like responses to a wide range of questions, but it is important to keep in mind that it is a machine and may not always provide accurate or complete information.
  4. ChatGPT is not a substitute for professional advice or guidance, and it should not be used as the sole basis for making important decisions.
  5. ChatGPT is intended for general informational purposes only, and it is not a substitute for professional judgment or expertise. It is important for CPAs to consult with appropriate sources and to exercise their professional judgment when making decisions.
Reading through the takeaways, it seems like it added an extra dash of risk management. Could it have accounted for our risk-aversion?

Then I thought, why not ask it directly about the video itself

Question to ChatGPT: "What are 5 key takeaways from It’s Time to Pay Attention to A.I. (ChatGPT and Beyond)"

Answer:
  1. Artificial intelligence (AI) is a rapidly developing field with the potential to significantly impact many aspects of society, including the accounting profession.
  2. AI technologies, such as chatbots and language models like ChatGPT, are becoming increasingly sophisticated and can be used for a wide range of applications.
  3. It is important for CPAs to stay informed about the latest developments in AI and to consider how these technologies may impact their profession and their clients.
  4. While AI has the potential to improve efficiency and accuracy, it is important to carefully consider the risks and limitations of these technologies, including the potential for errors or biases.
  5. CPAs should consider ways to incorporate AI into their practices, such as by using AI tools to automate certain tasks or by providing guidance to clients on the use of AI. However, they should also ensure that they have the necessary skills and knowledge to effectively use and understand these technologies.
Notice how I did not have to tailor this questions for CPAs? ChatGPT automatically brought the context over from the previous question.  (As for how good this summary is, you can judge for yourself.)

Then, I decided to ask it about how exponential technologies will impact the CPA profession. 

First question, "Will blockchain replace the need for financial auditors?"

Answer:

"Blockchain technology has the potential to revolutionize many aspects of the financial industry, including auditing. However, it is unlikely that blockchain will completely replace the need for financial auditors.

One of the main features of blockchain is its ability to provide a tamper-evident record of transactions. This can be beneficial for auditing purposes, as it allows auditors to more easily verify the accuracy and completeness of financial records. However, auditors still need to be able to interpret and analyze the data contained in these records, and to ensure that the data is consistent with other information and documentation.

Additionally, while blockchain can provide a secure and reliable record of transactions, it is not a substitute for the expertise and judgment of trained financial professionals. Auditors bring a deep understanding of financial principles and practices, as well as the ability to identify and assess risks and potential areas of concern.

Overall, it is likely that blockchain will augment and enhance the role of financial auditors, rather than replacing them entirely."

Notice the nuanced response? It should have spoke to validity, instead of completeness when it came to how the blockchain will assist the auditor. However, it didn't just list all the audit objectives just "to be safe". 

Then I asked about artificial intelligence:

Question: "How can AI be used to automate financial auditing?"

Answer:
"Artificial intelligence (AI) can be used to automate certain aspects of financial auditing, such as the analysis and interpretation of financial data. This can potentially improve the efficiency and accuracy of the audit process.

For example, AI algorithms can be trained to analyze large amounts of financial data and identify patterns and trends that may not be immediately apparent to human auditors. This can help auditors to more quickly and effectively identify areas of potential risk or concern.

AI can also be used to automate the testing of transactions and the evaluation of internal controls. For example, an AI system could be trained to perform automated tests on transactions to ensure that they are recorded correctly in the financial records. This can help to reduce the time and resources needed to perform these tasks manually.

Overall, the use of AI in financial auditing can help to improve the efficiency and accuracy of the audit process, while also freeing up auditors to focus on more complex and higher-level tasks that require human judgment and expertise."

Recognize the theme of "race with the machine" in both this response and the previous blockchain response? This is especially integrated into the last paragraph, which speaks to what AI can do versus what "human judgment and expertise" can do. Also, notice how it explains AI role in both risk assessment and testing of internal controls. That is, it had enough where "understanding" to breakdown the response into different aspects of the audit. 

When looking at this, we need to ask ourselves: is this inflection point for exponential rise of AI? 

Do review the last part of the video, where Samuel H. Altman, CEO of OpenAI, explains how he anticipates the impact of such technology on the legal profession. (The video earlier takes about how Josh Browder's Do Not Pay, will leverage the tech. I had previously seen Browder on a panel in 2016; see this post for the video). 

It's not quite a stretch to apply what he says to the CPA profession. There are key differences; in that we opine on financial statements, provide tax advice based on financial data, and the like. However, audit data analytics tech has been around for decades. It's just a matter of getting the different parts to talk to each other. 

Clearly, it's early days for ChatGPT and many issues need to be sorted out. For example, it has already earned the moniker "CheatGPT" for how it can be potentially used as a short-cut by students. That being said, it's clearly the biggest watershed moment for AI and the white-collar workforce, since IBM's Watson defeated Ken Jennings and Brad Rutter

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else

Wednesday, December 14, 2022

Figuring out FTX (Part 3): 5 Videos You Must Watch on SBF’s Arrest!

Sam Bankman-Fried (SBF) was scheduled to testify in front of congress on Tuesday. Forbes caught a copy of his testimony, which is available here.  However, this testimony won’t be delivered. Instead, SBF got arrested in the Bahamas. The US Department of Justice (DoJ), in a joint conference with the SEC and CFTC, unveiled the following 8 charges:

  • Conspiracy to Commit Wire Fraud on Customers
  • Wire Fraud on Customers
  • Conspiracy To Commit Wire Fraud on Lenders
  • Wire Fraud on Lenders
  • Conspiracy to Commit Commodities Fraud
  • Conspiracy to Commit Securities Fraud
  • Conspiracy to Commit Money Laundering
  • Conspiracy to Defraud the United States and Violate the Campaign Finance Laws

 

CryptoSlate reported that “the Securities and Exchange Commission (SEC) and the Commodity Futures Trading Commission (CFTC) have both filed separate charges against the ex-CEO…The SEC has charged SBF with offering securities for sale and selling securities for fraud. Specifically, the charges fall under the Securities Exchange Act of 1934 and the Securities Act of 1933 pertaining to anti-fraud provisions…Further, the CFTC charged SBF with fraudulent misstatements and omissions.” 

 

The SEC complaint is available here (and here is their press release). The CFTC’s complaint is available here (and here is their press release). The full press conference is available here.

 

Video #1 – Will SBF’s Ill Advised Media Tour contribute to his Downfall?

Since the November collapse, SBF has been giving interviews (e.g. NYTimesABCWSJBBC). This is of course against the advice of counsel.  Regardless, he has been talking up a storm, claiming to be the incompetent CEO and then apologizing for his screwups. That is, he was able to successfully bob-and-weave during these interviews and avoid admission of fraud.

 

That was until he was interviewed by Coffeezilla.

 

Coffeezilla admitted he too got outmaneuvered when he attempted to pin him down here and here. However, by reviewing his mistake and the mistake of others, he cornered SBF in this video and got him to admit to the lack of segregation of funds:

 



 

SBF, in contrast to his other interviews, got annoyed and accused Coffeezilla of monopolizing the interview time. Coffeezilla easily refutes this claim, noting SBF was factually incorrect. That being said, it would be surprising if this interview is not captured as part of the evidence that the DoJ, SEC, and CFTC will ultimately use against SBF.

 

Video #2 – WSJ Overview on the Players and Places

The following gives a good overview of what is happening. This includes the jurisdictions involved, the charges laid, and the Congressional hearings that are occurring in parallel.

 



See here for Coffeezilla’s analysis of the charges. The video also includes commentary from Legal Eagle.  

 

Video #3 – Summary of the Criminality at FTX and Alameda

 

FTX’s CEO, John J. Ray III, testified before congress about his findings so far. Ray was the same CEO that managed the Enron bankruptcy and has previously said that he has never seen “a complete failure of corporate controls and such a complete absence of trustworthy financial information as occurred [at FTX]”. This video, from the Washington Post, seems to capture the essence of the criminality that was a foot at the crypto-exchange and the hedge fund:

 


 

Video #4 – Did the multi-Billion Dollar FTX use QuickBooks and Slack?


For the poor state of governance and controls, see Ray’s opening remarks during his congressional testimony:
 


And to answer the question, yes they did!



Video #5 – The Nature of Crypto is making it hard for Ray to Locate FTX’s Assets

 

Locating assets in a bankruptcy is usually a matter of following the paper trail. Admittedly, FTX has poor records as noted in the last video. However, the problem is compounded by the ethereal nature of crypto/digital assets. See WSJ’s summary of Ray’s testimony, which highlights this unique challenge within FTX’s bankruptcy proceedings:

 


 The FTX story continues to evolve. There is definitely more to discuss, as additional details come to light. However, another story that will be explored in future post(s) is the contagion that has spread through the crypto-verse due to FTX and the other collapses in the nascent industry.

 

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

 

Thursday, December 8, 2022

Figuring out FTX (Part 2): The Ten Days In November that Brought FTX Down

In our last post, we looked at the epic rise of SBF and FTX. In this post, we examine the Ten Days of November that shook FTX to its core and resulted in its spectacular collapse.

Post #1: The CoinDesk FTX Timeline

The best place to start is this timeline, which is taken from this CoinDesk post. I’ve also added the amount withdrawn from FTX, which was taken from this Reuters article:


Post #2: The Leaking of the Alameda Balance Sheet

What must be said, the much vaunted transparency of the blockchain did not bring down the FTX empire. Instead, it was classic journalism at CoinDesk. The killer quote from the article:

“That balance sheet is full of FTX – specifically, the FTT token issued by the exchange that grants holders a discount on trading fees on its marketplace. While there is nothing per se untoward or wrong about that, it shows Bankman-Fried’s trading giant Alameda rests on a foundation largely made up of a coin that a sister company invented, not an independent asset like a fiat currency or another crypto. The situation adds to evidence that the ties between FTX and Alameda are unusually close.”

As noted in the above timeline, this is what prompted Changpeng Zhao (CZ) to tweet this and then caused the billions to be withdrawn, as mentioned in the Reuters article.

Post #3: Prelude to the FTX Collapse

The first of Coffeezilla’s video on the collapse really captures not just the rivalry between SBF and CZ, but the killer-business logic that was potentially at play. Far from the crypto-utopian visions of an egalitarian ecosystem, we see the same sort of cutthroat competition in the banking world itself. For example, one theory holds that Bear-Stearns collapse was triggered in the 2007-2008 Financial Crisis. The reason? Payback. Bear-Stearns did not help out in the Long-Term Capital Management (LTCM) bailout and so Goldman-Sachs returned the favour almost 20 years to the day.  



Post #4: FTX and the Mystery of the Stolen Crypto

This video, published 3 days after the last, explores the complex web of relationships that is FTX (far more complex than Lehman), but zooms in on the entanglement between Alameda Research and FTX. The big reveal here is that an Alameda insider noted that “not only did they [Alameda Research] have access to FTX's back end [but] they [also] managed withdrawals for FTX and had a giant line of credit that they could draw on, which seems like partially may have been users funds something that no separate entity would normally have”.  The insider was corroborated by the Wall Street Journal.


Post #5: An Inside Look at the Chaos and Ineptitude at FTX/Alameda

Shout out to Tim Bauer for passing on this link from MilkyEggs! (Bloomberg’s Matt Levine, also referred to the post here with all the necessary caveats). It gives more details around the sheer chaos and ineptitude that existed at FTX and Alameda. With respect to the chaos, it gives some details around SBF’s mental state. It is quite the contrast to the image that was portrayed to the outside world, which we saw in the first video in the last post. With respect to ineptitude, it highlights the “farcically simplistic” accounting records the company kept.  

In terms of the top three takeaways, it firstly casts doubts on the origin story of SBF. The post alleges (based on an insider) that SBF quickly lost all the wealth he made from those bitcoin US vs Japan arbitrage trades. Secondly, it gives some insight into the inordinate amount of risk SBF was taking. Lastly, it attempts to breakdown the losses incurred by FTX-Alameda. That is, they attempt to piece together where the money - $15.5 billion in total – was spent. Also, do check out the postscript where they “found” another $3 billion in losses. Of course, this is not an official audit or anything like that. However, it’s nice to get a more wholistic understanding of the FTX-Alameda situation – beyond the puff pieces in the mainstream press.

In our next post, we will begin exploring the aftermath of FTX collapse.

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

 

Thursday, November 24, 2022

Figuring out FTX and SBF (Part 1): The Epic Rise in 5 videos

The recent weeks have seen the fall of crypto-king Sam Bankman-Fried (aka SBF). The epic rise and the spectacular fall of him and his FTX empire has us all asking one question: what happened?

We’ll initially explore this saga over several posts. 

 

In this post, we look at 5 videos that cover the rise of SBF and his FTX crypto-exchange. We’ll start by looking at how he initially got rich through arbitrage trading, look at the FTX “business model”, his connections to the US government, and then close with the celebrity endorsements that helped propel him to stardom.

 

Video #1: The Benevolent Billionaire?

Probably my first encounter with SBF, was in his interview with Nas Daily. We learn how SBF rose to fame in the crypto space by taking advantage of price differences in bitcoin in the US and in Japan. More importantly, the video captures the hope and the hype around the “earn to give” concept that SBF promoted. Turns out it was all hype and no hope. Vox published their twitter DMs with SBF, where he explains “I feel bad for those who get f***** by it…this dumb game we woke westerners play where we say all the right shiboleths [sic] and so everyone likes us” (link).

 



 

Video #2: Did SBF really say that Crypto is a Ponzi Scheme – 7 months ago?

On Bloomberg’s Odd Lot’s podcast aired back in April 2022, SBF seems to admit to Wall Street’s Wiseman Matt Levine that “magic internet money” is a Ponzi scheme. If you don’t have time to listen to the whole podcast, check out Coffeezilla’s quick takes. The technique that SBF discusses appears to  have been used in Celsius, which we discussed previously.  

 



 

Video #3: There are magic boxes, but magic business models?

On a separate but related note, FTX offered high rates of interests to depositors. As shown in Wall Street Millennial’s video, the rates were much higher than the fed rate that was close to zero at the time. This is not so much about the magic money box that SBF referred to in the previous video, but rather pure magic. How else can we explain the sustainability of such a high interest rate on bitcoin/Ethereum deposits? As CPAs know, when expenses exceed revenues – a business failure is inevitable.  (Click here to go to the timestamp where they discuss the issue of interest rate deposits)

 


Video #4: If you’re for regulations, you’re one of the good guys – right?

Stepping back, Cold Fusion’s video gives a good broad understanding of the key events that happened with the rise and sudden fall of FTX. (The previous video is also quite good as well). That being said, this video takes time to highlight SBF’s connection with the government and regulatory bodies.

 

Firstly, his mom is connected to the Democratic party. Secondly, he testified before congress, donated to both parties (not just the Democrats), and met with Gary Gensler from the SEC. If you look carefully at the screen shot where he speaks about the meeting with Gensler, you may recognize IEX’s Brad Katsuyama. (Reuters reported here on this meeting). He’s the protagonist of Michael Lewis’s Flash Boys. And that’s not the only connection to Lewis. Reports have emerged that Lewis had spent 6 months with SBF and is planning to publish a book about him. Lastly, there is an odd connection between FTX’s CEO, Caroline Ellison, and Gary Gensler. As Cold Fusion points out, Caroline’s father, Glenn Ellison, was Gensler’s boss at MIT.



 

Video #5: Is FTX a smart bet, with all these celeb endorsements?

Both Wall Street Millennial and Cold Fusion pointed out the role of influencers played in pumping the FTX/SBF brand name. But we don’t just need to take their word for it. We can see it for ourselves. Some of the promo videos are still live on FTX’s YouTube Channel. This video features Canada’s own Kevin O’Leary:




If you didn’t have a chance to go through the whole video, check out this part where he and the host emphasize the compliance-orientation of SBF and FTX. Now, that definitely didn’t age well. But perhaps what’s more surprising is O’Leary’s comments post-FTX meltdown. Apparently, he would back SBF in another crypto-venture (link).

 

Lastly, if you are interested in seeing the actual adverts aired on TV, check out this one featuring the actor Larry David.

 

In our next post, we’ll look at some more videos that take a closer look at the “Ten Days in November” that broke FTX and could have possibly ushered in the “Crypto Ice Age”.


Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

 

Tuesday, November 1, 2022

Lessons Learned: Flashback to Summer’s Great Rogers Outage (Part 2)

In our last post, we looked at the Great Rogers Outage of 2022.

Millions of Canadians experienced life without mobile and Internet service – a necessity in our pandemic life. The cause was traced back to a system-change gone wrong. It appears that though Rogers had tested some parts of the planned change, it was insufficient to identify all the issues. The result was that the network got flooded with traffic and then the systems went down.

 

What are some lessons we can learn from this outage?

Major Controls Frameworks, like COBIT and ISO27001, and audit standards, like SOC2, require that management implement change management controls. Consequently, the outage presents a unique opportunity to understand what can go wrong when it comes to change management. Moreover, it highlights what types of controls are relevant from a real-live scenario - as Rogers documented in its submission to the CRTC. 


With that in mind, let’s look at four lessons from the Great Rogers Outage of 2022. 


Lesson #1: The Importance of Redundancy

When commenting on the impact of the outage on governments within Canada, Rogers noted: “It is important to note that in most of the cases, we provide a portion of the telecommunications solution, but not all underlying services. Many institutional customers have redundant services” [emphasis added].


Also, as previously noted that they had “established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity.”


The implication of this lesson is that we should try to diversify the telecom providers within our professional and personal lives. For example, my personal device is provisioned through Fido (a Rogers sub-brand), while my work cell is provisioned through Bell.  


Lesson #2: Test, Test, Test

They say in real-estate it’s about location, location, location. In change management it’s test, test, test. In the aftermath of the outage, Rogers doesn’t deny that they need to review their change implementation process:

“Most importantly, Rogers is examining its “change, planning and implementation” process to identify improvements to eliminate risk of further service interruptions.”


To be fair, it’s not like there was no testing done. Instead, Rogers had used a phased approach to rolling out the change:

Concerning the July 8th outage, the proposed activities were very carefully reviewed, as we normally do with all network changes. We validated all aspects of this change.  In fact, we had begun introducing this change weeks ago, on February 8th and had already implemented successfully the first five (5) phases in our core network.” [emphasis added]


It’s a good reminder that in the world of IT General Controls, and IT Risk Management more broadly, it’s not about what goes right but what goes wrong. Consequently, companies should ensure that the scenarios tested are comprehensive enough to identify hidden assumptions or dependencies. For example, Rogers had a procedure that relied on “alternate carrier SIM Cards”. Hypothetically, testing whether this worked ahead of time could help identify whether the employee could find their SIM cards or how they activated such SIM cards when they have no Internet.


Lesson #3: Planning Crisis Communications from Content to Channels

According to the Rogers submission, the company conducted the following communications:

“During the outage, Rogers communicated with customers across several different channels, including social media, media outlets, Rogers Sports & Media properties, website banners, virtual assistants, interactive voice responses (“IVR”), public service announcements and community forums. In addition, Rogers’ CEO conducted broadcast interviews with CP24, Global News, CTV News, BNN, and CityNews. Rogers SVP of Access Networks & Operations also conducted broadcast interviews on CBC and CityNews.”


The following CBC news clip illustrates what was communicated and how:



As can be seen, the reporter was a little surprised that they got message from the IT team – instead of Rogers themselves. However, Rogers did admit that they “will be updating [their] plans and procedures”. Specifically, they plan to:

  • Equip the communications team with “back-up devices on [an] alternate network”
  • Be more timely “in posting details to customer care channels, web properties, social media, as well as public service announcements (“PSAs”) across media properties”
  • Provide more frequent updates “even if there is limited or no additional information to share”
  • Determine an alternative way for the communications team to authenticate themselves, when the second-factor registered with the social media service is reliant on “a device on the Rogers network”
  • Provide specific “status of critical services (such as 9-1-1), how they may be impacted by the outage, and advice for customers”


The outage is a good illustration of how critical crisis communications can be. Maintaining effective communications with customers or other stakeholders is key to minimizing the reputational damage that such incidents can potentially have.


Lesson #4: Monitoring

The final takeaway is the importance of having resources and tools to monitor the restoration efforts. That is, the fixes deployed may not resolve all the issues. Rogers reported the following results with respect to bringing things back online:

“Once the technology team confirmed stability of our core network, and that traffic volumes were returning to normal level across the network, we proceeded to inform customers that our network and systems were returning to fully operational service for the vast majority of our customers. We also notified them that some customers may experience intermittent issues, and that our technology teams are monitoring and would work to resolve any issue as quickly as possible.” [emphasis added]


As can be seen, Rogers was able to restore the service for the vast majority of customers. However, there were a few that still experienced lingering issues. Consequently, it’s important to have continuous monitoring in place to ensure that the service is restored fully before returning to business as usual.

 

Closing thoughts

The incident highlights how dependent society has become on the wireless carriers for the day-to-day transactions and functioning of society. Vass Bednar (also interviewed in the above CBC newsclip) summarized the situation in an op-ed in the Globe and Mail as follows: 


“Enormous advances in mobile tech have made Canada's telecoms enormously powerful, and that power has consolidated in just five major players. That number threatens to get smaller, too, with the proposed Rogers-Shaw merger currently under review by Canada's Competition Bureau. If the deal goes through, the company that caused so many Canadians to lose connection with each other would serve roughly 40 per cent of all households in English Canada… it reinforced the idea that our telecommunication networks are vital public infrastructure that is controlled by private corporations. We've lost sight of that balance, despite the ways we rely on those networks.”


As discussed in the first takeaway, the issue of redundancy is paramount when it comes to ensuring ongoing access. Ironically, the lack of sufficient alternatives in the mobile carrier space amplifies the availability risk for us all.


Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

Tuesday, October 4, 2022

Fiona’s Fury: Flashback to Summer’s Great Rogers Outage (Part 1)

Canadians continue to pick up the pieces after tropical storm Fiona battered the maritime provinces. Although estimates of the damage are yet to be calculated, the “Nova Scotia Premier Tim Houston announced over C$40 million ($29.1 million) in support to help those who were impacted by Fiona” (link). In terms of cellphone outages, CBC reported that “there are still areas of the province without cellphone service Monday although companies declined to say exactly  how many customers have been affected.”

 

The Canadian Radio-television and Telecommunications Commission (CRTC) has asked for estimates on how many people were affected by the outage, but the telecom companies are reticent to share this information. As CBC reported: “Bell and Telus asked for some of their submissions to be redacted, while Eastlink and Rogers demanded their entire reports be kept confidential.”


Photo by Pixabay: link

 

Rogers Outage in Review: What happened last summer?

When looking at the outage that hit the Maritimes, it reminds us of the situation that unfolded over the summer. In July 2022, the Rogers outage was not limited to the East Coast. Instead, it affected the entire country. When Rogers was requested to explain what happened, it appears that they had a more conciliatory tone:

Rogers Communications Canada Inc. (“Rogers”) is in receipt of a letter containing Requests for Information (“RFIs”) from the Canadian Radio-television and Telecommunications Commission (“CRTC” or the “Commission”), dated July 12, 2022, concerning the above-mentioned subject. Attached, please find our Response to that letter… At the outset, Rogers appreciates the opportunity to explain to the Commission, the Government of Canada and all Canadians what transpired on July 8th, 2022.  The network outage experienced by Rogers was simply not acceptable. We failed in our commitment to be Canada’s most reliable network. We know how much our customers rely on our networks and we sincerely apologize.” [Emphasis added]

 

Though the documented was redacted, it did provide some background as to what happened. For this post, we will take a look at the outage itself. For the next post, we will look at the lessons learned.  

 

Cause of the outage

Rogers explained the cause of the outage as follows:

“Given the magnitude of the outage, it appears that Rogers had to be more forthcoming with what happened and were “Maintenance and update windows always take place in the very early morning hours when network traffic is at its quietest. At 4:43AM EDT, a specific coding was introduced in our Distribution Routers which triggered the failure of the Rogers IP core network starting at 4:45AM… The configuration change deleted a routing filter and allowed for all possible routes to the Internet to pass through the routers. As a result, the routers immediately began propagating abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their capacity levels and were then unable to route traffic, causing the common core network to stop processing traffic. As a result, the Rogers network lost connectivity to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers.” [Emphasis added]

In other words, the change inadvertently resulted in an attack pattern similar to a denial-of-service attack – where the network shutdown because it became overwhelmed with traffic.

They also go on to explain that the company “uses a common core network, essentially one IP network infrastructure, that supports all wireless, wireline and enterprise services. The common core is the brain of the network that receives, processes, transmits and connects all Internet, voice, data and TV traffic for our customers… Certain network routing equipment became flooded, exceeded their memory and processing capacity and were then unable to route and process traffic, causing the common core network to shut down.” The implication being that the common core network became a single point of failure.

 

What was and was not impacted

With respect to Rogers Bank (yes, Rogers operates a bank):

“The impact to the Bank’s customers was minimal as the Bank services were available and the Bank’s customers were able to transact on their Rogers Bank credit cards. There was no interruption in the Bank’s core systems (credit card processing, Interactive Voice Response (“IVR”), Call Centre and customer self-serve mobile application) and these core systems remained available to the Bank’s customers. No critical Bank systems were impacted, and all daily processing was completed as required, including by the Bank’s statement printing vendor and its card personalization bureau which received their daily files and were processing them per standard service level agreements and procedures.”

 

This was a different story for those that relied on Rogers phone lines to process payments at their businesses with Interac tweeting:

“There is a nationwide Rogers outage that encompasses all their business and consumer network services. This is impacting INTERAC Debit and INTERAC eTransfer. INTERAC Debit is currently unavailable online and at checkout..

 

Beyond the millions who had no service, emergency communications were also impacted:

  • “Unfortunately, the outage of July 8th did impact 9-1-1 service across Rogers’ service area, to both wireline and wireless services.
  • Wireline impact:  There were approximately [REDACTED] 9-1-1 calls placed successfully across Rogers’ network on July 8th.  The typical daily average of total wireline 9-1-1 calls is [REDACTED] per day. Data is unavailable for unsuccessful wireline 9-1-1 calls.  On July 9th, there were approximately [REDACTED] 9-1-1 calls placed successfully across Rogers’ network.
  • Wireless impact: As can be seen in table below, the outage similarly affected wireless 9-1-1. Total successful calls were [REDACTED] the average daily amount of about [REDACTED] 9-1-1 calls made from Rogers wireless devices.
  •  

Rogers offered service outage credits

The key remedy offered was service credits, but this was not due to breaches in service agreements:

“There was no breach of our service agreements with our retail customers. However, in order to address our customers’ disappointment with the outage, Rogers has already announced it will be crediting 5 days of service fees to its customers. This will be applied automatically to their next invoice.”

 

Cooperation with Bell and Telus

Regardless of the highly-competitive nature of the business, it does appear the Rogers, Bell and Telus were coordinating with each other:

  • “On July 17th, 2015, the Canadian Telecom Resiliency Working Group (“CTRWG”), formerly called Canadian Telecom Emergency Preparedness Association, established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity.”
  • “As we stated in Rogers(CRTC)11July2022-1.xviii above, our Chief Technology and Information Officer reached out to his counterparts at Bell and TELUS early on July 8th. Assistance was offered by both Bell and TELUS. However, given the nature of the issue, Rogers rapidly assessed and concluded that it was not possible to make the necessary network changes to enable our wireless customers to move to their wireless networks.”
  • “Rogers, Bell and TELUS are presently assessing potential options and will report further findings and potential solutions per the creation of the Memorandum of Understanding that will be delivered in September 2022 to the Minister of ISED by CSTAC.”

In closing, the outage comes down to change management. The error was exacerbated by the industry-standard approach to using a single platform to provide the various telecommunication services. Rogers did offer service credits, but were careful to note that this was not due to breach of agreements. Finally, the industry does come together during crisis situation, putting their competitive differences aside. 


In our next post, we’ll take a look at the lessons learned from this outage. Stay tuned!

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

Friday, August 5, 2022

Time to Upgrade the Internet? A look at the hope and hype around Web3

Is the Internet ready for a version upgrade? Some blockchain enthusiasts think so, but others - Tim O'Reilly in particular - think we need to hold off. 

What is Web3?

Deloitte, for its part, sees the Web3 as part of a larger concept of the “semantic web”:

“Many people identify Web 3.0 with the Semantic Web, which centers on the capability of machines to read and interact with content in a manner more akin to humans. Recently, definitions of Web 3.0 have begun to include distributed ledger technologies, such as blockchain, focusing on their ability to authenticate and decentralize information. Theoretically, this could remove the power of platform owners over individual users.”

Gartner links the origins of the term to “Gavin Wood, co-founder of Ethereum, who argues that centralization is not socially tenable long-term. Also called Web 3 and Web 3.0, Web3 eliminates the need for, and functions of, Web 2.0 central authorities and “gatekeepers,” such as major search engines and social media platforms.” [Emphasis from the original quote]

Ethereum, while admitting “it's challenging to provide a rigid definition of what Web3”, lists 4 “core guiding principles, including decentralization, permissionless, use native payments (i.e., cryptocurrencies instead of “outdated infrastructure of banks and payment processors”), and trustless (e.g. relies on miners instead of “trusted third-parties”).

What does Tim O’Reilly, Bill Gates, and Gartner have to say about this?

Tim O’Reilly coined the term “Web 2.0” back in 2005. According to his seminal post on the topic, he introduces the jump from Web 1.0 to Web 2.0 by looking at how Google (which he believes is “the standard bearer for Web 2.0”) compares to Netscape. Specifically, he notes that “the value of the software is proportional to the scale and dynamism of the data it helps to manage.”. He also touches on a number of other concepts, including the ability to harness the wisdom of the crowds, cloud computing, as well as the long tail

The original post is worth the read because it gives a benchmark of sorts as to what does “good look like” when claiming the web has gone through a version upgrade.

In terms of what O’Reilly thinks about Web3, it can be found here. He summarizes his primary challenge in a single sentence:

“None of the examples in the article focus on the utility of what is being created, just the possibility that they will make their investors and creators rich.”

The article he is referring to was this one published by NY Time in the fall of 2021. The article mentions, social media, collectibles, and gaming.

Bill Gates is a bit more direct:

“Speaking at a TechCrunch talk on climate change Tuesday, the billionaire Microsoft co-founder described the phenomenon as something that’s “100% based on greater fool theory,” referring to the idea that overvalued assets will go up in price when there are enough investors willing to pay more for them… Gates joked that “expensive digital images of monkeys” would “improve the world immensely,” referring to the much-hyped Bored Ape Yacht Club NFT collection.”

Regardless, O’Reilly and Gates end-up in the same place. Compared to the Railway, Radio, and Internet Bubbles of the past, there is no infrastructure being built here to move people/goods, broadcast programming through the air, or enable the routing of packets of information in a dynamic way that enables us to work from home during a pandemic.

In contrast, there is literally nothing when it comes to crypto. With bitcoin, you do not actually have a tangible thing to hold on to; there are no digital coins or pieces of code to point to. Instead, your holding are mathematical calculation of your “ins” and “outs” (see here for our post/process flow of bitcoin).  Sure, that’s part of the security – but from an economic perspective that is quite a difficult pill to swallow. Add on top of that, there is no centralized intermediaries to turn to when things don’t work out with these “assets” – you have massive issues in understanding how this different than people paying fortunes for tulip bubbles, I mean bulbs.

As noted in the previous post on NFTs, I do think that NFTs offer some type of infrastructure to the future. O’Reilly is not so sure. However, what we do agree is there massive gap on the institutional side of things:

“The failure to think through and build interfaces to existing legal and commercial mechanisms is in stark contrast to previous generations of the web…The easy money to be made speculating on crypto assets seems to have distracted developers and investors from the hard work of building useful real-world services.”

O’Reilly points out that the Web 2.0 – despite the DotCom Crash – still had successful ventures that could be pointed to, such as Amazon and Yahoo that were making money, hiring people, providing services to millions of users and “had all built unique, substantial, and lasting assets in the form of data, infrastructure, and differentiated business model”.

And Gartner?

Gartner on a recent blogpost unveiling its Hype Cycle for Blockchain and Web3, 2002, made an important observation:

“In the meantime, other than cryptocurrency trading, we still have not seen killer use cases yet. They need to leapfrog over current applications in terms of making our lives better.”

Though Web3 is something new, there’s a lot more that needs to be done before it can be crowned a Version 3.0 of the World Wide Web. 


Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else