Big Data Auditing Revisited: Context is King

It has been a few years since I wrote up on Big Data and the Audit.  It was one of the more popular posts with over a 1,000 hits to date.

The post looks at Big Data: A Revolution That Will Transform How We Live, Work, and Think by Kenneth Cukier and Viktor Mayer-Schönberger. I enjoyed the book as it really broke down the business impact of big data without getting in technical details of the underlying technology.

Why take a second look at big data auditing?

Big data and the accompanying analytical models are key a precursor to artificial intelligence. Machine learning algorithms that power the AI bots requires the users to analyse the problem and teach the underlying algorithm.

Part 1: Context is King

To make things a bit more digestible, I thought it would be good to divide the post into two parts. The first post is more palatable as I want to explore the second use case in a bit more detail and its relevance to today.  The second post will be a bit more controversial as I will take a look at the difficulty of applying fraud or cancer-fighting algorithms in the realm of (external) financial audit.

But let's look at the first issue: how can big data analytics give us better context? 

In the original post, I spoke discussed the use case used in Cukier and Mayer-Schönberger's work around Inrix. The book gives the example of how an investment firm is using traffic analysis, from Inrix, to determine the sales that a retailer will make and then buy or sell the stock of the retailer on that information. In a sense, the investment is using vehicular traffic as a proxy for sales. In an audit context, auditors can develop expectations of what sales should be based on the number of vehicles going around stores. For example, if sales are going up, but the number of vehicles are going down then the auditor would need to take a closer look.

What I realized from this example is that what big data can give auditors better context around things and assess reasonability of things. That is as more sensor data and other data are available to auditors to integrate into statistical models, the more they will be able to spot anomalies. 

One of the issues with Barry Minkow's ZZZBest accounting fraud was the lack of context. For more on the fraud, check this video:

I actually studied this case in my auditing class at the University of Waterloo. One of the lessons we were take away from this case was that the auditors didn't know how much a site restoration would cost on average (see the first bullet in this text on page 129). But how would an auditor be able to access such data? Even with the advent of the internet, it is not simply a matter of Googling for the information.

More recently, an accounting professor was found to have generated data fraudulently. The way he got caught was that a statistic he used didn't correspond to reality. Specifically:

"misrepresented the number of U.S.-based offices it had: not 150, as the paper maintained (and as a reader had noticed might be on the high side, triggering an inquiry from the journal)" [Emphasis added]

Again, the reader had the context to understand what was presented was unreasonable causing the study to unravel and exposing the academic fraud perpetrated by Hunton. 

What will it take to make this a reality? 

What's missing is a data aggregation tool that can connect to the private, third party, and public data feeds that an auditor can leverage for statistical analysis. Furthermore, for this to be useful to clients and the business community large are visualized depictions that enable the auditor to tell the story in a better way rather than handing over complex spreadsheets.

Of course for auditors to present such materials requires them to have deeper training in data wrangling, statistics and visualization tools and techniques. 

In the next post, we will revisit the first use case that I presented in the original post that explored how the New York City was better able to audit illegal conversions through the use of big data analytical techniques. Originally, I had thought this would be a good model to apply in the world of audit. However, I am revisiting this idea. 

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else

SEC and the Quants: Will RoboCop get a BigData overhaul?

As reported in this Forbes article in 2013, the SEC began to use so-called RoboCop to assist with their regulatory duties.

Who is RoboCop?


No, it's not that infamous crime-fighting cyborg from the late-80s (coincidentally remade in 2014). It is actually the Accounting Quality Model (AQM) - not quite as exciting I know. According to Forbes:

"AQM is an analytical tool which trawls corporate filings to flag high-risk activity for closer inspection by SEC enforcement teams. Use of the AQM, in conjunction with statements by recently-confirmed SEC Chairman Mary Jo White and the introduction of new initiatives announced July 2, 2013, indicates a renewed commitment by the SEC to seek out violations of financial reporting regulations. This pledge of substantial resources means it is more important than ever for corporate filers to understand SEC enforcement strategies, especially the AQM, in order to decrease the likelihood that their firm will be the subject of an expensive SEC audit."

Another interesting point raised by the Forbes article is the use of XBRL in this accounting model: "AQM relies on the newly-mandated XBRL data which is prone to mistakes by the inexperienced. Sloppy entries could land your company’s filing at the top of the list for close examination."

(On a side note: AICPA has published this study to assist XBRL filers ensure that they are preparing quality statements, given that there are many possible errors; as noted in this study).

Within this context, we should take note of how the SEC is hiring "quantitative analysts" (or "quants" for short). As noted in this WSJ article:

"And Wall Street firms, for their part, are able to offer quantitative analysts—or “quants”—far higher pay packages than the regulator. The SEC’s access to market data also remains limited. In 2012, it approved a massive new computer system to track markets, known as the Consolidated Audit Trail, but the system isn’t likely to come online for several years, experts say."

Could the SEC pull a fast one and become the source of innovation? Although the WSJ article seems to downplay the possibility that the SEC can outpace the firms, it is not something that the audit industry can ignore.

As noted in a previous post on Big Data, it was just this type of mindset that Mike Flowers of New York City looked to revolutionize how the NYC leveraged big data to improve its "audit" of illegal conversions. Perhaps the SEC may follow in his stead.

Worth mentioning: KPMG's take on the state of tech in the audit profession

In a recent post (as in just this week) on Forbes, KPMG's  James P. Liddy who is the Vice Chair, Audit and Regional Head of Audit, Americas put out a great post that summarizes the current state of analytics in financial audits.

He diplomatically summarizes the current state of the financial audit as "unchanged for more than 80 years since the advent of the classic audit" while stating "[a]dvances in technology and the massive proliferation of available information have created a new landscape for financial reporting. With investors now having access to a seemingly unlimited breadth and depth of information, the need has never been greater for the audit process to evolve by providing deeper and more relevant insights about an organization’s financial condition and performance –while maintaining and continually improving audit quality." [Emphasis added]

For those that have started off our careers in the world of financial audit as professional accountants and then moved to the world of audit analytics or IT risk management, we have always felt that technology could help us to get audits done more efficiently and effectively.

I was actually surprised that he stated that auditors "perform procedures over a relatively small sample of transactions – as few as 30 or 40 – and extrapolate conclusions across a much broader set of data". We usually don't see this kind of openness when it comes to discussing the inner-workings of the profession. However, I think that discussing such fundamentals is inevitable given those outside the profession are embracing big data analytics in "non-financial audits". For example, see this post where I discuss the New York City fire department's use of big data analytics to identify a better audit population when it comes to identifying illegal conversions that are a high risk and need to be evacuated.

For those that take comfort in the regulated nature of the profession as protection of disruption, we should take note of how the regulators are embracing big data analytics. Firstly, the SEC is using RoboCop to better target financial irregularities. Secondly, according to the Wall Street Journal, FINRA is eyeing an automated audit approach to monitoring to risk. The program is known as "Comprehensive Automated Risk Data System" (CARDS). As per FINRA:

"CARDS program will increase FINRA's ability to protect the investing public by utilizing automated analytics on brokerage data to identify problematic sales practice activity. FINRA plans to analyze CARDS data before examining firms on site, thereby identifying risks earlier and shifting work away from the on-site exam process". In the same post, Susan Axelrod, FINRA's Executive Vice President of Regulatory Operations, is quoted as saying "The information collected through CARDS will allow FINRA to run analytics that identify potential "red flags" of sales practice misconduct and help us identify potential business conduct problems with firms, branches and registered representatives".

As a result, I agree with Mr. Libby: sticking to the status quo is no longer a viable strategy for the profession.

Big Data Audit Analytics: Dirty data, explainability and data driven decision making

This is the second instalment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ). In this instalment, I explore another example of Big Data Audit analytics noted in the book and highlight the lessons learned from it. 

Con Edison and Exploding Manhole Covers

The book discussed the case of Con Edison (the public utility that provides electricity to New York City)  and its efforts to better predict, which of their manhole covers will experience "technical difficulties" from the relatively benign (e.g. smoking, heating up, etc) to the potentially deadly (where a 300 pound manhole can explode into the air and potentially harm someone). Given the potentially implications on life and limb, Con Edison needed a better audit approach, if you will, then random guessing as to which manhole cover would need maintenance to prevent such problems from occurring.

And this is where Cynthia Rudin, currently an associate professor of statistics at MIT, comes into the picture. She and her team of statisticians at Columbia University worked with Con Edison to devise a model that would predict, where the maintenance dollars should be focused.

The team developed a model with 106 (with the biggest factors being age of the manhole covers and if there were previous incidents) data predictors that ranked manhole covers in terms of which ones were most likely to have issues to those least likely. How accurate was it?  As noted in the book, the top 10% of those ranked most likely to have incidents ended up accounting for 44% of the manhole covers with potentially deadly incidents. In other words, Con Edison through big data analytics was able to better "audit" the population of manhole covers for potential safety issues.  The following video goes into some detail on what the team did:

What lessons can be drawn from this use of Big Data Analytics?
Firstly, algorithms can overcome dirty data. When Professor Rudin was putting together the data to analyse, it included data from the early days of Con Edison, i.e. as in 1880s when Thomas Edison was alive! To illustrate the book notes how there 38 different ways to enter the word "service box" into service records. This is on top of the fact that some of these records were hand written and were documented by people who didn't have a concept of a computer let alone big data analytics.

Second, although the biggest factors seem obvious in hindsight, we should be aware of such conclusions. The point is that data driven decision making is more defensible than a "gut feel", which speaks directly to the professional judgement versus statistical approach of performing audit procedures. The authors further point out that there at least 104 other variables that were contenders and their relative importance cannot be known without preforming such a rigorous analysis.  The point here is that for organizations to succeed and take analytics to the next level need to embrace culturally the concept that, where feasible, organizations should invest in the necessary leg work to obtain conclusions based on solid analysis.

Third, the authors highlight the importance of "explainability". They attribute to the world of artificial intelligence, which refers to the ability of the human user to drill deeper into the analysis generated by the model and explain to operational and executive management why a specific manhole needs to be investigated. In contrast, the authors point out that models that are complex due to the inclusion of numerous variables are difficult to explain. This is a critical point for auditors. As the auditors must be able defend why a particular transaction was chosen over another for audit, big data audit analytics needs to incorporate this concept of explainability.

Finally, it is but another example of how financial audits can benefit from such techniques, given the way non-financial "audits" are using big data techniques to audit and assess information. So internal and external auditors can highlight this (along with two examples identified in the previous post) as part of their big data audit analytics business case.

Big Data: Towards a "data driven audit"

This is the first installment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ). In this installment, I explore why this is a "must read" for those interested in data driven decision making, big data and information. I will also discuss some examples included in the book that make the case for "data driven audits". 

Why read this book?

This book is written by journalist, Kenneth Cukier, (who claims in this video to have to used the term "big data" before it was commonly used) and Viktor Mayer-Schönberger (an Internet and Governance professor at University of Oxford).

Given the background of the authors, it is an easy to digest book that gives the reader a good understanding of how access to large volumes of data and the use of correlations will change the way business is done and how society has a whole functions - without going into the technical detail of how big data is "crunched" at the back end.  The authors also discuss the following:

• Why more is better: Algorithms improve by being exposed to more data - regardless of how messy it is. On the topic of size, it also comments how statistical sampling is a feature of an era when organizations could not wrap their arms around the data.
• Consumer and business implications: The book is filled with examples that anyone can relate to, such as predicting whether the price of an airplane ticket will go up or down, as well as how Google uses search queries to predict flu outbreaks.
• Enter "Datafication": It also distinguishes "datafication" versus "digitization", where the latter is making something into bits and bytes, whereas the former is something that can be analyzed by some sort of analytic engine. 
• Potentially challenges and negative consequences of big data driven decisions: One of the challenges cited by the author is the "black box" nature of algorithms: how does a common person challenge the an algorithm, when it takes a rocket scientist to understand the algorithm itself? The authors also take the risk of explaining the  danger of subordinating human decision making to algorithms. For example, they note it would be problematic for governments to round up and quarantine people just because they looked up terms related to the flu. 

There are other interesting pieces, but I will bring them up over the next few blog posts. But if you want to get a full understanding, please buy the book - it's worth it!

The case for data driven audits

The book is filled with examples that illustrate the power of big data and how they impact business and society. However, there are a couple of examples that illustrate how financial audits can benefit from such techniques, given the way non-financial "audits" are using big data techniques to audit and assess information. 

Case 1: New York City and Auditing Illegal Conversions: As discussed in this excerpt of the book, Mike Flowers applied big data techniques to the problems of "illegal conversion" in New York city. As noted in the article, illegal conversions is the "the practice of cutting up a dwelling into many smaller units so that it can house as many as 10 times the number of people it was designed for. They are major fire hazards, as well as cauldrons of crime, drugs, disease, and pest infestation. A tangle of extension cords may snake across the walls; hot plates sit perilously on top of bedspreads. People packed this tightly regularly die in blazes". The data scientists working for Flowers, took the 900,000 property lots in the city and correlated "five years of fire data ranked by severity" against the following pieces of data:

• Delinquency in paying property taxes,
• Foreclosure proceedings,
• Odd patterns in their usage of utilities, 
• Non-payment of utilities,
• Type of building,
• Date building was built,
• Ambulance visits,
• Rodent complaints,
• External brickwork.

By correlating all this and other information, they were able to improve the effectiveness of their 200 person inspection team from a "hit rate" of 13% to 70%. (Note: hit rate refers to conditions of building identified as being so bad that it warrants a "vacate order"). 

This is a pretty straightforward evidence for "data driven audits": financial auditors can identify correlations between financial data and non-financial data to determine which financial transactions need more scrutiny than others.

Not convinced?

Well, investors are already doing this. The book gives the example of how an investment firm is using traffic analysis, from Inrix, to determine the sales that a retailer will make and then buy or sell the stock of the retailer on that information. In a senses, the investment is using this as a proxy for sales. In an audit context, auditors can study the vehicular traffic around stores against the sales recorded against such stores and determine if there are issues worth investigating. 

Of course, this endeavor is not merely a matter of copying & pasting data from StatsCan and cobbling up a spreadsheet or two. It is a lot of hard work. However, this is not surprising to anyone who has been performing computer assisted audit techniques for the last decade or so. The challenge has always been in cleaning up the data and making it usable. Some of the challenges that the New York team of statisticians had, include:

• Inconsistent data formats: The team had to bring together data sets from 19 different agencies. Each agency had a different way of describing location. Consequently, this has to be standardized so that each of the 19 data sets can be correlated to the same property.
• Datafying expert intuition: The article describes how brickwork got added as an element to the correlation model. The data scientists on the team observed how the fire inspector could look at a building and know whether it was okay or not.
• Understanding significance of each variable: Each variable must be assessed in its own right to avoid the problem of generalization. For example, rodent infestation is not uniform in its significance across New York city. As noted in the article, " A rat spotted on the posh Upper East Side might generate 30 calls within an hour, but it might take a battalion of rodents before residents in the Bronx felt moved to dial 311".

Although such challenges exist, there's a promise for big data to transform the way audits are done If you recall my blog post on the Oracle vs Google trial, I made a similar point: "the profession should use this opportunity to re-examine what aspects of technology should be a part of the audit practitioners skill set - given that is clear that society's attitude towards technology has clearly changed". Applying this to data driven audits, is actually less of a stretch as auditors have the skill set of dealing with data - instead of learning to code in Java. As a profession,  we need t o understand what data is out there and how this data can be correlated with financial data to make a more effective, efficient, and insightful audit.