Big Data Auditing Revisited: Context is King

It has been a few years since I wrote up on Big Data and the Audit.  It was one of the more popular posts with over a 1,000 hits to date.

The post looks at Big Data: A Revolution That Will Transform How We Live, Work, and Think by Kenneth Cukier and Viktor Mayer-Schönberger. I enjoyed the book as it really broke down the business impact of big data without getting in technical details of the underlying technology.

Why take a second look at big data auditing?

Big data and the accompanying analytical models are key a precursor to artificial intelligence. Machine learning algorithms that power the AI bots requires the users to analyse the problem and teach the underlying algorithm.

Part 1: Context is King

To make things a bit more digestible, I thought it would be good to divide the post into two parts. The first post is more palatable as I want to explore the second use case in a bit more detail and its relevance to today.  The second post will be a bit more controversial as I will take a look at the difficulty of applying fraud or cancer-fighting algorithms in the realm of (external) financial audit.

But let's look at the first issue: how can big data analytics give us better context? 

In the original post, I spoke discussed the use case used in Cukier and Mayer-Schönberger's work around Inrix. The book gives the example of how an investment firm is using traffic analysis, from Inrix, to determine the sales that a retailer will make and then buy or sell the stock of the retailer on that information. In a sense, the investment is using vehicular traffic as a proxy for sales. In an audit context, auditors can develop expectations of what sales should be based on the number of vehicles going around stores. For example, if sales are going up, but the number of vehicles are going down then the auditor would need to take a closer look.

What I realized from this example is that what big data can give auditors better context around things and assess reasonability of things. That is as more sensor data and other data are available to auditors to integrate into statistical models, the more they will be able to spot anomalies. 

One of the issues with Barry Minkow's ZZZBest accounting fraud was the lack of context. For more on the fraud, check this video:

I actually studied this case in my auditing class at the University of Waterloo. One of the lessons we were take away from this case was that the auditors didn't know how much a site restoration would cost on average (see the first bullet in this text on page 129). But how would an auditor be able to access such data? Even with the advent of the internet, it is not simply a matter of Googling for the information.

More recently, an accounting professor was found to have generated data fraudulently. The way he got caught was that a statistic he used didn't correspond to reality. Specifically:

"misrepresented the number of U.S.-based offices it had: not 150, as the paper maintained (and as a reader had noticed might be on the high side, triggering an inquiry from the journal)" [Emphasis added]

Again, the reader had the context to understand what was presented was unreasonable causing the study to unravel and exposing the academic fraud perpetrated by Hunton. 

What will it take to make this a reality? 

What's missing is a data aggregation tool that can connect to the private, third party, and public data feeds that an auditor can leverage for statistical analysis. Furthermore, for this to be useful to clients and the business community large are visualized depictions that enable the auditor to tell the story in a better way rather than handing over complex spreadsheets.

Of course for auditors to present such materials requires them to have deeper training in data wrangling, statistics and visualization tools and techniques. 

In the next post, we will revisit the first use case that I presented in the original post that explored how the New York City was better able to audit illegal conversions through the use of big data analytical techniques. Originally, I had thought this would be a good model to apply in the world of audit. However, I am revisiting this idea. 

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else

Big Data Audit Analytics: Dirty data, explainability and data driven decision making

This is the second instalment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ). In this instalment, I explore another example of Big Data Audit analytics noted in the book and highlight the lessons learned from it. 

Con Edison and Exploding Manhole Covers

The book discussed the case of Con Edison (the public utility that provides electricity to New York City)  and its efforts to better predict, which of their manhole covers will experience "technical difficulties" from the relatively benign (e.g. smoking, heating up, etc) to the potentially deadly (where a 300 pound manhole can explode into the air and potentially harm someone). Given the potentially implications on life and limb, Con Edison needed a better audit approach, if you will, then random guessing as to which manhole cover would need maintenance to prevent such problems from occurring.

And this is where Cynthia Rudin, currently an associate professor of statistics at MIT, comes into the picture. She and her team of statisticians at Columbia University worked with Con Edison to devise a model that would predict, where the maintenance dollars should be focused.

The team developed a model with 106 (with the biggest factors being age of the manhole covers and if there were previous incidents) data predictors that ranked manhole covers in terms of which ones were most likely to have issues to those least likely. How accurate was it?  As noted in the book, the top 10% of those ranked most likely to have incidents ended up accounting for 44% of the manhole covers with potentially deadly incidents. In other words, Con Edison through big data analytics was able to better "audit" the population of manhole covers for potential safety issues.  The following video goes into some detail on what the team did:

What lessons can be drawn from this use of Big Data Analytics?
Firstly, algorithms can overcome dirty data. When Professor Rudin was putting together the data to analyse, it included data from the early days of Con Edison, i.e. as in 1880s when Thomas Edison was alive! To illustrate the book notes how there 38 different ways to enter the word "service box" into service records. This is on top of the fact that some of these records were hand written and were documented by people who didn't have a concept of a computer let alone big data analytics.

Second, although the biggest factors seem obvious in hindsight, we should be aware of such conclusions. The point is that data driven decision making is more defensible than a "gut feel", which speaks directly to the professional judgement versus statistical approach of performing audit procedures. The authors further point out that there at least 104 other variables that were contenders and their relative importance cannot be known without preforming such a rigorous analysis.  The point here is that for organizations to succeed and take analytics to the next level need to embrace culturally the concept that, where feasible, organizations should invest in the necessary leg work to obtain conclusions based on solid analysis.

Third, the authors highlight the importance of "explainability". They attribute to the world of artificial intelligence, which refers to the ability of the human user to drill deeper into the analysis generated by the model and explain to operational and executive management why a specific manhole needs to be investigated. In contrast, the authors point out that models that are complex due to the inclusion of numerous variables are difficult to explain. This is a critical point for auditors. As the auditors must be able defend why a particular transaction was chosen over another for audit, big data audit analytics needs to incorporate this concept of explainability.

Finally, it is but another example of how financial audits can benefit from such techniques, given the way non-financial "audits" are using big data techniques to audit and assess information. So internal and external auditors can highlight this (along with two examples identified in the previous post) as part of their big data audit analytics business case.

Big Data: Towards a "data driven audit"

This is the first installment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ). In this installment, I explore why this is a "must read" for those interested in data driven decision making, big data and information. I will also discuss some examples included in the book that make the case for "data driven audits". 

Why read this book?

This book is written by journalist, Kenneth Cukier, (who claims in this video to have to used the term "big data" before it was commonly used) and Viktor Mayer-Schönberger (an Internet and Governance professor at University of Oxford).

Given the background of the authors, it is an easy to digest book that gives the reader a good understanding of how access to large volumes of data and the use of correlations will change the way business is done and how society has a whole functions - without going into the technical detail of how big data is "crunched" at the back end.  The authors also discuss the following:

• Why more is better: Algorithms improve by being exposed to more data - regardless of how messy it is. On the topic of size, it also comments how statistical sampling is a feature of an era when organizations could not wrap their arms around the data.
• Consumer and business implications: The book is filled with examples that anyone can relate to, such as predicting whether the price of an airplane ticket will go up or down, as well as how Google uses search queries to predict flu outbreaks.
• Enter "Datafication": It also distinguishes "datafication" versus "digitization", where the latter is making something into bits and bytes, whereas the former is something that can be analyzed by some sort of analytic engine. 
• Potentially challenges and negative consequences of big data driven decisions: One of the challenges cited by the author is the "black box" nature of algorithms: how does a common person challenge the an algorithm, when it takes a rocket scientist to understand the algorithm itself? The authors also take the risk of explaining the  danger of subordinating human decision making to algorithms. For example, they note it would be problematic for governments to round up and quarantine people just because they looked up terms related to the flu. 

There are other interesting pieces, but I will bring them up over the next few blog posts. But if you want to get a full understanding, please buy the book - it's worth it!

The case for data driven audits

The book is filled with examples that illustrate the power of big data and how they impact business and society. However, there are a couple of examples that illustrate how financial audits can benefit from such techniques, given the way non-financial "audits" are using big data techniques to audit and assess information. 

Case 1: New York City and Auditing Illegal Conversions: As discussed in this excerpt of the book, Mike Flowers applied big data techniques to the problems of "illegal conversion" in New York city. As noted in the article, illegal conversions is the "the practice of cutting up a dwelling into many smaller units so that it can house as many as 10 times the number of people it was designed for. They are major fire hazards, as well as cauldrons of crime, drugs, disease, and pest infestation. A tangle of extension cords may snake across the walls; hot plates sit perilously on top of bedspreads. People packed this tightly regularly die in blazes". The data scientists working for Flowers, took the 900,000 property lots in the city and correlated "five years of fire data ranked by severity" against the following pieces of data:

• Delinquency in paying property taxes,
• Foreclosure proceedings,
• Odd patterns in their usage of utilities, 
• Non-payment of utilities,
• Type of building,
• Date building was built,
• Ambulance visits,
• Rodent complaints,
• External brickwork.

By correlating all this and other information, they were able to improve the effectiveness of their 200 person inspection team from a "hit rate" of 13% to 70%. (Note: hit rate refers to conditions of building identified as being so bad that it warrants a "vacate order"). 

This is a pretty straightforward evidence for "data driven audits": financial auditors can identify correlations between financial data and non-financial data to determine which financial transactions need more scrutiny than others.

Not convinced?

Well, investors are already doing this. The book gives the example of how an investment firm is using traffic analysis, from Inrix, to determine the sales that a retailer will make and then buy or sell the stock of the retailer on that information. In a senses, the investment is using this as a proxy for sales. In an audit context, auditors can study the vehicular traffic around stores against the sales recorded against such stores and determine if there are issues worth investigating. 

Of course, this endeavor is not merely a matter of copying & pasting data from StatsCan and cobbling up a spreadsheet or two. It is a lot of hard work. However, this is not surprising to anyone who has been performing computer assisted audit techniques for the last decade or so. The challenge has always been in cleaning up the data and making it usable. Some of the challenges that the New York team of statisticians had, include:

• Inconsistent data formats: The team had to bring together data sets from 19 different agencies. Each agency had a different way of describing location. Consequently, this has to be standardized so that each of the 19 data sets can be correlated to the same property.
• Datafying expert intuition: The article describes how brickwork got added as an element to the correlation model. The data scientists on the team observed how the fire inspector could look at a building and know whether it was okay or not.
• Understanding significance of each variable: Each variable must be assessed in its own right to avoid the problem of generalization. For example, rodent infestation is not uniform in its significance across New York city. As noted in the article, " A rat spotted on the posh Upper East Side might generate 30 calls within an hour, but it might take a battalion of rodents before residents in the Bronx felt moved to dial 311".

Although such challenges exist, there's a promise for big data to transform the way audits are done If you recall my blog post on the Oracle vs Google trial, I made a similar point: "the profession should use this opportunity to re-examine what aspects of technology should be a part of the audit practitioners skill set - given that is clear that society's attitude towards technology has clearly changed". Applying this to data driven audits, is actually less of a stretch as auditors have the skill set of dealing with data - instead of learning to code in Java. As a profession,  we need t o understand what data is out there and how this data can be correlated with financial data to make a more effective, efficient, and insightful audit.