Big Data Auditing Revisited: Context is King

It has been a few years since I wrote up on Big Data and the Audit.  It was one of the more popular posts with over a 1,000 hits to date.

The post looks at Big Data: A Revolution That Will Transform How We Live, Work, and Think by Kenneth Cukier and Viktor Mayer-Schönberger. I enjoyed the book as it really broke down the business impact of big data without getting in technical details of the underlying technology.

Why take a second look at big data auditing?

Big data and the accompanying analytical models are key a precursor to artificial intelligence. Machine learning algorithms that power the AI bots requires the users to analyse the problem and teach the underlying algorithm.

Part 1: Context is King

To make things a bit more digestible, I thought it would be good to divide the post into two parts. The first post is more palatable as I want to explore the second use case in a bit more detail and its relevance to today.  The second post will be a bit more controversial as I will take a look at the difficulty of applying fraud or cancer-fighting algorithms in the realm of (external) financial audit.

But let's look at the first issue: how can big data analytics give us better context? 

In the original post, I spoke discussed the use case used in Cukier and Mayer-Schönberger's work around Inrix. The book gives the example of how an investment firm is using traffic analysis, from Inrix, to determine the sales that a retailer will make and then buy or sell the stock of the retailer on that information. In a sense, the investment is using vehicular traffic as a proxy for sales. In an audit context, auditors can develop expectations of what sales should be based on the number of vehicles going around stores. For example, if sales are going up, but the number of vehicles are going down then the auditor would need to take a closer look.

What I realized from this example is that what big data can give auditors better context around things and assess reasonability of things. That is as more sensor data and other data are available to auditors to integrate into statistical models, the more they will be able to spot anomalies. 

One of the issues with Barry Minkow's ZZZBest accounting fraud was the lack of context. For more on the fraud, check this video:

I actually studied this case in my auditing class at the University of Waterloo. One of the lessons we were take away from this case was that the auditors didn't know how much a site restoration would cost on average (see the first bullet in this text on page 129). But how would an auditor be able to access such data? Even with the advent of the internet, it is not simply a matter of Googling for the information.

More recently, an accounting professor was found to have generated data fraudulently. The way he got caught was that a statistic he used didn't correspond to reality. Specifically:

"misrepresented the number of U.S.-based offices it had: not 150, as the paper maintained (and as a reader had noticed might be on the high side, triggering an inquiry from the journal)" [Emphasis added]

Again, the reader had the context to understand what was presented was unreasonable causing the study to unravel and exposing the academic fraud perpetrated by Hunton. 

What will it take to make this a reality? 

What's missing is a data aggregation tool that can connect to the private, third party, and public data feeds that an auditor can leverage for statistical analysis. Furthermore, for this to be useful to clients and the business community large are visualized depictions that enable the auditor to tell the story in a better way rather than handing over complex spreadsheets.

Of course for auditors to present such materials requires them to have deeper training in data wrangling, statistics and visualization tools and techniques. 

In the next post, we will revisit the first use case that I presented in the original post that explored how the New York City was better able to audit illegal conversions through the use of big data analytical techniques. Originally, I had thought this would be a good model to apply in the world of audit. However, I am revisiting this idea. 

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else

Cafe X and Amazon Go: Auditing a robot-operated store?

By now you've probably heard of the robot-barista - Cafe X.  If not check out this video from Wired, where David Pierce walks us not only through how the robot will make your latte, but why he thinks it better than the human alternative:



Amazing isn't it?

In a presentation I did last year on how these forces of automation could impact auditing & accounting, I noted it's easier to see how technology disrupts someone other than you.

And so it looks like baristas have met their match.

As Pierce notes in the video, the inconvenience of dealing with imperfect people is something that most people want to avoid in the rat-race we live in: who wants the barista to remake your coffee 11 times as he says? ;) 

The Wired article also notes that Cafe X is 'high-quality at a cheaper price': 

"Surprisingly delicious coffee, starting at $2.25—cheaper than you’d find at Sightglass or even Starbucks. Cafe X’s location in the corner of the Metreon may not entice you out of your daily routine."

Amazon Go: Walkthrough Technology 
Amazon has also wowed the "techthusiasts" out there with their cashier-less store concept:



In the FAQ section, Amazon summarizes how this cashier-less store works:

"Our checkout-free shopping experience is made possible by the same types of technologies used in self-driving cars: computer vision, sensor fusion, and deep learning. Our Just Walk Out Technology automatically detects when products are taken from or returned to the shelves and keeps track of them in a virtual cart. When you’re done shopping, you can just leave the store. Shortly after, we’ll charge your Amazon account and send you a receipt."

Although this has the potential to revolutionize retail, Amazon has experienced some setbacks of late. The store can allegedly only handle 20 people at a time. So there maybe some kinks to work out before this goes mainstream.

Obviously, this could have a massive impact on entry level jobs: most of us who were young a while ago relied on these McJobs for spending money and funding our college/university tuition. They also gave students some practical work experience to help land a career accounting profession ;)

But let's save this discussion for a future post.

How would you audit cashier-less stores, like Cafe X or Amazon Go?

The retail industry has been a manual intensive industry that requires cashiers, stock room personnel and the like. Such a process naturally requires policies and procedures (aka internal controls) that ensure that merchandise makes it from the shelf to the cash register and into the customers possession. And there are those anti-theft mechanisms to prevent shoplifting as well. In the industry, "shrinkage", the amount of merchandise that is stolen, robbed, damaged, etc, is estimated by the National Retail Federation to be 1.38% of sales or $45.2 billion for 2015.

Cafe X and Amazon Go offer a glimpse into how automating traditional businesses can alter these fundamental risks that impact the way we go about conducting our financial audits.

With Cafe X, shrinkage is almost eliminated as there is no humans involved in the production process. Once the kiosk is loaded up with cups, coffee, syrup, sugar, milk, etc. the system is essentially fully automated - no manual intervention by baristas or customers.

Amazon Go, on the other hand, uses a whole lot of automation that is watching and analyze every move of the customers (and employees) throughout the store. Consequently, this would not be the store to steal from! And let's not forget Amazon is experimenting with those drones and are we really sure that they are unarmed?


Given this level of automation of the actual business process and controls, could auditors stick to the tried, tested and true retail audit procedures? Or would this enable a more automated approach?

I was directly involved with the recent test-audit of the blockchain involving loyalty points. One of the realities of auditing such exponential technologies is that it makes controls testing a must. For example, for the financial auditor to rely on the digital signatures there needs to be some testing around the wallets to ensure that the signatures are reliable.

Consequently, testing such automated stores would require either a SOC2 or modified SOC report to meet the needs of such a store. For example, the SOC2 would need to have some way of having comfort of how the stock and inventory gets loaded into the store. Likely the auditor would rely on the automated process which the store uses to replenish stock, but it's that hand off between the delivery person (assuming it's still human) that would be the area there is a risk of shrinkage. For example, how does legitimately damaged inventory get accounted for at that point? Whatever process and controls Amazon/Cafe X put in place would need to be tested from a controls perspective.

For the substantive component, I think that's where things get interesting: enter the "embedded audit module". This concept has been around since at least 1989. The idea is that the auditor installs independent software onto the client's system and then transmits it back to the auditor, who uses it as a basis for conducting the necessary audit procedures and tests. The core idea is that the auditor has full control over such a system and the client cannot tamper with the code.

What would be relatively straightforward would be the data capture-component: sales data, stock data, spoilage, etc. would be uploaded from the automated store right into the auditor's system. But this then requires the additional step of verifying the data to independent source documents (e.g. invoices, purchase orders, etc.). In other words, the audit procedure would still require manual intervention as the auditee would need to send this information back to the auditor to complete their audit.

Where I think the audit innovation would be is exploring how video footage can act as a substitute for physical/direct observation by the auditor. That is, could the auditor install a video camera in the automated store as a part of the EAM that would then act as actual independent audit evidence of the actual sale or purchase? For example, in the Cafe X example the auditor could actually use the footage and the visual software to count the cups sold that day and reconcile that to the sales data transmitted back from the EAM for the day?

Although one can argue such transactions are not material and therefore such procedures are overkill.

However, I think now is the right time to conduct experiments and test audits to see whether we can reinvent the classic audit to meet the technology of today. In a future post, we will explore what this means broadly for jobs and more specifically how this could impact the profession.

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else

Low Decision Agility: BigData's Insurmountable Challenge?

Working in the field of data analytics for over decade there is one recurring theme that never seems to go away: the overall struggle organizations have with getting their data in order.

Courtesy of this link. 

Although this is normally framed in terms of data quality and data management, it's important to link this back to the ultimate raison d'etre for data and information: organizational decision making. Ultimately, an organization has significant data and information management challenges it culminates into a lack of "decision agility" for executive or operational management. I define decision agility as follows:

"Decision agility is the ability of an entity to provide relevant information 

to a decision maker in a timely manner." 

Prior to getting into the field, you would think that with all the hype of the Information Age it would be easy as pressing a button for a company to get you the data that you need to perform the analysis you need to do. However, after getting into the field, you soon realize how wrong this thinking: most organizations have low-decision agility.

I would think it is fair to say that this problem hits those involved in external (financial) audits the hardest. As we have tight budgets, low-decision agility at the clients we audit makes it cost-prohibitive to perform what is now known as audit analytics (previously known as CAATs). Our work is often reigned in by the (non-IT) auditors running the audit engagement because it is "cheaper" do the same test manually rather than parse our way through the client's data challenges

So what does this have to do with Big Data Analytics?

As I noted in my last post, there is the issue of veracity - the final V in the 4 Vs definition of Big Data. However, veracity is part of the larger problem of low decision agility that you can find at organizations. Low-decision agility emerges from a number of factors and can have implications on a big data analytics initiative at an organization. These factors and implications include:

• Wrong data:  Fortune, in this article, notes there is the obvious issue of "obsolete, inaccurate, and missing information" data records itself. Consequently, the big data analytics initiative needs to assess the veracity of the underlying data to understand how much work needs to be done to clean up the data before meaningful insights can be drawn from the data. 
• Disconnect between business and IT: The business has one view of the data and the IT folks see the data in a different way. So when you try to run a "simple" test it takes a significant amount of time to reconcile business's view of the data model to IT's view of the data model. To account for this problem there needs to be some effort in determining how to sync the user's view of the data and IT's view of the data on an ongoing basis to enable the big data analytic to rely on the data that sync's up with the ultimate decision maker's view of the world.  
• Spreadsheet mania: Let's fact it: organizations treat IT as an expense not as an investment. Consequently, organizations will rely on spreadsheets to do some of the heavy lifting for the information processing because it is the path of least resistance. The overuse of spreadsheets can be a sign of an IT system that fails to meets the needs of the users. However, regardless of why they are used, the underlying problem is dealing with these vast array of business-managed applications that are often fraught with errors and outside the controls of production system. The control and related data issues become obvious during compliance efforts, such as SOX 404 or major transitions to new financial/data standards, such as the move to IFRS. When developing big data analytics, how do you account for the information trapped in these myriad little apps outside of IT's purview? 
• Silo thinking: I remember the frustration of dealing with companies that lacked a centralized function that had a holistic view of the data. Each department would know it's portion of the processing rules, etc. but would have no idea of what happened upstream or downstream. Consequently, an organization needs to create a data governance structure that understands the big picture and can identify and address the potential gaps in the data set before it is fed into the Hadoop cluster.  
• Heterogenous systems: Organizations with a patch-work of systems require extra effort from getting the data formatted and synchronized. InfoSec specialists deal with this issue of normalization when it come to security log analysis: the security logs that are extracted from different systems need to have the event IDs, codes, etc. "translated" into a common language to enable a proper analysis of events that are occurring across the enterprise. The point is that big data analytics must also perform a similar "translation" to enable analysis of data pulled from different systems. Josh Sullivan of Booz Allen states: "...training your models can take weeks and weeks" to recognize what content fed into the system are actually the same value. For example, it will take a while for the system to learn that female and woman are the same thing when looking at gender data. 
• Legacy systems:  Organizations may have legacy systems which do not retain data, are hard to extract from and difficult to import into other tools. Consequently, this can cost time and money to get the data into a usable format that will also need to be factored into the big data analytics initiative.
• Business rules and semantics: Beyond the heterogenity differences between systems there can also be a challenge in how something is commonly defined. A simple example is currency: an ERP that expand multiple countries the amount reported may be in the local currency or the dollar, but requires the metadata to give that meaning. Another issue can be that different user group define something different. For example, for a sale for the sales/marketing folks may not mean the same thing as a sale for the finance/accounting group (e.g. the sales & marketing people may not account for doubtful accounts or incentives that need to be factored in for accounting purposes). 

Of course these are not an exhaustive list of issues, but it gives an idea of how the reality of analytics is obscured the tough reality of state of data.  

In terms of the current state of data quality, a recent blog post by Michele Goetz of Forrester noted that 70% of the executive level business professionals they interviewed spent more than 40% of their time vetting and validating data. (Forrester notes the following caveat about the data: "The number is too low to be quantitative, but it does give directional insight.")

Until organizations get to a state of high decision agility - where business users spend virtually no time vetting/validating the data - organizations may not be able to reap the full benefits of a big data analytics initiative. 

Worth mentioning: KPMG's take on the state of tech in the audit profession

In a recent post (as in just this week) on Forbes, KPMG's  James P. Liddy who is the Vice Chair, Audit and Regional Head of Audit, Americas put out a great post that summarizes the current state of analytics in financial audits.

He diplomatically summarizes the current state of the financial audit as "unchanged for more than 80 years since the advent of the classic audit" while stating "[a]dvances in technology and the massive proliferation of available information have created a new landscape for financial reporting. With investors now having access to a seemingly unlimited breadth and depth of information, the need has never been greater for the audit process to evolve by providing deeper and more relevant insights about an organization’s financial condition and performance –while maintaining and continually improving audit quality." [Emphasis added]

For those that have started off our careers in the world of financial audit as professional accountants and then moved to the world of audit analytics or IT risk management, we have always felt that technology could help us to get audits done more efficiently and effectively.

I was actually surprised that he stated that auditors "perform procedures over a relatively small sample of transactions – as few as 30 or 40 – and extrapolate conclusions across a much broader set of data". We usually don't see this kind of openness when it comes to discussing the inner-workings of the profession. However, I think that discussing such fundamentals is inevitable given those outside the profession are embracing big data analytics in "non-financial audits". For example, see this post where I discuss the New York City fire department's use of big data analytics to identify a better audit population when it comes to identifying illegal conversions that are a high risk and need to be evacuated.

For those that take comfort in the regulated nature of the profession as protection of disruption, we should take note of how the regulators are embracing big data analytics. Firstly, the SEC is using RoboCop to better target financial irregularities. Secondly, according to the Wall Street Journal, FINRA is eyeing an automated audit approach to monitoring to risk. The program is known as "Comprehensive Automated Risk Data System" (CARDS). As per FINRA:

"CARDS program will increase FINRA's ability to protect the investing public by utilizing automated analytics on brokerage data to identify problematic sales practice activity. FINRA plans to analyze CARDS data before examining firms on site, thereby identifying risks earlier and shifting work away from the on-site exam process". In the same post, Susan Axelrod, FINRA's Executive Vice President of Regulatory Operations, is quoted as saying "The information collected through CARDS will allow FINRA to run analytics that identify potential "red flags" of sales practice misconduct and help us identify potential business conduct problems with firms, branches and registered representatives".

As a result, I agree with Mr. Libby: sticking to the status quo is no longer a viable strategy for the profession.

Auditing the Algorithm: Is it time for AlgoTrust?

This is the third instalment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ).  In the last two posts we explored the more tactical examples of how big data can assist auditors in executing audits resulting in a more efficient and effective audit. The book, however, also examines the societal implications of big data. In this instalment, we look explore the role of the algorithmist.

Why do we need to audit the "secret sauce"?
When it comes to big data analytics, the decisions and conclusions the analyst will make hinges greatly on the underlying actual algorithm.  Consequently, as big data analytics become more and more part of the drivers of actions in companies and societal institutions (e.g. schools, government, non-profit organizations, etc.), the more dependent society becomes on the "secret sauce" that powers these analytics. The term "secret sauce" is quite apt because it highlights the underlying technical opaqueness that is commonplace with such things: the common person likely will not be able to understand how the big data analytic arrived at a specific conclusion. We discussed this in our previous post as the challenge of explainability, but the nuance here is that is how do you explain algorithms to external parties, such as customers, suppliers, and others.

To be sure this is not the only book  that points to the importance of the role of algorithms in society. Another example is "Automate This: How Algorithms Came to Rule Our World" by Chris Steiner, which (as you can see by the title) explains how algorithms are currently dominating our society. The book bring ups common examples the "flash crash" and the role that "algos" are playing on Wall Street in the banking sector as well as how NASA used these alogrithms to assess personality types for its flight missions. It also goes into the arts. For example, it discusses how there's an algorithm that can predict the next hit song and hit screenplay as well as how algorithms can generate classical music that impresses aficionados - until they find out it is an algorithm that generated it! The author, Chris Steiner, discusses this trend in the follow TedX talk:



So what Mayer-Schönberger and Cukier suggest is the need for a new profession which they term as "algorithmists". According to them:

"These new professionals would be experts in the areas of computer science, mathematics, and statistics; they would act as reviewers of big-data analyses and predictions. Algorithmists would take a vow of impartiality and confidentiality, much as accountants and certain other professionals do now. They would evaluate the selection of data sources, the choice of analytical and predictive tools, including algorithms and models, and the interpretation of results. In the event of a dispute, they would have access to the algorithms, statistical approaches, and datasets that produced a given decision."

The also extrapolate this thinking to an "external algorithmist": who would "act as impartial auditors to review the accuracy or validity of big-data predictions whenever the government required it, such as under court order or regulation. They also can take on big-data companies as clients, performing audits for firms that wanted expert support. And they may certify the soundness of big-data applications like anti-fraud techniques or stock-trading systems. Finally, external algorithmists are prepared to consult with government agencies on how best to use big data in the public sector.

As in medicine, law, and other occupations, we envision that this new profession regulates itself with a code of conduct. The algorithmists’ impartiality, confidentiality, competence, and professionalism is enforced by tough liability rules; if they failed to adhere to these standards, they’d be open to lawsuits. They can also be called on to serve as expert witnesses in trials, or to act as “court masters”, which are experts appointed by judges to assist them in technical matters on particularly complex cases.

Moreover, people who believe they’ve been harmed by big-data predictions—a patient rejected for surgery, an inmate denied parole, a loan applicant denied a mortgage—can look to algorithmists much as they already look to lawyers for help in understanding and appealing those decisions."

They also envision such professionals would work also work internally within companies, much the way internal auditors do today.

WebTrust for Certification Authorities: A model for AlgoTrust?
The authors bring up a good point: how would you go about auditing an algo? Although auditors lack the technical skills of algoritmists, it doesn't prevent them from auditing algorithms. The WebTrust for Certification Authorities (WebTrust for CAs) could be a model where assurance practitioners develop a standard in conjunction with algorithmists and enable audits to be performed against the standard. Why is WebTrust for CAs a model? WebTrust for CAs is a technical standard where an audit firm would "assess the adequacy and effectiveness of the controls employed by Certification Authorities (CAs)". That is, although the cryptographic key generation process is something that goes beyond the technical discipline of a regular CPA, it did not prevent the assurance firms from issuing an opinion.

So is it time for CPA Canada and the AICPA to put together a draft of "AlgoTrust"?

Maybe.

Although the commercial viability for such a service would be hard to predict, it would help at least start the discussion around of how society can achieve the outcomes Mayer-Schönberger and Cukier describe above. Furthermore, some of the ground work for such a service is already established. Fundamentally, an algorithm takes data inputs, processes it and then delivers a certain output or decision. Therefore, one aspect of such a service is to understand whether the algo has "processing integrity" (i.e. as the authors put it, to attest to the "accuracy or validity of big-data predictions"), which is something the profession established a while back through its SysTrust offering. To be sure this framework would have to be adapted. For example, algos are used to make decisions so there needs to be some thinking around how we would identify materiality in terms of  total number of "wrong" decisions as well as defining "wrong" in an objective and is auditable manner.

AlgoTrust, as a concept, illustrates not only a new area where auditors can move its assurance skill set into an emerging area but also how the profession can add thought leadership around the issue of dealing with opaqueness of algorithms - just as it did with financial statements nearly a century ago.

Big Data Audit Analytics: Dirty data, explainability and data driven decision making

This is the second instalment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ). In this instalment, I explore another example of Big Data Audit analytics noted in the book and highlight the lessons learned from it. 

Con Edison and Exploding Manhole Covers

The book discussed the case of Con Edison (the public utility that provides electricity to New York City)  and its efforts to better predict, which of their manhole covers will experience "technical difficulties" from the relatively benign (e.g. smoking, heating up, etc) to the potentially deadly (where a 300 pound manhole can explode into the air and potentially harm someone). Given the potentially implications on life and limb, Con Edison needed a better audit approach, if you will, then random guessing as to which manhole cover would need maintenance to prevent such problems from occurring.

And this is where Cynthia Rudin, currently an associate professor of statistics at MIT, comes into the picture. She and her team of statisticians at Columbia University worked with Con Edison to devise a model that would predict, where the maintenance dollars should be focused.

The team developed a model with 106 (with the biggest factors being age of the manhole covers and if there were previous incidents) data predictors that ranked manhole covers in terms of which ones were most likely to have issues to those least likely. How accurate was it?  As noted in the book, the top 10% of those ranked most likely to have incidents ended up accounting for 44% of the manhole covers with potentially deadly incidents. In other words, Con Edison through big data analytics was able to better "audit" the population of manhole covers for potential safety issues.  The following video goes into some detail on what the team did:

What lessons can be drawn from this use of Big Data Analytics?
Firstly, algorithms can overcome dirty data. When Professor Rudin was putting together the data to analyse, it included data from the early days of Con Edison, i.e. as in 1880s when Thomas Edison was alive! To illustrate the book notes how there 38 different ways to enter the word "service box" into service records. This is on top of the fact that some of these records were hand written and were documented by people who didn't have a concept of a computer let alone big data analytics.

Second, although the biggest factors seem obvious in hindsight, we should be aware of such conclusions. The point is that data driven decision making is more defensible than a "gut feel", which speaks directly to the professional judgement versus statistical approach of performing audit procedures. The authors further point out that there at least 104 other variables that were contenders and their relative importance cannot be known without preforming such a rigorous analysis.  The point here is that for organizations to succeed and take analytics to the next level need to embrace culturally the concept that, where feasible, organizations should invest in the necessary leg work to obtain conclusions based on solid analysis.

Third, the authors highlight the importance of "explainability". They attribute to the world of artificial intelligence, which refers to the ability of the human user to drill deeper into the analysis generated by the model and explain to operational and executive management why a specific manhole needs to be investigated. In contrast, the authors point out that models that are complex due to the inclusion of numerous variables are difficult to explain. This is a critical point for auditors. As the auditors must be able defend why a particular transaction was chosen over another for audit, big data audit analytics needs to incorporate this concept of explainability.

Finally, it is but another example of how financial audits can benefit from such techniques, given the way non-financial "audits" are using big data techniques to audit and assess information. So internal and external auditors can highlight this (along with two examples identified in the previous post) as part of their big data audit analytics business case.

Big Data: Towards a "data driven audit"

This is the first installment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ). In this installment, I explore why this is a "must read" for those interested in data driven decision making, big data and information. I will also discuss some examples included in the book that make the case for "data driven audits". 

Why read this book?

This book is written by journalist, Kenneth Cukier, (who claims in this video to have to used the term "big data" before it was commonly used) and Viktor Mayer-Schönberger (an Internet and Governance professor at University of Oxford).

Given the background of the authors, it is an easy to digest book that gives the reader a good understanding of how access to large volumes of data and the use of correlations will change the way business is done and how society has a whole functions - without going into the technical detail of how big data is "crunched" at the back end.  The authors also discuss the following:

• Why more is better: Algorithms improve by being exposed to more data - regardless of how messy it is. On the topic of size, it also comments how statistical sampling is a feature of an era when organizations could not wrap their arms around the data.
• Consumer and business implications: The book is filled with examples that anyone can relate to, such as predicting whether the price of an airplane ticket will go up or down, as well as how Google uses search queries to predict flu outbreaks.
• Enter "Datafication": It also distinguishes "datafication" versus "digitization", where the latter is making something into bits and bytes, whereas the former is something that can be analyzed by some sort of analytic engine. 
• Potentially challenges and negative consequences of big data driven decisions: One of the challenges cited by the author is the "black box" nature of algorithms: how does a common person challenge the an algorithm, when it takes a rocket scientist to understand the algorithm itself? The authors also take the risk of explaining the  danger of subordinating human decision making to algorithms. For example, they note it would be problematic for governments to round up and quarantine people just because they looked up terms related to the flu. 

There are other interesting pieces, but I will bring them up over the next few blog posts. But if you want to get a full understanding, please buy the book - it's worth it!

The case for data driven audits

The book is filled with examples that illustrate the power of big data and how they impact business and society. However, there are a couple of examples that illustrate how financial audits can benefit from such techniques, given the way non-financial "audits" are using big data techniques to audit and assess information. 

Case 1: New York City and Auditing Illegal Conversions: As discussed in this excerpt of the book, Mike Flowers applied big data techniques to the problems of "illegal conversion" in New York city. As noted in the article, illegal conversions is the "the practice of cutting up a dwelling into many smaller units so that it can house as many as 10 times the number of people it was designed for. They are major fire hazards, as well as cauldrons of crime, drugs, disease, and pest infestation. A tangle of extension cords may snake across the walls; hot plates sit perilously on top of bedspreads. People packed this tightly regularly die in blazes". The data scientists working for Flowers, took the 900,000 property lots in the city and correlated "five years of fire data ranked by severity" against the following pieces of data:

• Delinquency in paying property taxes,
• Foreclosure proceedings,
• Odd patterns in their usage of utilities, 
• Non-payment of utilities,
• Type of building,
• Date building was built,
• Ambulance visits,
• Rodent complaints,
• External brickwork.

By correlating all this and other information, they were able to improve the effectiveness of their 200 person inspection team from a "hit rate" of 13% to 70%. (Note: hit rate refers to conditions of building identified as being so bad that it warrants a "vacate order"). 

This is a pretty straightforward evidence for "data driven audits": financial auditors can identify correlations between financial data and non-financial data to determine which financial transactions need more scrutiny than others.

Not convinced?

Well, investors are already doing this. The book gives the example of how an investment firm is using traffic analysis, from Inrix, to determine the sales that a retailer will make and then buy or sell the stock of the retailer on that information. In a senses, the investment is using this as a proxy for sales. In an audit context, auditors can study the vehicular traffic around stores against the sales recorded against such stores and determine if there are issues worth investigating. 

Of course, this endeavor is not merely a matter of copying & pasting data from StatsCan and cobbling up a spreadsheet or two. It is a lot of hard work. However, this is not surprising to anyone who has been performing computer assisted audit techniques for the last decade or so. The challenge has always been in cleaning up the data and making it usable. Some of the challenges that the New York team of statisticians had, include:

• Inconsistent data formats: The team had to bring together data sets from 19 different agencies. Each agency had a different way of describing location. Consequently, this has to be standardized so that each of the 19 data sets can be correlated to the same property.
• Datafying expert intuition: The article describes how brickwork got added as an element to the correlation model. The data scientists on the team observed how the fire inspector could look at a building and know whether it was okay or not.
• Understanding significance of each variable: Each variable must be assessed in its own right to avoid the problem of generalization. For example, rodent infestation is not uniform in its significance across New York city. As noted in the article, " A rat spotted on the posh Upper East Side might generate 30 calls within an hour, but it might take a battalion of rodents before residents in the Bronx felt moved to dial 311".

Although such challenges exist, there's a promise for big data to transform the way audits are done If you recall my blog post on the Oracle vs Google trial, I made a similar point: "the profession should use this opportunity to re-examine what aspects of technology should be a part of the audit practitioners skill set - given that is clear that society's attitude towards technology has clearly changed". Applying this to data driven audits, is actually less of a stretch as auditors have the skill set of dealing with data - instead of learning to code in Java. As a profession,  we need t o understand what data is out there and how this data can be correlated with financial data to make a more effective, efficient, and insightful audit.  

The Power of Visualized Analytics

In my new role at Deloitte, I have recently come across tools, such as Tableau or Qlikview, that allow users to "visualize data". To be honest I didn't think they would add much value compared to "rule-based analytic tools", such as IDEA and ACL. However, after using these tools I realized the real power of being able to visualize data in contrast to producing an exception. It brings the dashboard concept within the executive management suite to the analyst or other business professional. But as they say "seeing is believing".

So let's try an experiment.

I recently came across an amazing visualization that really illustrates that power of visualization that visualizes economic data, specifically the distribution of wealth.

But don't click on it yet!

To get the most of the experiment first read this report (which the visualization is based on) to see how the stats hit you in terms of impact.

So here is an excerpt from the Oxfam report which the visualization is based on. (The numbers at the end of the sentence are footnotes; see the original report for the sources)

And

So now let's see how this data (plus other sources) hits you when it is visualized:


Is there really any contest?

What I've realized is that the visualization really enables the business user to bring together multiple dimensions into a single sheet of paper and enables you to tell the story about the underlying data. Having said that, I do believe that there is a complementary relationship between visualized analytics and rule-based analytics. For example, if you want to quantify the difference between budgets-and-actuals, produce a list of exceptions, etc, then rule based analytics are better for such a purpose. Furthermore, visualizations can help explain the results of rule-based analytic procedures.

Technology and Audit: Rising tide of tech floats all boats

Norman Marks, evangelist at SAP, neatly summarizes  in this 5 minute video the implications of the how the recent technologies, such as cloud, analytics, and the like has implications on the auditing profession.

As he notes, "if it's good enough for our clients, it's good enough for us" (i.e. us being the auditing profession).  He mentions how individual analysts and other business professionals are using tablets and other devices to perform analytics. He also cautions that we should not make the same mistake as we did when analyzing the potential for desktop computing. In the video he narrates an amusing anecdote about the reaction of the accounting firm that he worked at to the nascent, desktop computer in the 70s.

Although I agree with his comments, I would say that this also extends beyond the corporate IT environment I have written in the past about cloud and mobile tech and what I see is that these technologies favour the small and medium sized business (SMBs) over the large ones. Basically, SMBs can now afford enterprise class technology and are probably using this technology within their personal spheres. Hence the term "consumerization of IT": advances in technology are focused in the consumer space not the corporate IT Department. As illustrated by the use of the iPhone within the corporate IT, consumers brought or demanded that IT let them use the iPhone instead of the standard  issue (e.g. BlackBerry) smartphone. Furthermore, widespread familiarity with these technologies allows SMBs access to employees who know how to use these technologies - without specialized training. The sum of it: it is much harder for auditors to justify being low-tech, when even the employee of the SMB has gone high tech.

Big Data: Some resources

For the CAs and CISAs, looking for the coles notes version of what's going in the world of big data, check out the following podcast by David Linthicum and company; some of the most knowledgeable people on Cloud computing. Chris Daly (who works with Dave) provides a good nine item list based on this article. Chris did us all a big favour by breaking down the slideshow into a nice list of nine points. I will let you click on the link to see what they are, but I thought it was interesting to comment on the first two:

• "Define the business drivers". It's pretty amazing how this single premise is one of the most critical concepts on business-technology, that requires constant attention! Ironically, I just finished answering a question to a fellow accounting profession who is taking a course on IT controls to emphasize this point. What I explained to him was that the fundamental concept here is that technology changes are driven by business. In other words, IT Strategy or investments must be driven by the overall value drivers of the business.As for why you would not make changes to the system because of technological improvements is because those technologies may have no actual "Return on Investment" (ROI) for the business. In other words, companies should not adopt technology for the sake of technology. 
• "Discover the data and it’s location". Wow! For those of IT-auditors that run computer assisted audit techniques (CAATS) can really appreciate how these six words can represent a mountain of work! When I teach the computer-assisted audit techniques course at University of Waterloo, I always make a point of warning my students of the practical limits of running CAATs: getting the data can be the hardest aspect of the whole process. For those of you not familiar what CAATs are, they are basically  automated tests that auditors will run using "generalize audit software" on data that is used to support items on the financial audit. You can also use these technique to identify security issues or fraud; see IDEA's Caseware. (Full disclosure: Caseware is a both a sponsor of the Center that supports this blog as well as the course I teach at UW). For example, these tools help perform full analysis of a set of data e.g. identify all the negative amounts in an inventory file or link files to together using a unique identifier to compare the data from one file (e.g. credit limits) to the data in another file (e.g. total amounts owing by the customer. Also, check out the wikipedia entry it's pretty good. 

For other sources, check out the massive (and free) report from McKinsey on Big Data, which they have even made available to run on your Kindle App on your Android or iOS device. Also, check out this CAMagazine article on the topic. This HBR blog post has provides a look at the overall issues, including privacy problems. 

If you have any other resources, especially from an IT Auditor (i.e. security, data integrity, etc), perspective, please do share.