Will "data lakes" cure companies info management problems?

Anyone working in the analytics space will know that one of the primary obstacles to performing analytics is the state of the underlying data. One may, naively, expect that organizations would see systems as a strategic asset that requires top attention. The reality is that the state of systems is much to be desired. This phenomenon was actually highlighted in Michael Lewis's latest book: Flash Boys. He notes in this interview, that his investigation into High Frequency Trading (HFT) began with the arrest of Sergey Aleynikov:


Aleynikov was arrested for stealing code from Goldman Sachs related to its HFT platform and it was assumed that he would use this code at his new employers trading department. However, as Lewis explains, the new firm used a totally different code base and the information he took was the equivalent of the notes one keeps in their notebook (Lewis explains that this was the judgement of Aleynikov's programming peers).

So if this is relatively useless information, why was Aleynikov kept in jail for a year?

The theory is that Goldman was afraid that the inadequacy of the HFT platform would be revealed to the world. Yes. Goldman. A bank deemed "too big too fail" - thereby having access to unlimited government funding and bailouts - can't spare the capital to invest in a state of the art trading platform. This was one of the reasons that seems to explain why Goldman ultimately backed the IEX platform that Brad Katsyuma set up to essentially eliminate the advantage that HFT firms have over everyone else.

Give that this is the reality of the technology, it should be no surprise that the promise of data analytics often gets killed due to the inability to get to the data. And this is where data lakes come into the picture.

As noted in this Forbes article, "data lakes" differ from "data warehouses" in that there is no upfront cleansing, sorting and categorization of the data into a specific structure. Instead, the data is stored in "in a massive, easily accessible repository based on the cheap storage that’s available today. Then, when there are questions that need answers, that is the time to organize and sift through the chunks of data that will provide those answers."

Will this then solve the world's data access and cleansing problems? Gartner does not think so.

In an analyst report released last week, Gartner effectively slammed this concept pretty hard. According to Gartner, they noted the following issues with the concept of data lakes:

• No consensus around  what "data lakes" means. As with any new techno-buzzword, each vendor will define data lakes to mean something different. With cloud computing, NIST quickly came on to the scene to define the concept and most commentators at least referred to this definition as a starting point before proposing their own. 
• Skills gap exist preventing the common user from leveraging "data lakes": According to Gartner, data lakes "assumes that all are highly skilled at data manipulation and analysis, as data lakes lack semantic consistency and governed metadata". In other words, companies who save time/effort on the upfront preprocessing of the data, need to equip their users with the necessary skills to leverage the technology. 
• Data lakes remove context, security and other meta data. Without context, users likely cannot use the data carefully. Imagine the difficulty in assessing the dimensions of inventory when some of the subsidiaries use the metric system, where others do not. 

It appears that data lakes in their current iteration are a work in progress when it comes to solving the data accessibility and quality issues that analytics expert face. However, it appears with more robust tools on the front-end that would address the issues, can make data lakes a potential tool to access the organization's disparate data. 

Privacy to be cast aside to make Big Data a reality?

This is the fourth and final instalment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ).  In the last two posts we explored the more tactical examples of how big data can assist auditors in executing audits resulting in a more efficient and effective audit. The book also examines the societal implications of big data. In this instalment, we look explore the privacy implications of big data.

What's are the privacy implications of Big Data?
In the past 3 instalments, we've explored the opportunities that big data affords to audit profession and society at large. In this article we look at the privacy implications raised by the book.

When we think of a totalitarian state we flash back to the regimes of world war II or the Soviet era. The book talks about how the East German Communist State invested vast amounts of resources on gathering data from its citizens in order to see who conformed with the state's ideology and who didn't. The book notes that East German secret police (the Ministerium für Staatssicherheit or "stasi") accumulated (amongst other things) 70 miles of documents. However, now big data analytics essentially enables corporations and governments to mine the digital exhaust people leave through social media, using their cell phones or logging into their email accounts and essentially eliminate the privacy people have.

Some may point to anonymization as a potential solution to the problem. However, the authors highlight how New York Times reporters were able to comb through anonymized data published by AOL to positively establish the identity of the users. This highlights that the powerful tools that have emerged from big data alter the privacy landscape. Consequently, privacy controls need to be rethought from this perspective.

The authors, however, raise a much more interesting point when discussing privacy in the era of big data. They highlight the conflict between privacy and profiting from big data. They note how the value of big data emerges from the secondary uses of big data. However, privacy policies require the user to consent to a specific uses of data at the time they sign up for the service. This means future big data analytics are essentially limited by what uses the user agreed upon sign-up. However, corporations in their drive to maximize profits will ultimately make privacy policies so loose (i.e. to cover secondary uses) that the user essentially has to give up all their privacy in order to use the service. What the authors propose is an accountability framework. Similar to how stock issuing companies are accountable to the security regulators, the idea is that organizations would be accountable to a privacy body of sorts that reviews the use of the big data and ensures that companies are accountable for the negative consequences of the data.

For those of use that have been involved in privacy compliance, such an approach would make it real for companies to deal with the privacy issues in proactive manner. We saw how companies attitudes towards controls over financial reporting shifted from mild interest (or indifference) to active concern with the passage of Sarbanes-Oxley. In contrast, no similar fervour could be found the business landscape when addressing privacy issues. Although the solution is not obvious, the reality is that companies will make their privacy notices meaningless in order to reap the ROI from investments made in big data.

Auditing the Algorithm: Is it time for AlgoTrust?

This is the third instalment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ).  In the last two posts we explored the more tactical examples of how big data can assist auditors in executing audits resulting in a more efficient and effective audit. The book, however, also examines the societal implications of big data. In this instalment, we look explore the role of the algorithmist.

Why do we need to audit the "secret sauce"?
When it comes to big data analytics, the decisions and conclusions the analyst will make hinges greatly on the underlying actual algorithm.  Consequently, as big data analytics become more and more part of the drivers of actions in companies and societal institutions (e.g. schools, government, non-profit organizations, etc.), the more dependent society becomes on the "secret sauce" that powers these analytics. The term "secret sauce" is quite apt because it highlights the underlying technical opaqueness that is commonplace with such things: the common person likely will not be able to understand how the big data analytic arrived at a specific conclusion. We discussed this in our previous post as the challenge of explainability, but the nuance here is that is how do you explain algorithms to external parties, such as customers, suppliers, and others.

To be sure this is not the only book  that points to the importance of the role of algorithms in society. Another example is "Automate This: How Algorithms Came to Rule Our World" by Chris Steiner, which (as you can see by the title) explains how algorithms are currently dominating our society. The book bring ups common examples the "flash crash" and the role that "algos" are playing on Wall Street in the banking sector as well as how NASA used these alogrithms to assess personality types for its flight missions. It also goes into the arts. For example, it discusses how there's an algorithm that can predict the next hit song and hit screenplay as well as how algorithms can generate classical music that impresses aficionados - until they find out it is an algorithm that generated it! The author, Chris Steiner, discusses this trend in the follow TedX talk:



So what Mayer-Schönberger and Cukier suggest is the need for a new profession which they term as "algorithmists". According to them:

"These new professionals would be experts in the areas of computer science, mathematics, and statistics; they would act as reviewers of big-data analyses and predictions. Algorithmists would take a vow of impartiality and confidentiality, much as accountants and certain other professionals do now. They would evaluate the selection of data sources, the choice of analytical and predictive tools, including algorithms and models, and the interpretation of results. In the event of a dispute, they would have access to the algorithms, statistical approaches, and datasets that produced a given decision."

The also extrapolate this thinking to an "external algorithmist": who would "act as impartial auditors to review the accuracy or validity of big-data predictions whenever the government required it, such as under court order or regulation. They also can take on big-data companies as clients, performing audits for firms that wanted expert support. And they may certify the soundness of big-data applications like anti-fraud techniques or stock-trading systems. Finally, external algorithmists are prepared to consult with government agencies on how best to use big data in the public sector.

As in medicine, law, and other occupations, we envision that this new profession regulates itself with a code of conduct. The algorithmists’ impartiality, confidentiality, competence, and professionalism is enforced by tough liability rules; if they failed to adhere to these standards, they’d be open to lawsuits. They can also be called on to serve as expert witnesses in trials, or to act as “court masters”, which are experts appointed by judges to assist them in technical matters on particularly complex cases.

Moreover, people who believe they’ve been harmed by big-data predictions—a patient rejected for surgery, an inmate denied parole, a loan applicant denied a mortgage—can look to algorithmists much as they already look to lawyers for help in understanding and appealing those decisions."

They also envision such professionals would work also work internally within companies, much the way internal auditors do today.

WebTrust for Certification Authorities: A model for AlgoTrust?
The authors bring up a good point: how would you go about auditing an algo? Although auditors lack the technical skills of algoritmists, it doesn't prevent them from auditing algorithms. The WebTrust for Certification Authorities (WebTrust for CAs) could be a model where assurance practitioners develop a standard in conjunction with algorithmists and enable audits to be performed against the standard. Why is WebTrust for CAs a model? WebTrust for CAs is a technical standard where an audit firm would "assess the adequacy and effectiveness of the controls employed by Certification Authorities (CAs)". That is, although the cryptographic key generation process is something that goes beyond the technical discipline of a regular CPA, it did not prevent the assurance firms from issuing an opinion.

So is it time for CPA Canada and the AICPA to put together a draft of "AlgoTrust"?

Maybe.

Although the commercial viability for such a service would be hard to predict, it would help at least start the discussion around of how society can achieve the outcomes Mayer-Schönberger and Cukier describe above. Furthermore, some of the ground work for such a service is already established. Fundamentally, an algorithm takes data inputs, processes it and then delivers a certain output or decision. Therefore, one aspect of such a service is to understand whether the algo has "processing integrity" (i.e. as the authors put it, to attest to the "accuracy or validity of big-data predictions"), which is something the profession established a while back through its SysTrust offering. To be sure this framework would have to be adapted. For example, algos are used to make decisions so there needs to be some thinking around how we would identify materiality in terms of  total number of "wrong" decisions as well as defining "wrong" in an objective and is auditable manner.

AlgoTrust, as a concept, illustrates not only a new area where auditors can move its assurance skill set into an emerging area but also how the profession can add thought leadership around the issue of dealing with opaqueness of algorithms - just as it did with financial statements nearly a century ago.

Big Data Audit Analytics: Dirty data, explainability and data driven decision making

This is the second instalment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ). In this instalment, I explore another example of Big Data Audit analytics noted in the book and highlight the lessons learned from it. 

Con Edison and Exploding Manhole Covers

The book discussed the case of Con Edison (the public utility that provides electricity to New York City)  and its efforts to better predict, which of their manhole covers will experience "technical difficulties" from the relatively benign (e.g. smoking, heating up, etc) to the potentially deadly (where a 300 pound manhole can explode into the air and potentially harm someone). Given the potentially implications on life and limb, Con Edison needed a better audit approach, if you will, then random guessing as to which manhole cover would need maintenance to prevent such problems from occurring.

And this is where Cynthia Rudin, currently an associate professor of statistics at MIT, comes into the picture. She and her team of statisticians at Columbia University worked with Con Edison to devise a model that would predict, where the maintenance dollars should be focused.

The team developed a model with 106 (with the biggest factors being age of the manhole covers and if there were previous incidents) data predictors that ranked manhole covers in terms of which ones were most likely to have issues to those least likely. How accurate was it?  As noted in the book, the top 10% of those ranked most likely to have incidents ended up accounting for 44% of the manhole covers with potentially deadly incidents. In other words, Con Edison through big data analytics was able to better "audit" the population of manhole covers for potential safety issues.  The following video goes into some detail on what the team did:

What lessons can be drawn from this use of Big Data Analytics?
Firstly, algorithms can overcome dirty data. When Professor Rudin was putting together the data to analyse, it included data from the early days of Con Edison, i.e. as in 1880s when Thomas Edison was alive! To illustrate the book notes how there 38 different ways to enter the word "service box" into service records. This is on top of the fact that some of these records were hand written and were documented by people who didn't have a concept of a computer let alone big data analytics.

Second, although the biggest factors seem obvious in hindsight, we should be aware of such conclusions. The point is that data driven decision making is more defensible than a "gut feel", which speaks directly to the professional judgement versus statistical approach of performing audit procedures. The authors further point out that there at least 104 other variables that were contenders and their relative importance cannot be known without preforming such a rigorous analysis.  The point here is that for organizations to succeed and take analytics to the next level need to embrace culturally the concept that, where feasible, organizations should invest in the necessary leg work to obtain conclusions based on solid analysis.

Third, the authors highlight the importance of "explainability". They attribute to the world of artificial intelligence, which refers to the ability of the human user to drill deeper into the analysis generated by the model and explain to operational and executive management why a specific manhole needs to be investigated. In contrast, the authors point out that models that are complex due to the inclusion of numerous variables are difficult to explain. This is a critical point for auditors. As the auditors must be able defend why a particular transaction was chosen over another for audit, big data audit analytics needs to incorporate this concept of explainability.

Finally, it is but another example of how financial audits can benefit from such techniques, given the way non-financial "audits" are using big data techniques to audit and assess information. So internal and external auditors can highlight this (along with two examples identified in the previous post) as part of their big data audit analytics business case.

Big Data: Towards a "data driven audit"

This is the first installment of a multi-part exploration of the audit, assurance, compliance and related concepts brought up in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think (the book is also available as an audiobook and hey while I am at it, here's the link to the e-book ). In this installment, I explore why this is a "must read" for those interested in data driven decision making, big data and information. I will also discuss some examples included in the book that make the case for "data driven audits". 

Why read this book?

This book is written by journalist, Kenneth Cukier, (who claims in this video to have to used the term "big data" before it was commonly used) and Viktor Mayer-Schönberger (an Internet and Governance professor at University of Oxford).

Given the background of the authors, it is an easy to digest book that gives the reader a good understanding of how access to large volumes of data and the use of correlations will change the way business is done and how society has a whole functions - without going into the technical detail of how big data is "crunched" at the back end.  The authors also discuss the following:

• Why more is better: Algorithms improve by being exposed to more data - regardless of how messy it is. On the topic of size, it also comments how statistical sampling is a feature of an era when organizations could not wrap their arms around the data.
• Consumer and business implications: The book is filled with examples that anyone can relate to, such as predicting whether the price of an airplane ticket will go up or down, as well as how Google uses search queries to predict flu outbreaks.
• Enter "Datafication": It also distinguishes "datafication" versus "digitization", where the latter is making something into bits and bytes, whereas the former is something that can be analyzed by some sort of analytic engine. 
• Potentially challenges and negative consequences of big data driven decisions: One of the challenges cited by the author is the "black box" nature of algorithms: how does a common person challenge the an algorithm, when it takes a rocket scientist to understand the algorithm itself? The authors also take the risk of explaining the  danger of subordinating human decision making to algorithms. For example, they note it would be problematic for governments to round up and quarantine people just because they looked up terms related to the flu. 

There are other interesting pieces, but I will bring them up over the next few blog posts. But if you want to get a full understanding, please buy the book - it's worth it!

The case for data driven audits

The book is filled with examples that illustrate the power of big data and how they impact business and society. However, there are a couple of examples that illustrate how financial audits can benefit from such techniques, given the way non-financial "audits" are using big data techniques to audit and assess information. 

Case 1: New York City and Auditing Illegal Conversions: As discussed in this excerpt of the book, Mike Flowers applied big data techniques to the problems of "illegal conversion" in New York city. As noted in the article, illegal conversions is the "the practice of cutting up a dwelling into many smaller units so that it can house as many as 10 times the number of people it was designed for. They are major fire hazards, as well as cauldrons of crime, drugs, disease, and pest infestation. A tangle of extension cords may snake across the walls; hot plates sit perilously on top of bedspreads. People packed this tightly regularly die in blazes". The data scientists working for Flowers, took the 900,000 property lots in the city and correlated "five years of fire data ranked by severity" against the following pieces of data:

• Delinquency in paying property taxes,
• Foreclosure proceedings,
• Odd patterns in their usage of utilities, 
• Non-payment of utilities,
• Type of building,
• Date building was built,
• Ambulance visits,
• Rodent complaints,
• External brickwork.

By correlating all this and other information, they were able to improve the effectiveness of their 200 person inspection team from a "hit rate" of 13% to 70%. (Note: hit rate refers to conditions of building identified as being so bad that it warrants a "vacate order"). 

This is a pretty straightforward evidence for "data driven audits": financial auditors can identify correlations between financial data and non-financial data to determine which financial transactions need more scrutiny than others.

Not convinced?

Well, investors are already doing this. The book gives the example of how an investment firm is using traffic analysis, from Inrix, to determine the sales that a retailer will make and then buy or sell the stock of the retailer on that information. In a senses, the investment is using this as a proxy for sales. In an audit context, auditors can study the vehicular traffic around stores against the sales recorded against such stores and determine if there are issues worth investigating. 

Of course, this endeavor is not merely a matter of copying & pasting data from StatsCan and cobbling up a spreadsheet or two. It is a lot of hard work. However, this is not surprising to anyone who has been performing computer assisted audit techniques for the last decade or so. The challenge has always been in cleaning up the data and making it usable. Some of the challenges that the New York team of statisticians had, include:

• Inconsistent data formats: The team had to bring together data sets from 19 different agencies. Each agency had a different way of describing location. Consequently, this has to be standardized so that each of the 19 data sets can be correlated to the same property.
• Datafying expert intuition: The article describes how brickwork got added as an element to the correlation model. The data scientists on the team observed how the fire inspector could look at a building and know whether it was okay or not.
• Understanding significance of each variable: Each variable must be assessed in its own right to avoid the problem of generalization. For example, rodent infestation is not uniform in its significance across New York city. As noted in the article, " A rat spotted on the posh Upper East Side might generate 30 calls within an hour, but it might take a battalion of rodents before residents in the Bronx felt moved to dial 311".

Although such challenges exist, there's a promise for big data to transform the way audits are done If you recall my blog post on the Oracle vs Google trial, I made a similar point: "the profession should use this opportunity to re-examine what aspects of technology should be a part of the audit practitioners skill set - given that is clear that society's attitude towards technology has clearly changed". Applying this to data driven audits, is actually less of a stretch as auditors have the skill set of dealing with data - instead of learning to code in Java. As a profession,  we need t o understand what data is out there and how this data can be correlated with financial data to make a more effective, efficient, and insightful audit.  

Materially Mistweeted? Tale of the Ticker Symbol

My coworkers informed of a fascinating story of how one letter and an overactive stock rumour mill can provide us a valuable lesson in defining materiality in the world of a 140 character "tweet". As described in this post on Tech Crunch, the stock, "TWTRQ", went from less than a penny to a high of $0.15, which works out to - I am relying on TechCrunch on the math on this one - to be a rise of 1,400%. Wow! Was it the birth of new valuation model that made Wall Street realize the value of TWTRQ? Did the Federal Reserve grant TWTRQ, or  Tweeter Home Entertainment Group, the right to print money out of thin air (as they do with other banks)?

Looks like the free market fundamentalist really got it handed to them on this :) 

Apparently, the collective "wisdom" of the markets that drove investors into a feeding frenzy over TWTRQ. The reason? Well, investors apparently bought the stock of this company, which has been bankrupt since 2007, thinking it was the initial public offering of Twitter (symbol: TWTR). Who would have thought to assess the materiality of a ticker symbol? Could the investors sue the auditors on this one? Now that would be a court case worth watching.  

Of course on a more serious note, it really illustrates how little people do when investing their money in stocks. We are not talking about running the latest financial model pulling XBRL tagged information in real time to determine the value of the company. We're talking about checking financial news sites to see when the stock was released. 

And to finish a lighter a note, we can at least chuckle at Despair.com's take on this. 

UWCISA 8th Biennial Research Symposium: A Unique Experience

Last weekend UWCISA held it's 8th Biennial Research Symposium.

For those who attended, it was a great opportunity to get together and understand what is the leading edge in terms of research in information security, data analytics and assurance issues related to technology. For those that may not be familiar with the conference it is a truly unique format. The Symposium brings academics together to present papers, but also brings together discussants from academia as well as practitioners from the field (click here for the list of papers presented as well as the list of practitioner/academic discussants). It is this unique blend of perspectives that makes the symposium a unique experience.

In prior years, the conference was only on the Friday and Saturday. However, this year the conference included a special XBRL session on the Thursday. This portion of the conference was standalone and actually was sold out! Gerry Trites, head of XBRL Canada, informed the attendees at the conference that over 250 Canadian companies are working to implement XBRL because they file in US GAAP they are effectively forced to produce in XBRL tagged financial statements. He is in the process of pulling together a study that will explore this in greater detail.

On the opening panel, it was interesting to see the different perspectives that were presented about the current state of data and assurance. It really highlighted the challenge of innovation in audit. On the one hand, there is an agreement regarding the tremendous potential that exists due to automate audit tasks and identify issues through analytic techniques. But on the other hand, due to the regulatory nature of audits, there was a consensus that the slow pace that standards change, it will be a while before auditors can take advantage of such techniques. However, the challenge from the audit firm side (as noted by one of presenters) is that the cost or quality advantage gained through R&D will be lost if it is scrutinized and thereby shared with the rest of the audit industry. Another panelists pointed out to another conondrum; can auditors be truly independent of management? His argument was that a more data-driven audit would make the team more objective, which is a more attainable goal.

All in all, it was a great symposium. Special thanks to Efrim Boritz, Lia Boritz, Jenny Thomspson, and the others behind the scenes for getting this mammoth event up and going. Thanks to  Bill Swirsky for keeping us on track and going). And thanks, of course, to the presenters and discussants who presented their papers and views.

See you all in 2 years!