Big Data: Sliding down the Peak of Inflated expectations?

Anyone who follows enterprise IT or has had to research prospective solution would be familiar with the infamous Gartner Hype Cycle. As described in the link, it looks at technology going through a "bubblistic" growth curve. I am not sure if Gartner mentions this explicitly in the book that they published , but the Hype Cycle essentially captures the "herd mentality" that causes Bubbles to form in the Capitalist economic system. Efrim Boritz and I wrote a paper, "A Brief Review of Investment Bubbles throughout History", over a decade ago that analyzes the history of Bubbles going back to Tulipmania back in 1600s to the DotCom Bubble in 2000. In the paper we reference, John Cassidy's "Dot.con: The Greatest Story Ever Sold", as follows:

"According to Cassidy (2002) all speculative bubbles go through four stages: 1) displacement, when something changes people’s expectations about the future; 2) boom, when prices rise sharply and skepticism gives way to greed; 3) euphoria, when people realize the bubble can’t last but they want to cash in on it before it bursts; and 4) bust, when prices plummet and speculators incur great losses. "

So where does Gartner currently see Big Data fitting into the Hype Cycle?

According to the following, it is sliding down "Hype of Inflated Expectations":

Source: Gartner

And actually, there are a couple of facts that back up Gartner's claim. As I discussed in a previous blog post, Forrester cites data woes as continuing problem. For big data to be effective, the underlying data has to have "veracity"  (i.e. as per the definition). The more market driven, perhaps, is the fact that InfoWorld reported, citing a survey from Foote Partners, that "[p]remium pay for 58 big data-related skills and certifications declined by an average of 4.7 percent during the last nine months of 2014". More to the point InfoWord notes "there are signs of a slowdown as businesses learn that jumping into big data is not a recipe for instant ROI".

These factors illustrate how the Gartner Hype Cycle is useful for organizations to identify the current state tech trends to identify when is the right time for that organization to invest in the technology in order to minimize risk of the "bleeding edge" of early adoption but at the same time avoiding the risk of joining the party too late - being taken over by competitors.

Explaining Big Data Technology in under 2 minutes

The class I was teaching this week was looking at Big Data from multiple perspectives, including security. The approach I used with cloud last week was to identify the key differences between ASP and cloud. With Big Data the key difference is between the SQL world of relational databases and the non-relational world of NoSQL technologies, such as Hadoop.

I took a course on Big Data that explained how there is a distributed architecture that enables a "master" to send out the job to vast army of "slaves" to complete the processing. However, how do I explain this in a succinct and effective way to the students? 

In a word, YouTube.

I found this video that gives a pretty good overview of Big Data, but it's real value is how it explains how Hadoop works at a high level (go to 4:10):

Of course, we will be covering  social media later in the term :)