Randy Bean - Fail Fast, Learn Faster

Much of the credit for bringing Big Data to the forefront of business and public attention and into general awareness deservedly goes to a set of industry influencers who first widely employed and coined the term Big Data. In a May 2011 special research report, “Big Data: The Next Frontier for Innovation, Competition, and Productivity,” the management consulting firm McKinsey put forth the argument that “Big data will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus.” 3 The McKinsey report continued, “The amount of data in our world has been exploding. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future.”

Common usage of the term “Big Data” can be traced to the McKinsey report and similar reports from IBM that were published around this time. The McKinsey report was prescient in its observation that “Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers.” To me, this was the distinguishing insight. From this point forward, data would no longer be the purview of “a few data-oriented managers” but rather would become the purview of “leaders in every sector.” Data would be open to the masses. Call this the democratization of data. The McKinsey report went on to characterize the dawning era of Big Data as heralding “new waves of productivity growth, innovation, and consumer surplus.” Big Data, like the Internet before it, was destined to change the world. Notably, the McKinsey report did include one important caveat, or small print disclaimer, that the great promises and grand benefits of Big Data would transform business “as long as the right policies and enablers are in place.”

The term Big Data has served to popularize data in a way that decades of data work had never previously achieved, by making it simple, accessible, understood, and big and important. The sudden ascension of Big Data led to my own two-year stint as a Wall Street Journal guest columnist during 2014–2015, writing monthly columns on Big Data and its industry-transforming power and potential. The attentions of Fortune 1000 companies were now engaged at the highest level, compelling the interest of the corporate executive committee and board of directors. This was the moment that leading global corporations embraced data as a critical business asset, and recognized that to compete in the twenty-first century they must possess a data strategy and data leadership.

The advent of Big Data was met by an enthusiastic embrace by the venture capital, private equity, and global investment communities as well. By November 2011, the prominent venture capital firm Accel Partners, an early investor in Facebook, had announced the launch of a “$100M Fund to Invest in Disruptive Big Data Companies.” Accel partner Ping Li noted, “this Big Data wave will usher in a new era of multi-billion software companies that solve existing problems differently or new problems previously unthinkable.” 4 Accel also announced the launch of a Big Data conference to “network and drive deeper discussion on technology trends in the space.”

On a picture-perfect May morning in 2012 on the campus of Stanford University in Palo Alto, California, I attended the inaugural Big Data conference at the invitation of Accel Partners. This event represented a milestone in the history of Big Data. Accel Partners had emerged as the hottest venture firm in Silicon Valley on the heels of their successful investment in Facebook. The event was billed as a “who's who” of the Silicon Valley technology elite. A parade of notable speakers, including Andy Bechtolsheim, co-founder of Sun Microsystems and Doug Cutting, originator of Hadoop and chief architect at Cloudera, the latest Accel investment, followed one another, each extolling the “revolutionary” potential of Big Data. The speakers set forth grand claims for the promise and potential of Big Data. “Big Data was the next wave in technology innovation,” remarked one speaker. 5 “Big Data would change how the world made use of information,” stated another. “Big Data would enable insights that would change society,” commented a third.

Each speaker extolled the technological underpinnings that made Big Data truly compelling, which was the notion that you could simply take your data and “load and go,” thereby shortcutting the massive effort of data preparation and data engineering that was the bane of corporate existence, comprising 80% of data activities. This specifically pertained to the tremendous time and effort required to transform data (“dirty data”) into a usable asset that has meaningful business value. Whole companies and an entire industry had been built to respond to this need. It was the ongoing lament of many a data analyst who complained that they spent “20% of our time on data analysis, but 80% of our time on accessing and preparing the data.”

The big attraction and promise of Big Data for many corporations was the ability to bypass the hundreds of hours of upfront data engineering to access the data much sooner and more easily, for purposes of analysis and putting this information to good use. Anyone who has ever worked in the corporate world knows the painful refrain to how long it takes to answer a new business question that requires adding a new data source: “Fifteen months and five million dollars.” Senior business executives were resigned to a state in which getting value out of data quickly was not something that they could expect to see in their business lifetimes. Then, a cadre of engineers, data experts, and venture investors were heralding a breakthrough. Data would be liberated from the tyranny of the data gatekeepers. Data could now be made available quicker and faster to all.

The radical implication of the load-and-go notion was that data users would no longer have to wait through the long and arduous processes of data engineering that had long hamstrung and thwarted the ambitions of data analysts. Speed would be realized through the ability to shorten the cycle from data access to analytical results. An analogy would be the reduction in travel time resulting from the railroad versus the horse and buggy, or air transport versus travel by land or sea. Big Data would make it possible to get more done for less by lowering the cost of managing data by reducing the need for costly upfront data preparation and data engineering, the dreaded 80% of time and effort.

Long saddled with disparate sources of legacy data, corporations could for the first time successfully integrate these sources due to the cost and speed advantages resulting from Big Data technologies. Corporations could access new sources of information, such as unstructured data sources including documents, text, and pictures, and behavioral data captured through social media channels. The result was a growing sophistication in the data and analytics capabilities of mainstream companies.

Jonathan Bennett is a long-time top executive, EVP, CFO, and business head at The Hartford, a $26 billion insurance and investment firm founded in 1810 with a long history of actuarial analysis, where data has always mattered. Mr. Bennett has maintained a clear-eyed view of both the opportunity and the challenge represented by Big Data. In 2013, he noted that keeping a focus on cost and the benefits of better managing data “is just as important as breaking into new Big Data opportunities. If we can figure it out, cost reductions from the former will help fund expansion in the latter.” 6