Alan R. Simon - Data Lakes For Dummies

The Hadoop open source environment, particularly the HDFS, is one of the first and most popular examples of big data. Some of the earliest data lakes were built, or at least begun, using HDFS as the foundation.

For purposes of establishing a data lake foundation, Amazon’s S3 and Microsoft’s ADLS both qualify as big data. Why? Both S3 and ADLS support the three Vs of big data, which are as follows:

Storing extremely large volumes of data

Supporting a variety of data, including structured, unstructured, and semi-structured data

Allowing very high velocity for incoming data into the data lake rather than requiring or at least encouraging periodic batches of data

Think of big data as a core technology foundation that supports the three Vs of next-generation data management. Big data by itself, however, is just a platform. It’s the natural body of water — the lake itself — at a popular lakeside resort. When you divide your big data into multiple zones, add capabilities to transmit data across those zones, and then govern the whole environment, you’ve built a data lake surrounding that big data foundation. You’ve done the analytical data equivalent of building the docks, the restaurants, and the boat slips surrounding the lake itself.

In addition to data lakes, you may come across references to data ponds, data puddles, data rivers, data oceans, and data hot tubs. (Just kidding about the last one.) What’s going on here?

Your job when planning, architecting, building, and using a data lake is complicated by the fact that you don’t have an official definition published by some sort of standards body, such as the American National Standards Institute (ANSI) or the International Organization for Standardization (ISO). That means that you or anyone else can define, use, and even publish your own terminology. You can call a smaller portion of a data lake a “data pond” if you want, or refer to a collection of data lakes as a “data ocean.”

Don’t panic! Of all the “data plus a body of water” terms you’ll run across, data lake is by far the most commonly used. All the characteristics of a data lake — solid architecture, support for multiple forms of data, a support ecosystem surrounding the data — apply to what you can call a data pond or any other term.

If William Shakespeare were still around and plied his trade as an enterprise data architect rather than as a writer, he would put it this way: “A data lake by any other name would still be worth the time and effort to build.”

In the early 1990s, data warehousing was the newest and most popular game in town for analytical data management. By the mid-’90s, the concept of a data warehouse was adapted to a data mart — essentially, a smaller-scale data warehouse. The original idea behind a data mart called for the data warehouse feeding a subset of its data into one or more data marts — sort of a “wholesaler-retailer” model.

The first generation of data warehouse projects, especially very large ones, was hallmarked by a high failure rate. By the late ’90s, data warehouses were viewed as large, complex, and expensive efforts that were also very risky. A data mart, on the other hand, was smaller, less complex, and less expensive, and, thus, considered to be less risky.

The need for integrated analytical data was stronger than ever by the end of the ’90s. But just try to get funding for a data warehousing project! Good luck!

Time for plan B.

Data warehouses went out of style for a while. Instead, data marts became the go-to solution for analytic data. No matter how big and complex an environment was, chances are, you’d refer to it as a data mart rather than a data warehouse. In fact, the idea of an independent data mart sprung up, and the original architecture for a data mart — receiving data from a data warehouse rather than directly from source systems — became known as a dependent data mart.

Fast-forward a couple of decades, and it’s back to the future. First, big data sort of evolved into data lakes. Now you have analysts, consultants, and vendors complicating the picture with their own terminology. This won’t be the last time you’ll see shifting names and terminology in the world of analytic data, so stay tuned!

Chapter 2

IN THIS CHAPTER

Taking advantage of big data

Broadening your data type horizons

Implementing a built-to-last analytical data environment

Reeling in existing stand-alone data marts

Blockading new stand-alone data marts

Deciding what to do about your data warehouses

Aligning your data lake plans with your organization’s analytical needs

Setting your data velocity speed limits

Getting a handle on your analytical costs

Suppose that you and about 15 other family members or friends all head to your favorite lake for a weeklong summer vacation.

You love going to the lake because you jump into your sailboat every day and spend hours out on the water. Others in your group, though, have their own favorite pastimes. Some prefer a boat with a little more “oomph” and spend their days in speedboats, zooming up and down the length of the lake. Others prefer leisurely canoeing. Some are into waterskiing, so they take turns latching onto one of those speedboats and zipping along the water. Others in your group are into fishing, and that’s how they spend most of their time at the lake. Still others aren’t all that interested in even going out on the water at all — they plop down on the beach to read, soak up some rays, and even grab a snooze every afternoon.

A data lake is very much like that weeklong trip to your favorite lake. Because a data lake is an enterprise-scale effort, spanning numerous organizations and departments, as well as many different business functions, you and your coworkers will likely seek a variety of varying benefits and outcomes from all that hard work.