Alan R. Simon - Data Lakes For Dummies

9 Part 5: Making Trips to the Data Lake a Tradition Chapter 17: Checking Your GPS: The Data Lake Road Map Getting an Overhead View of the Road to the Data Lake Assessing Your Current State of Data and Analytics Putting Together a Lofty Vision Building Your Data Lake Architecture Deciding on Your Kickoff Activities Expanding Your Data Lake Chapter 18: Booking Future Trips to the Data Lake Searching for the All-in-One Data Lake Spreading Artificial Intelligence Smarts throughout Your Data Lake

10 Part 6: The Part of Tens Chapter 19: Top Ten Reasons to Invest in Building a Data Lake Supporting the Entire Analytics Continuum Bringing Order to Your Analytical Data throughout Your Enterprise Retiring Aging Data Marts Bringing Unfulfilled Analytics Ideas out of Dry Dock Laying a Foundation for Future Analytics Providing a Region for Experimentation Improving Your Master Data Efforts Opening Up New Business Possibilities Keeping Up with the Competition Getting Your Organization Ready for the Next Big Thing Chapter 20: Ten Places to Get Help for Your Data Lake Cloud Provider Professional Services Major Systems Integrators Smaller Systems Integrators Individual Consultants Training Your Internal Staff Industry Analysts Data Lake Bloggers Data Lake Groups and Forums Data-Oriented Associations Academic Resources Chapter 21: Ten Differences between a Data Warehouse and a Data Lake Types of Data Supported Data Volumes Different Internal Data Models Architecture and Topology ETL versus ELT Data Latency Analytical Uses Incorporating New Data Sources User Communities Hosting

11 Index

12 About the Author

13 Connect with Dummies

14 End User License Agreement

1 Chapter 1 TABLE 1-1 Data Lake Zones

2 Chapter 2 TABLE 2-1 Matching Analytics and Business Questions

3 Chapter 9TABLE 9-1 Hospital Data Lake Permissions

4 Chapter 13TABLE 13-1 ADLS Storage Tiers

5 Chapter 15TABLE 15-1 Data Lake Remediation PrioritiesTABLE 15-2 Defining Data Lake Remediation Success

6 Chapter 17TABLE 17-1 Your Five-Phase A LAKE Data Lake Road MapTABLE 17-2 A LAKE Confirmation Loopbacks

1 Chapter 1FIGURE 1-1: A logically centralized data lake with underlying physical decentra...FIGURE 1-2: Cloud-based data lake solutions.FIGURE 1-3: Different types of data in your data lake.FIGURE 1-4: Source applications feeding data into your data lake.

2 Chapter 2FIGURE 2-1: The vision of an enterprise data warehouse.FIGURE 2-2: The reality of numerous stand-alone data marts.FIGURE 2-3: Using a data lake to retire data marts.FIGURE 2-4: Leaving a data mart intact and alongside your data lake.FIGURE 2-5: Incorporating a data mart into your data lake.FIGURE 2-6: Migrating your data warehouse into your new data lake.FIGURE 2-7: A data pipeline into, through, and then out of the data lake.FIGURE 2-8: An easy way to understand data pipelines and data lakes.

3 Chapter 3FIGURE 3-1: Playing “find the data lake.”

4 Chapter 4FIGURE 4-1: A reference architecture for data lake reference architectures.FIGURE 4-2: Two classes of inbound data flows for your data lake.FIGURE 4-3: Object storage as the fundamental storage technology for your data ...FIGURE 4-4: Incorporating database technology along with object storage.FIGURE 4-5: Embedding a data warehouse into your data lake environment.FIGURE 4-6: Adding heterogeneity to your data lake’s bronze zone.FIGURE 4-7: Adding heterogeneity to your data lake’s bronze zone.FIGURE 4-8: Incorporating the user layer of a legacy data warehouse into your d...FIGURE 4-9: Subsuming an end-to-end legacy data warehouse into your new data la...FIGURE 4-10: Your data lake feeding your data warehouse.FIGURE 4-11: Split-streaming data feeds to support both your data lake and your...FIGURE 4-12: Ongoing data interchange between your data lake and your data ware...FIGURE 4-13: A data lake that is much larger than a data warehouse.FIGURE 4-14: A data warehouse that is much larger than a data lake.FIGURE 4-15: Feeding external data into the data lake.FIGURE 4-16: On-demand access to external data for your analytics.FIGURE 4-17: Drilling-site sensors and a data lake at an energy exploration com...FIGURE 4-18: Edge analytics existing outside the control of the data lake.FIGURE 4-19: Remote data from edge analytics can also be sent to the data lake.

5 Chapter 5FIGURE 5-1: Data flowing into your data lake bronze zone.FIGURE 5-2: Three different operational data feeds into your data lake bronze z...FIGURE 5-3: Multiple subscribers to sensor and video data streams.FIGURE 5-4: Using a streaming service to split-stream data into both a data lak...FIGURE 5-5: Under-the-covers “micro-batching” within streaming input to your da...FIGURE 5-6: The Lambda data ingestion architecture for your data lake.FIGURE 5-7: The Kappa data ingestion architecture for your data lake.FIGURE 5-8: Going for storage simplicity with only object storage in your bronz...FIGURE 5-9: Implementing a multi-component bronze zone.FIGURE 5-10: Ingesting data from a database: object storage versus database in ...FIGURE 5-11: Carrying a bronze zone database through to your data lake gold zon...FIGURE 5-12: Carrying bronze zone object storage through to your data lake gold...FIGURE 5-13: Going back to a database in a multi-component gold zone.FIGURE 5-14: Data streaming doing double duty as bronze zone storage for raw da...FIGURE 5-15: Three different models for linking your analytics with streaming d...

6 Chapter 6FIGURE 6-1: Refining an image between the bronze zone and the silver zone.FIGURE 6-2: Enriching an image for storage in the data lake silver zone.FIGURE 6-3: Enriching a tweet by determining and attaching sentiment analysis.FIGURE 6-4: Building a master data taxonomy for your data lake.FIGURE 6-5: Decisions, decisions: What should you do with bronze zone data dest...FIGURE 6-6: Redefining your data lake zone boundaries rather than unnecessarily...FIGURE 6-7: Ingesting a raw tweet.FIGURE 6-8: Enriching a tweet followed by shifting your zone boundary rather th...FIGURE 6-9: Step 1: Ingesting raw data into your bronze zone.FIGURE 6-10: Step 2: Moving data into the silver zone rather than copying data.FIGURE 6-11: Deciding whether to keep a raw image after refinement and enhancem...FIGURE 6-12: Your data lake silver zone using Amazon S3.FIGURE 6-13: Dividing your silver zone content among three different flavors of...FIGURE 6-14: Carrying hierarchical storage back into your data lake bronze zone...FIGURE 6-15: Step 1: Refine and enrich an image in your data lake silver zone.FIGURE 6-16: Step 2: Move bronze zone image to S3 Glacier to save on storage co...

7 Chapter 7FIGURE 7-1: Peeking inside the gold zone.FIGURE 7-2: Building a curated gold zone data package.FIGURE 7-3: Adding database data to object store data inside a gold zone curate...FIGURE 7-4: Using persistent data streams for your gold zone curated data.FIGURE 7-5: Using a specialized data store in your data lake gold zone.FIGURE 7-6: Relocating an infrequently used or retired data package to less-exp...

8 Chapter 8FIGURE 8-1: Using the data lake sandbox for analytical development.FIGURE 8-2: Migrating curated data from the sandbox to the gold zone as analyti...FIGURE 8-3: Using a data lake sandbox to explore architectural options.FIGURE 8-4: Moving a graph database curated data package from the sandbox into ...FIGURE 8-5: Exploratory analytics and your data lake sandbox.

9 Chapter 9FIGURE 9-1: Data lakes and passive analytics users.FIGURE 9-2: Light analytics user access to a data lake gold zone.FIGURE 9-3: Light analytics user access to a database within the data lake gold...FIGURE 9-4: A multistep gold zone integration process for a light analytics use...FIGURE 9-5: Using a data abstraction tool for data lake access simplicity.FIGURE 9-6: Using a data abstraction tool to integrate database and object data...