Alan R. Simon - Data Lakes For Dummies

Data Lakes For Dummies®

Published by: John Wiley & Sons, Inc.,111 River Street, Hoboken, NJ 07030-5774, www.wiley.com

Copyright © 2021 by John Wiley & Sons, Inc., Hoboken, New Jersey

Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without the prior written permission of the Publisher. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions .

Trademarks:Wiley, For Dummies, the Dummies Man logo, Dummies.com, Making Everything Easier, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and may not be used without written permission. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.

LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: THE PUBLISHER AND THE AUTHOR MAKE NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS WORK AND SPECIFICALLY DISCLAIM ALL WARRANTIES, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE. NO WARRANTY MAY BE CREATED OR EXTENDED BY SALES OR PROMOTIONAL MATERIALS. THE ADVICE AND STRATEGIES CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY SITUATION. THIS WORK IS SOLD WITH THE UNDERSTANDING THAT THE PUBLISHER IS NOT ENGAGED IN RENDERING LEGAL, ACCOUNTING, OR OTHER PROFESSIONAL SERVICES. IF PROFESSIONAL ASSISTANCE IS REQUIRED, THE SERVICES OF A COMPETENT PROFESSIONAL PERSON SHOULD BE SOUGHT. NEITHER THE PUBLISHER NOR THE AUTHOR SHALL BE LIABLE FOR DAMAGES ARISING HEREFROM. THE FACT THAT AN ORGANIZATION OR WEBSITE IS REFERRED TO IN THIS WORK AS A CITATION AND/OR A POTENTIAL SOURCE OF FURTHER INFORMATION DOES NOT MEAN THAT THE AUTHOR OR THE PUBLISHER ENDORSES THE INFORMATION THE ORGANIZATION OR WEBSITE MAY PROVIDE OR RECOMMENDATIONS IT MAY MAKE. FURTHER, READERS SHOULD BE AWARE THAT INTERNET WEBSITES LISTED IN THIS WORK MAY HAVE CHANGED OR DISAPPEARED BETWEEN WHEN THIS WORK WAS WRITTEN AND WHEN IT IS READ.

For general information on our other products and services, please contact our Customer Care Department within the U.S. at 877-762-2974, outside the U.S. at 317-572-3993, or fax 317-572-4002. For technical support, please visit https://hub.wiley.com/community/support/dummies .

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com . For more information about Wiley products, visit www.wiley.com .

Library of Congress Control Number: 2021939570

ISBN 978-1-119-78616-0 (pbk); ISBN 978-1-119-78617-7 (ebk); ISBN 978-1-119-78618-4 (ebk)

1 Cover

2 Title Page

3 Copyright

4 Introduction About This Book Foolish Assumptions Icons Used in This Book Beyond the Book Where to Go from Here

5 Part 1: Getting Started with Data Lakes Chapter 1: Jumping into the Data Lake What Is a Data Lake? The Data Lake Olympics Data Lakes and Big Data The Data Lake Water Gets Murky Chapter 2: Planning Your Day (and the Next Decade) at the Data Lake Carpe Diem: Seizing the Day with Big Data Managing Equal Opportunity Data Building Today’s — and Tomorrow’s — Enterprise Analytical Data Environment Reducing Existing Stand-Alone Data Marts Eliminating Future Stand-Alone Data Marts Establishing a Migration Path for Your Data Warehouses Aligning Data with Decision Making Speedboats, Canoes, and Lake Cruises: Traversing the Variable-Speed Data Lake Managing Overall Analytical Costs Chapter 3: Break Out the Life Vests: Tackling Data Lake Challenges That’s Not a Data Lake, This Is a Data Lake! Exposing Data Lake Myths and Misconceptions Navigating Your Way through the Storm on the Data Lake Building the Data Lake of Dreams Performing Regular Data Lake Tune-ups — Or Else! Technology Marches Forward

6 Part 2: Building the Docks, Avoiding the Rocks Chapter 4: Imprinting Your Data Lake on a Reference Architecture Playing Follow the Leader Guiding Principles of a Data Lake Reference Architecture A Reference Architecture for Your Data Lake Reference Architecture Incoming! Filling Your Data Lake Supporting the Fleet Sailing on Your Data Lake The Old Meets the New at the Data Lake Bringing Outside Water into Your Data Lake Playing at the Edge of the Lake Chapter 5: Anybody Hungry? Ingesting and Storing Raw Data in Your Bronze Zone Ingesting Data with the Best of Both Worlds Joining the Data Ingestion Fraternity Storing Data in Your Bronze Zone Just Passing Through: The Cross-Zone Express Lane Taking Inventory at the Data Lake Bringing Analytics to Your Bronze Zone Chapter 6: Your Data Lake’s Water Treatment Plant: The Silver Zone Funneling Data further into the Data Lake Bringing Master Data into Your Data Lake Impacting the Bronze Zone Getting Clever with Your Storage Options Working Hand-in-Hand with Your Gold Zone Chapter 7: Bottling Your Data Lake Water in the Gold Zone Laser-Focusing on the Purpose of the Gold Zone Looking Inside the Gold Zone Deciding What Data to Curate in Your Gold Zone Seeing What Happens When Your Curated Data Becomes Less Useful Chapter 8: Playing in the Sandbox Developing New Analytical Models in Your Sandbox Comparing Different Data Lake Architectural Options Experimenting and Playing Around with Data Chapter 9: Fishing in the Data Lake Starting with the Latest Guidebook Taking It Easy at the Data Lake Staying in Your Lane Doing a Little Bit of Exploring Putting on Your Gear and Diving Underwater Chapter 10: Rowing End-to-End across the Data Lake Keeping versus Discarding Data Components Getting Started with Your Data Lake Shifting Your Focus to Data Ingestion Finishing Up with the Sandbox

7 Part 3: Evaporating the Data Lake into the Cloud Chapter 11: A Cloudy Day at the Data Lake Rushing to the Cloud Running through Some Cloud Computing Basics The Big Guys in the Cloud Computing Game Chapter 12: Building Data Lakes in Amazon Web Services The Elite Eight: Identifying the Essential Amazon Services Looking at the Rest of the Amazon Data Lake Lineup Building Data Pipelines in AWS Chapter 13: Building Data Lakes in Microsoft Azure Setting Up the Big Picture in Azure The Magnificent Seven, Azure Style Filling Out the Azure Data Lake Lineup Assembling the Building Blocks

8 Part 4: Cleaning Up the Polluted Data Lake Chapter 14: Figuring Out If You Have a Data Swamp Instead of a Data Lake Designing Your Report Card and Grading System Looking at the Raw Data Lockbox Knowing What to Do When Your Data Lake Is Out of Order Too Fast, Too Slow, Just Right: Dealing with Data Lake Velocity and Latency Dividing the Work in Your Component Architecture Tallying Your Scores and Analyzing the Results Chapter 15: Defining Your Data Lake Remediation Strategy Setting Your Key Objectives Doing Your Gap Analysis Identifying Resolutions Establishing Timelines Defining Your Critical Success Factors Chapter 16: Refilling Your Data Lake The Three S’s: Setting the Stage for Success Refining and Enriching Existing Raw Data Making Better Use of Existing Refined Data Building New Pipelines with Newly Ingested Raw Data