Dave Fowler - The Informed Company

Knowledge is power. Knowing more about your surroundings, like which fruit is safe to eat and which will harm you, keeps us safe and successful. Knowing more about the relationships between events, like the inverse relationship between washing hands and influenza outbreaks, minimizes inevitable difficulties. And knowing which strategies lead to success, like which mass‐reforestation strategies lead to sustainable ecosystems, helps us optimize our efforts. In the modern world information on our environment is often in the form of data.

Whether by an individual or a team, data used intentionally and critically is a key part of success. While it's true that anyone can get lucky and stumble upon the right answer, having an idea of which tactics confer the most stability or which metrics indicate big opportunities for growth can drive teams forward quicker.

Our thesis is this: being informed implies being successful. But what does it mean for a person or an organization to be informed exactly? If you are sufficiently informed, you can make decisions with a sense of conviction that your thinking is correct a majority of the time. In the end, your gut feeling (intuition) factors into decision making. However, with well‐organized and accessible information added into the mix, your “gut” can make more thoughtful (and better) decisions.

On the other hand, bad decisions are the result of not knowing enough: not knowing the root causes of numeric anomalies, not knowing where there is potential to invest more capital, and not knowing what the next few months could look like based on previous quarterly performance. Intuition is a powerful and necessary tool, but we believe that the individual and the company owe it to themselves to hone their intuitions. We believe that anyone with access to performance metrics should integrate that information into their decision making. Being informed is a key part of the puzzle to avoiding bad decisions. But what does it look like to be informed in the twenty‐first century?

Companies and their operations continue to become more digitized. Companies no longer have direct access to each customer and must increasingly rely on data to improve and compete. So, organizations end up tracking a lot of data taken from many input streams. But why are so many teams struggling with organizing and leveraging insights from their data? We believe that the problem begins with tooling. We believe that the modern data workflow requires a data stack and operational structure that is ready for today's challenges.

In this book, we will outline what the modern data stack looks like starting from a brand‐new startup all the way through a data‐driven enterprise. We will cover architectures, tools, team organizations, common pitfalls, and best practices. Armed with these, we are confident that your team can identify the evolving needs of your business and implement solutions in your data infrastructure.

There is a chasm between the people who know the technical aspects of how to work with data and the people who know the context behind that data. The technical people can write structured query language (SQL), whereas the business people know which marketing campaigns they've run and what types of users comprise an analysis. This creates friction between the stakeholders who want access to data insights (without always knowing what that data looks like) and the analysts who know their data but not all of its history or significance.

Figure I.1Business context vs technical know how chart.

A modern data stack seeks to bridge that gap. It enables an organization, not just analysts, to work with data. The modern data stack combines disparate input sources into a single understandable format and then stores that data in a single location that a business intelligence (BI) product can connect to. It should provide the information they need at their fingertips, and it should help everyone ask more questions. The modern data stack should empower teams to organize around a key performance indicator (KPI) and make decisions using data ( Figure I.1).

Here are examples of being uninformed versus being informed:

Marketers who keep doubling down on all of their channels versus the ones who iterate or drops them depending on the success and failure of each campaign.

Sales leaders using anecdotes to support what type of customers they should target versus ones who know what customers are coming in and which are closing and can focus their teams accurately on the biggest opportunities.

Support persons who respond to an issue with “what browser are you using?” versus ones who have all of the error information, including the browser details, the account information, and recent actions.

Journalists writing about what matters to them versus ones who can see what resonates.

Product managers who justify their feature by showing the initial results versus ones who set up a test and let the data speak for itself.

Getting data into an understandable format that everyone has access to makes technical and business people collaborators rather than adversaries. It empowers everyone in the organization to make more informed decisions. This book aims to show how to set up a data stack to do just that.

After working with thousands of companies, we recognized four distinct stages of agile data organization ( Figure I.2). Each of these stages has best practices and guidance on how to progress further “up the stack.”

Figure I.2The four stages of agile data organization represent a process that describes the lifecycle of data management.

This book maps onto these stages:

1 Source

2 Lake

3 Warehouse

4 Mart

Each vertical stage pictured could be what a company needs, depending on the resources, size, importance, and needs of data within an organization. Not every company needs all four stages. If a business gets the insights they need while in the source stage, then they should stay in the source stage. Likewise, a company may find it needs a lake to combine all of its data together but may not need the transformations implied in later stages of warehouse development.

Let's begin with an overview of each step.

Many teams only have a few data sources and commonly manipulate and visualize data in Excel or Google sheets. Two common sources that company products run on are Google Analytics and “transactional” database(s), such as PostgreSQL or MySQL. If only a few people at a company need to work with data sources, it's possible to set them up with direct or replica access; it's more agile and straightforward for them to work with the data directly.

We discuss how to configure proper access to data and how to lay the foundation for future organization into a single, unified collection of company data. We also discuss best practices when using tools like in‐app dashboards, spreadsheets, SQL, and BI solutions. Behind these assertions is the principle that teams should be vigorously deliberate when organizing their sources and picking their toolkit. Keeping this in mind not only makes analysts effective at dealing with data early on, but also prepares them to reorganize their input streams into a data lake later on.