Rebekah B. Esmaili - Earth Observation Using Python

Being involved in a number of international collaborations to improve the integration of Earth observations, I can certainly emphasize the importance of working together, data sharing, and demonstrating the value of data fusion. I am very honored to write this Foreword, since this book focuses on these issues and provides an excellent guide with relevant examples for the reader to follow and relate to.

Dr. Mitch GoldbergChief Program Scientist NOAA-National Environmental Satellite, Data, and Information Service June 22, 2020

This book evolved from a series of Python workshops that I developed with the help of Eviatar Bach and Kriti Bhargava from the Department of Atmospheric and Oceanic Science at the University of Maryland. I am very grateful for their assistance providing feedback for the examples in this book and for leading several of these workshops with me.

This book would not exist without their support and contributions from others, including:

The many reviewers who took the time to read versions of this book, several of whom I have never met in person. Thanks to modern communication systems, I was able to draw from their expertise. Their constructive feedback and insights not only helped to improve this quality and breadth of the book but also helped me hone my technical writing skills.

Rituparna Bose, Jenny Lunn, Layla Harden, and the rest of the team at AGU and Wiley for keeping me informed, organized, and on track throughout this process. They were truly a pleasure to work with.

Nadia Smith and Chris Barnet, and my other colleagues at Science and Technology Corp., who provided both feedback and conversations that helped shape some of the ideas and content in this book.

Catherine Thomas, Clare Flynn, Erin Lynch, and Amy Ho for their endless encouragement and support.

Tracie and Farid Esmaili, my parents, who encouraged me to aim high even if they were initially confused when their atmospheric scientist daughter became interested in “snakes.”

Python is a programming language that is rapidly growing in popularity. The number of users is large, although difficult to quantify; in fact, Python is currently the most tagged language on stackoverflow.com, a coding Q&A website with approximately 3 million questions a year. Some view this interest as hype, but there are many reasons to join the movement. Scientists are embracing Python because it is free, open source, easy to learn, and has thousands of add‐on packages. Many routine tasks in the Earth sciences have already been coded and stored in off‐the‐shelf Python libraries. Users can download these libraries and apply them to their research rather than simply using older, more primitive functions. The widespread adoption of Python means scientists are moving toward a common programming language and set of tools that will improve code shareability and research reproducibility.

Among the wealth of remote sensing data available, satellite datasets are particularly voluminous and tend to be stored in a variety of binary formats. Some datasets conform to a “standard” structure, such as netCDF4.However, because of uncoordinated efforts across different agencies and countries, such standard formats bear their own inconsistencies in how data are handled and intended to be displayed. To address this, many agencies and companies have developed numerous “quick look” methods. For instance, data can be searched for and viewed online as Jpeg images, or individual files can be displayed with free, open‐source software tools like Panoply ( www.giss.nasa.gov/tools/panoply/) and HDFView ( www.hdfgroup.org/downloads/hdfview/).

Still, scientists who wish to execute more sophisticated visualization techniques will have to learn to code. Coding knowledge is not the only limitation for users. Not all data are “analysis ready,” i.e., in the proper input format for visualization tools. As such, many pre‐processing steps are required to make the data usable for scientific analysis. This is particularly evident for data fusion, where two datasets with different resolutions must first be mapped to the same grid before they are compared. Many data users are not satellite scientists or professional programmers but rather members of other research and professional communities, these barriers can be too great to overcome. Even to a technical user, the nuances can be frustrating. At worst, obstacles in coding and data visualization can potentially lead to data misuse, which can tarnish the work of an entire community.

The purpose of this text is to provide an overview of the common preparatory work and visualization techniques that are applied to environmental satellite data using the Python language. This book is highly example‐driven, and all the examples are available online. The exercises are primarily based on hands‐on tutorial workshops that I have developed. The motivation for producing this book is to make the contents of the workshops accessible to more Earth scientists, as very few Python books currently available target the Earth science community.

This book is written to be a practical workbook and not a theoretical textbook. For example, readers will be able to interactively run prewritten code interactively alongside the text to guide them through the code examples. Exercises in each section build on one another, with incremental steps folded in. Readers with minimal coding experience can follow each “baby step” to get them up to become “spun up” quickly, while more experienced coders have the option of working with the code directly and spending more time on building a workflow as described in Section III.

The exercises and solutions provided in this book use Jupyter Notebook, a highly interactive, web‐based development environment. Using Jupyter Notebook, code can be run in a single line or short blocks, and the results are generated within an interactive documented format. This allows the student to view both the Python commands and comments alongside the expected results. Jupyter Notebook can also be easily converted to programs or scripts than can be executed on Linux Machines for high‐performance computing. This provides a friendly work environment to new Python users. Students are also welcome to develop code in any environment they wish, such as the Spyder IDE or using iPython.

While the material builds on concepts learned in other chapters, the book references the location of earlier discussions of the material. Within each chapter, the examples are progressive. This design allows students to build on their understanding knowledge (and learn where to find answers when they need guidance) rather than memorizing syntax or a “recipe.” Professionally, I have worked with many datasets and I have found that the skills and strategies that I apply on satellite data are fairly universal. The examples in this book are intended to help readers become familiar with some of the characteristic quirks that they may encounter when analyzing various satellite datasets in their careers. In this regard, students are also strongly encouraged to submit requests for improvements in future editions.

Like many technological texts, there is a risk that the solutions presented will become outdated as new tools and techniques are developed. The sizable user community already contributing to Python implies it is actively advancing; it is a living language in contrast to compiled, more slowly evolving legacy languages like Fortran and C/C++. A drawback of printed media is that it tends to be static and Python is evolving more rapidly than the typical production schedule of a book. To mitigate this, this book intends to teach fluency in a few, well‐established packages by detailing the steps and thought processes needed for a user needs to carry out more advanced studies. The text focuses discipline‐agnostic packages that are widely used, such as NumPy, Pandas,and xarray,as well as plotting packages such as Matplotliband Cartopy.