Nathalie Peyrard - Statistical Approaches for Hidden Variables in Ecology

Many methods have been proposed for estimating the parameters of a model featuring latent variables. From a frequentist perspective, the oldest and most widespread means of computing the maximum likelihood estimator is the expectation–maximization (EM) algorithm, which draws on the fact that the parameters for many of these models would be easy to estimate if the latent variables could be observed. The EM algorithm alternates between two steps: in step E, all of the quantities involving latent variables are calculated in order to update the estimation of parameters in the second step, M. Step E focuses on determining the conditional distribution of latent variables given the observed data. This calculation may be immediate (as in the case of mixture models and certain mixed models) or possible but costly (as in the case of HMMs); alternatively, it may be impossible for combinatorial or formal reasons.

The estimation problem is even more striking in the context of Bayesian inference, as a conditional distribution must be established not only for the latent variables, but also for parameters. Once again, except in very specific circumstances, precise determination of this joint conditional law (latent variables and parameters) is usually impossible.

The inference methods used in models with a non-calculable conditional law fall into two broad categories: sampling methods and approximation methods. Sampling methods use a sample of data relating to the non-calculable law to obtain precise estimations of all relevant quantities. This category includes the Monte Carlo, the Markov chain Monte Carlo (MCMC) and the sequential Monte Carlo (SMC) methods. These algorithms are inherently random, and are notably used in Bayesian inference. Methods in the second category are used to determine an approximation of the conditional law of the latent variables (and, in the Bayesian case, of parameters) based on observations. This category includes variational methods and their extensions. These approaches vary in terms of the measure of proximity between the approximated law and the actual conditional law, and in terms of the distribution family used when searching for the approximation.

This book provides an overview of recent work on statistical modeling and estimation in latent variable models for ecology. The different chapters illustrate the main principles described above. In some cases, they present statistical methods based on classical models and algorithms; in others, the focus is on developments from recent research in others. Each chapter addresses a specific ecological issue and a modeling approach to solving the problem, illustrated using one or more case studies.

Readers may also access the R code 1in order to make use of the tools presented here, applied to their own data.

Most of the questions associated with the case studies presented here relate to the comprehension or description of systems. While the issue of forecasting and prediction is touched upon in some chapters, this subject lies outside the main scope of our work. The issue of missing data (i.e. values not observed in samples) is also not addressed either. Finally, note that this work is not an exhaustive summary of latent variable models, or of the inference methods and algorithms used with these models. Each chapter touches on the question of inference in relation to the selected model; readers wishing to explore the subject in greater depth may wish to consult the references provided.

This book is not designed to be read from front to back, but rather as a resource on which ecologists working with models or statisticians working in the field of ecology may draw. Chapters are arranged in order of ecological scale, from individuals up to ecosystems, providing an initial interpretive framework. Another approach would be to consider the nature of the hidden variable being modeled. One final approach would be to examine different statistical models: some models are used in several chapters, in connection with questions on different scales, and using different estimation methods.

Table I.1gives a summary of the contents of the different chapters and is designed to help readers identify material which is of interest to them.

The examples described above, along with those presented in the following chapters, highlight the immense flexibility of latent variables models. These models, involving one or more latent layers, provide a rich framework for the description of complex dependency structures, and/or for the approximation of a mechanistic description of the phenomena involved.

However, it is important to note that the most sophisticated models are almost always the most complex in terms of inference. It would be wrong to assume that inference simply “happens”, whatever the statistical approach (frequentist, Bayesian, etc.). At the time of writing, there is no fully generic approach suitable for use with all models, and this is unlikely to change in the near future. Even the best-established algorithms (EM, MCMC, etc.) require users to have a good understanding of the underlying principles in order to guide and control their behavior, and/or to adjust the algorithm as needed. This need for adjustment is clearly visible in the chapters of this book.

To conclude this introduction, we wish to highlight two areas for further research in ecology, drawing on statistical modeling of hidden variables, which are not covered in this book but which show promise: namely the integration (or combination) of data from multiple sources, and the use of participative scientific data.

Table I.1. Chapters and contents

Several works have recently been published on the integration of data from multiple sources in the field of ecology (Miller et al . 2019; Isaac et al . 2020). The aim of the authors is to systematically improve the precision of estimated data, potentially decreasing sample size, and to enable the estimation of parameters that cannot be approximated by any other means. Data integration generally involves a hierarchical modeling approach in which the hidden variable is present in all of the sources used in its estimation.

Data from participative scientific activity has also received increasing attention in the literature in recent years (Dickinson et al . 2012; McKinley et al . 2017). This is due to the increasing availability of the data, and to the fact that information can now be collected across an increasingly broad spatial and temporal scale. Participative data sources are a fascinating subject of study in statistical ecology, raising a number of new challenges in terms of spatial bias in sampling, or variations in participant expertise. Once again, a clear distinction between the ecological processes embodied by the hidden variables and the associated observation methods is essential in order to develop a full response to the ecological question.

Dickinson, J.L., Shirk, J., Bonter, D., Bonney, R., Crain, R.L., Martin, J., Phillips, T., Purcell, K. (2012). The current state of citizen science as a tool for ecological research and public engagement. Frontiers in Ecology and the Environment , 10(6), 291–297.

Isaac, N.J., Jarzyna, M.A., Keil, P., Dambly, L.I., Boersch-Supan, P.H., Browning, E., Freeman, S.N., Golding, N., Guillera-Arroita, G., Henrys, P.A., Jarvis, S., Lahoz-Monfort, J., Pagel, J., Pescott, O.L., Schmucki, R., Simmonds, E.G., O’Hara, R.B. (2020). Data integration for large-scale models of species distributions. Trends in Ecology & Evolution , 35(1), 56–67.