[Paper Review] Gaussian Process-Based Bayesian Nonparametric Inference of Population Trajectories from Gene Genealogies

Julia A. Palacios, Vladimir N. Minin|arXiv (Cornell University)|Dec 18, 2011

Genetic and phenotypic traits in livestock35 references4 citations

TL;DR

This paper proposes a novel Gaussian process-based Bayesian nonparametric method for inferring population size trajectories from genealogies using the coalescent model. By treating coalescent times as an inhomogeneous point process, the method leverages advanced nonparametric inference techniques for Poisson processes, achieving higher accuracy, precision, and more realistic uncertainty estimates than existing Gaussian Markov random field (GMRF) methods in simulations and real viral data analyses.

ABSTRACT

Changes in population size influence genetic diversity of the population and, as a result, leave a signature of these changes in individual genomes in the population. We are interested in the inverse problem of reconstructing past population dynamics from genomic data. We start with a standard framework based on the coalescent, a stochastic process that generates genealogies connecting randomly sampled individuals from the population of interest. These genealogies serve as a glue between the population demographic history and genomic sequences. It turns out that only the times of genealogical lineage coalescences contain information about population size dynamics. Viewing these coalescent times as a point process, estimating population size trajectories is equivalent to estimating a conditional intensity of this point process. Therefore, our inverse problem is similar to estimating an inhomogeneous Poisson process intensity function. We demonstrate how recent advances in Gaussian process-based nonparametric inference for Poisson processes can be extended to Bayesian nonparametric estimation of population size dynamics under the coalescent. We compare our Gaussian process (GP) approach to one of the state of the art Gaussian Markov random field (GMRF) methods for estimating population trajectories. Using simulated data, we demonstrate that our method has better accuracy and precision. Next, we analyze two genealogies reconstructed from real sequences of hepatitis C and human Influenza A viruses. In both cases, we recover more believed aspects of the viral demographic histories than the GMRF approach. We also find that our GP method produces more reasonable uncertainty estimates than the GMRF method.

Motivation & Objective

• To develop a flexible, nonparametric Bayesian method for estimating effective population size trajectories from genealogical data without assuming fixed grid or piecewise constant functions.
• To overcome limitations of existing nonparametric methods that rely on arbitrary discretization or piecewise continuous priors with fixed change points.
• To improve estimation accuracy and uncertainty quantification by leveraging modern Gaussian process techniques adapted to the coalescent process.
• To enable future extensions to multivariate modeling, multiple loci, and integration with molecular sequence data.
• To provide a computationally tractable framework that avoids the need for a priori grid specification during MCMC sampling.

Proposed method

• The coalescent process with variable population size is reformulated as an inhomogeneous point process, where coalescent times represent events in a continuous-time point process.
• The intensity function of this point process is modeled nonparametrically using a Gaussian process prior with zero mean and a covariance matrix controlling smoothness.
• A data augmentation scheme based on thinning algorithms—adapted from Adams et al. (2009) for Poisson processes—is developed specifically for the coalescent, enabling posterior computation.
• The method uses a non-GMRF Gaussian process prior (e.g., Brownian motion, Ornstein-Uhlenbeck) to ensure computational tractability through sparse precision matrices.
• Posterior inference is performed via MCMC, with the effective population size trajectory estimated as the posterior median of the GP realizations.
• The framework is designed to be extendable to multivariate settings, allowing joint modeling of population size and external time series.

Experimental results

Research questions

• RQ1 Can Gaussian process-based nonparametric inference improve the accuracy and precision of population size trajectory estimation compared to existing GMRF-based methods?
• RQ2 How does the proposed method handle uncertainty in population size dynamics, particularly in comparison to GMRF approaches?
• RQ3 Can the method be adapted to the coalescent model by treating coalescent times as a point process with a stochastic intensity function?
• RQ4 Does the method maintain robustness and computational feasibility when applied to real viral genealogies with complex demographic histories?
• RQ5 Can the framework be extended to incorporate genealogical uncertainty and multiple loci in future developments?

Key findings

• In simulations, the GP-based method demonstrated superior accuracy and precision in estimating population size trajectories compared to the state-of-the-art GMRF method.
• The method produced more reasonable and realistic uncertainty estimates than the GMRF approach, particularly in regions with sparse coalescent events.
• When applied to hepatitis C and human influenza A virus genealogies, the GP method recovered more biologically plausible demographic histories than the GMRF method.
• The choice of GP prior (e.g., Brownian motion, Ornstein-Uhlenbeck) had minimal impact on results, indicating robustness to prior specification.
• The precision parameter in the Brownian motion prior showed low sensitivity to perturbations, supporting stable inference.
• The method enables future extensions to multivariate modeling of population size and external covariates, such as environmental or epidemiological time series.