[Paper Review] Linking statistical and ecological theory: Hubbell's unified neutral theory of biodiversity as a hierarchical Dirichlet process

Keith Harris, Todd L. Parsons+4 more|arXiv (Cornell University)|Oct 15, 2014

Gut microbiota and health53 references4 citations

TL;DR

This paper proposes a Bayesian fitting strategy for Hubbell's Unified Neutral Theory of Biodiversity (UNTB) by showing that the multi-site UNTB converges to a hierarchical Dirichlet process (HDP) in the large population limit, enabling efficient and accurate parameter estimation across multiple sites. The method outperforms existing approximations in low-immigration scenarios and reveals that human gut microbiomes show non-neutral dynamics at higher taxonomic levels but near-neutrality within certain genera, with BMI negatively correlating with immigration rates in Ruminococcaceae.

ABSTRACT

Neutral models which assume ecological equivalence between species provide null models for community assembly. In Hubbell's Unified Neutral Theory of Biodiversity (UNTB), many local communities are connected to a single metacommunity through differing immigration rates. Our ability to fit the full multi-site UNTB has hitherto been limited by the lack of a computationally tractable and accurate algorithm. We show that a large class of neutral models with this mainland-island structure but differing local community dynamics converge in the large population limit to the hierarchical Dirichlet process. Using this approximation we developed an efficient Bayesian fitting strategy for the multi-site UNTB. We can also use this approach to distinguish between neutral local community assembly given a non-neutral metacommunity distribution and the full UNTB where the metacommunity too assembles neutrally. We applied this fitting strategy to both tropical trees and a data set comprising 570,851 sequences from 278 human gut microbiomes. The tropical tree data set was consistent with the UNTB but for the human gut neutrality was rejected at the whole community level. However, when we applied the algorithm to gut microbial species within the same taxon at different levels of taxonomic resolution, we found that species abundances within some genera were almost consistent with local community assembly. This was not true at higher taxonomic ranks. This suggests that the gut microbiota is more strongly niche constrained than macroscopic organisms, with different groups adopting different functional roles, but within those groups diversity may at least partially be maintained by neutrality.We also observed a negative correlation between body mass index and immigration rates within the family Ruminococcaceae.

Motivation & Objective

• To develop a computationally efficient and accurate Bayesian fitting strategy for the multi-site Unified Neutral Theory of Biodiversity (UNTB), which has been limited by intractable likelihood calculations in the general case.
• To establish a formal mathematical link between ecological neutral theory and a statistical model from machine learning—the hierarchical Dirichlet process (HDP)—by showing convergence in the large population limit.
• To distinguish between non-neutral metacommunity distributions and fully neutral UNTB dynamics by testing the fit of the HDP approximation to empirical data.
• To apply the method to real-world datasets, including tropical tree communities and human gut microbiomes, to assess the role of neutrality in macro- and micro-ecological systems.

Proposed method

• The authors demonstrate that a broad class of multi-site neutral models with mainland-island structure and fixed local community sizes converge to the hierarchical Dirichlet process (HDP) in the large population limit, regardless of local dynamics, provided species are ecologically equivalent.
• They adapt existing Bayesian inference techniques for the HDP—specifically Gibbs sampling with a Chinese Restaurant Franchise metaphor—to estimate UNTB parameters, including the fundamental biodiversity number (θ) and site-specific immigration rates (m_i).
• The method uses a non-informative prior and a Gibbs sampling algorithm that iteratively resamples species abundances and immigration rates, generating full posterior distributions for parameters.
• The approach is validated using simulated data with varying immigration rates and θ values, comparing performance against Etienne’s approximate and exact likelihood methods.
• The HDP approximation is applied to real data: a tropical tree dataset from Panama and a human gut microbiome dataset of 570,851 sequences across 278 samples, with taxonomic resolution tested at multiple levels.
• The method enables testing for neutrality by comparing the fit of the HDP model to data under different assumptions: full neutrality (metacommunity and local communities neutral) versus non-neutral metacommunity with neutral local assembly.

Experimental results

Research questions

• RQ1 Does the multi-site UNTB converge to the hierarchical Dirichlet process (HDP) in the large population limit, regardless of local community dynamics?
• RQ2 Can the HDP approximation provide a more accurate and computationally efficient Bayesian fitting strategy for the full multi-site UNTB than existing likelihood-based approximations?
• RQ3 Is the human gut microbiome consistent with full neutrality at the community level, or are niche processes dominant?
• RQ4 Are there taxonomic groups within the gut microbiome where local community assembly is approximately neutral, even if the whole community is not?
• RQ5 Does body mass index (BMI) correlate with immigration rates in specific gut microbial taxa, suggesting altered ecological dynamics in obesity?

Key findings

• The HDP approximation provides a more accurate estimate of UNTB parameters than Etienne’s approximate method, especially in low-immigration scenarios (e.g., when I << θ), where the HDP outperforms the approximation and matches the performance of Etienne’s 'exact' method.
• For the tropical tree dataset from Panama, the HDP approximation produced parameter estimates (θ ≈ 231 ± 22, immigration rates ≈ 65.5 ± 5.9 for BCI) that closely matched Etienne’s 'exact' method, validating the approach under high-θ conditions.
• In the human gut microbiome dataset, full-community neutrality was rejected, indicating that niche processes dominate at higher taxonomic levels.
• Within certain genera—particularly in the family Ruminococcaceae—species abundance distributions were nearly consistent with local neutral assembly, suggesting that neutrality may play a role in maintaining diversity within functional groups.
• A significant negative correlation was found between body mass index (BMI) and immigration rates within the Ruminococcaceae family, implying that obesity may reduce the influence of external immigration and favor local growth in this key carbohydrate-metabolizing group.
• The HDP-based method is scalable and efficient, handling large microbiome datasets (e.g., 570,851 sequences) with full posterior inference, outperforming existing methods in computational tractability and accuracy for complex, low-immigration scenarios.