[Paper Review] Elementary methods provide more replicable results in microbial differential abundance analysis

Juho Pelto, Kari Auranen+2 more|arXiv (Cornell University)|Apr 3, 2024

Gut microbiota and health4 citations

TL;DR

This study evaluates 14 differential abundance methods in microbiome research using 53 datasets from 16S rRNA and shotgun metagenomic studies. It finds that elementary methods—such as Wilcoxon tests, ordinal regression, linear regression/t-tests on relative abundances, and logistic regression on presence/absence—deliver more reproducible results across dataset partitions and independent studies, outperforming complex methods in consistency despite lacking formal ground truth.

ABSTRACT

Differential abundance analysis is a key component of microbiome studies. Although dozens of methods exist there is currently no consensus on the preferred methods. While the correctness of results in differential abundance analysis is an ambiguous concept and cannot be fully evaluated without setting the ground truth and employing simulated data, we argue that a well-performing method should be effective in producing highly reproducible results. We compared the performance of 14 differential abundance analysis methods by employing datasets from 53 taxonomic profiling studies based on 16S rRNA gene or shotgun metagenomic sequencing. For each method, we examined how the results replicated between random partitions of each dataset and between datasets from separate studies. While certain methods showed good consistency, some widely used methods were observed to produce a substantial number of conflicting findings. Overall, when considering consistency together with sensitivity, the best performance was attained by analyzing relative abundances with a non-parametric method (Wilcoxon test or ordinal regression model) or linear regression/t-test. Moreover, a comparable performance was obtained by analyzing presence/absence of taxa with logistic regression.

Motivation & Objective

• To assess the reproducibility of differential abundance analysis methods across diverse microbiome datasets.
• To identify which methods produce consistent results when applied to random partitions of the same dataset and across independent studies.
• To challenge the assumption that complex, specialized methods are inherently superior to basic statistical approaches in microbiome research.
• To provide evidence-based guidance for selecting methods that maximize result consistency in differential abundance analysis.

Proposed method

• The authors evaluated 14 differential abundance methods across 53 taxonomic profiling datasets from 16S rRNA and shotgun metagenomic sequencing.
• For each dataset, results were assessed for consistency across multiple random partitions to measure internal reproducibility.
• Inter-study consistency was evaluated by comparing results across datasets from separate studies, using overlap in detected differentially abundant taxa as a metric.
• Relative abundances were analyzed using non-parametric tests (Wilcoxon), ordinal regression, linear regression, and t-tests.
• Presence/absence data were analyzed using logistic regression to assess its reproducibility.
• Performance was evaluated based on consistency across partitions and studies, with sensitivity considered alongside reproducibility.

Experimental results

Research questions

• RQ1 Which differential abundance methods produce the most reproducible results when applied to the same dataset across random partitions?
• RQ2 How do widely used complex methods compare to elementary statistical methods in terms of inter-study consistency?
• RQ3 To what extent do methodological choices affect the reproducibility of microbial differential abundance findings?
• RQ4 Can simple statistical methods outperform sophisticated models in terms of result consistency in microbiome studies?

Key findings

• Elementary methods such as the Wilcoxon test and ordinal regression on relative abundances demonstrated high consistency across random partitions of the same dataset.
• Linear regression and t-tests on relative abundances also showed strong reproducibility, performing comparably to more complex models.
• Logistic regression on presence/absence data achieved a similar level of consistency, suggesting it is a robust alternative for binary data.
• Several widely used methods, including some specialized microbiome tools, produced a substantial number of conflicting findings across partitions and studies.
• The best overall performance was achieved by elementary methods when considering both consistency and sensitivity.
• The study found no evidence that complex, model-based methods consistently outperform basic statistical approaches in terms of reproducibility.