[Paper Review] OptimOTU: Taxonomically aware OTU clustering with optimized thresholds and a bioinformatics workflow for metabarcoding data

Brendan Furneaux, Sten Anslan+5 more|ArXiv.org|Feb 14, 2025

Environmental DNA in Biodiversity Studies3 citations

TL;DR

OptimOTU introduces a taxonomically aware OTU clustering algorithm that optimizes per-taxon thresholds and integrates a full Illumina metabarcoding workflow, enabling scalable processing of large, diverse datasets.

ABSTRACT

To turn environmentally derived metabarcoding data into community matrices for ecological analysis, sequences must first be clustered into operational taxonomic units (OTUs). This task is particularly complex for data including large numbers of taxa with incomplete reference libraries. OptimOTU offers a taxonomically aware approach to OTU clustering. It uses a set of taxonomically identified reference sequences to choose optimal genetic distance thresholds for grouping each ancestor taxon into clusters which most closely match its descendant taxa. Then, query sequences are clustered according to preliminary taxonomic identifications and the optimized thresholds for their ancestor taxon. The process follows the taxonomic hierarchy, resulting in a full taxonomic classification of all the query sequences into named taxonomic groups as well as placeholder "pseudotaxa" which accommodate the sequences that could not be classified to a named taxon at the corresponding rank. The OptimOTU clustering algorithm is implemented as an R package, with computationally intensive steps implemented in C++ for speed, and incorporating open-source libraries for pairwise sequence alignment. Distances may also be calculated externally, and may be read from a UNIX pipe, allowing clustering of large datasets where the full distance matrix would be inconveniently large to store in memory. The OptimOTU bioinformatics pipeline includes a full workflow for paired-end Illumina sequencing data that incorporates quality filtering, denoising, artifact removal, taxonomic classification, and OTU clustering with OptimOTU. The OptimOTU pipeline is developed for use on high performance computing clusters, and scales to datasets with millions of reads per sample, and tens of thousands of samples.

Motivation & Objective

• Motivate improved OTU clustering by accounting for taxon-specific genetic variation and incomplete reference libraries.
• Develop an algorithm that optimizes clustering thresholds per ancestor taxon to better match taxonomy.
• Provide a full, scalable pipeline from raw reads to taxonomically informed OTUs and placeholder pseudotaxa.
• Enable integration with existing taxonomic identification tools and open-source distance measures for efficiency.

Proposed method

• Introduce OptimOTU clustering algorithm with three phases: threshold optimization, preliminary taxonomic identification, and hierarchical clustering.
• Optimize thresholds by comparing taxonomic partitions across ranks using AMI (adjusted mutual information) to determine rank-specific cut thresholds.
• Cluster queries using a taxonomically guided hierarchy with closed-reference and de novo steps, producing named taxa and pseudotaxa.
• Implement distance calculations via multiple internal methods (Hamming, Edlib, WFA2) or external distance matrices, with optional USEARCH integration for performance.
• Provide a tree-based clustering algorithm as default, with parallelization strategies including concurrent, merge, and hierarchical modes.
• Wrap the clustering into an end-to-end OptimOTU pipeline that processes paired-end Illumina data, including quality filtering, denoising, chimera removal, and taxonomically guided clustering.

Experimental results

Research questions

• RQ1 Can taxonomically informed, per-rank threshold optimization improve OTU clustering accuracy over single-threshold methods across diverse taxonomic groups?
• RQ2 How does incorporating preliminary taxonomic identifications influence clustering efficiency and accuracy in datasets with incomplete reference libraries?
• RQ3 What are the performance and scalability characteristics of OptimOTU on large metabarcoding datasets (millions of reads, many samples) compared to traditional workflows?
• RQ4 How do different distance calculation and clustering configurations affect the resulting OTU partitions and downstream ecological analyses?

Key findings

• OptimOTU yields taxonomically guided clustering by using optimized thresholds per ancestor taxon, improving concordance with taxonomy across ranks.
• The pipeline integrates quality filtering, denoising, artifact removal, and hierarchical clustering, producing named taxa and pseudotaxa when needed.
• The clustering supports large-scale datasets and enables processing on HPC clusters, with a tree-based algorithm as the default and multiple parallelization strategies.
• Distances can be computed internally or read from external sources, including a USEARCH-based option for speed on complex markers like ITS.
• The workflow provides a full Illumina paired-end pipeline from raw reads to OTU-level taxonomic assignments, suitable for fungal ITS2 and metazoan COI analyses.