[Paper Review] BioCLIP: A Vision Foundation Model for the Tree of Life

Samuel Stevens, Jiaman Wu+10 more|arXiv (Cornell University)|Nov 30, 2023

Cell Image Analysis Techniques9 citations

TL;DR

The paper introduces TreeOfLife-10M, the largest ML-ready biology image dataset, and BioCLIP, a vision foundation model trained with CLIP-style multimodal contrastive learning leveraging taxonomic hierarchy to enable zero-shot and few-shot classification across the tree of life.

ABSTRACT

Images of the natural world, collected by a variety of cameras, from drones to individual phones, are increasingly abundant sources of biological information. There is an explosion of computational methods and tools, particularly computer vision, for extracting biologically relevant information from images for science and conservation. Yet most of these are bespoke approaches designed for a specific task and are not easily adaptable or extendable to new questions, contexts, and datasets. A vision model for general organismal biology questions on images is of timely need. To approach this, we curate and release TreeOfLife-10M, the largest and most diverse ML-ready dataset of biology images. We then develop BioCLIP, a foundation model for the tree of life, leveraging the unique properties of biology captured by TreeOfLife-10M, namely the abundance and variety of images of plants, animals, and fungi, together with the availability of rich structured biological knowledge. We rigorously benchmark our approach on diverse fine-grained biology classification tasks and find that BioCLIP consistently and substantially outperforms existing baselines (by 16% to 17% absolute). Intrinsic evaluation reveals that BioCLIP has learned a hierarchical representation conforming to the tree of life, shedding light on its strong generalizability. https://imageomics.github.io/bioclip has models, data and code.

Motivation & Objective

• Create a large-scale, diverse biology image dataset with taxonomic labels to support foundation-model pre-training (TreeOfLife-10M).
• Develop a vision foundation model (BioCLIP) that utilizes taxonomic structure to improve generalization to unseen taxa.
• Demonstrate strong zero-shot and few-shot performance across diverse fine-grained biology classification tasks.
• Investigate how text type (taxonomic, scientific, common names) affects model generalization.
• Provide an intrinsic analysis showing the learned hierarchical representations align with the tree of life.

Proposed method

• Curate TreeOfLife-10M by merging iNat21, Bioscan-1M, and Encyclopedia of Life images with standardized taxonomic hierarchies.
• Initialize from OpenAI CLIP weights and continue pre-training on TreeOfLife-10M using CLIP’s multimodal contrastive objective.
• Represent taxonomic labels as taxonomic names (flattened hierarchy) and train the model to match images with these names in a CLIP framework.
• Experiment with mixed text types (taxonomic, scientific, common names) to improve inference-time flexibility.
• Evaluate zero-shot and few-shot performance on 10 diverse fine-grained datasets spanning animals, plants, and fungi, including a Rare Species test set.
• Compare BioCLIP to CLIP and OpenCLIP trained on LAION-400M, and ablate data sources and text-type strategies.

Experimental results

Research questions

• RQ1 Can a vision foundation model trained on TreeOfLife-10M generalize to taxa not present in training data (zero-shot) across the tree of life?
• RQ2 Does encoding taxonomic structure in the label space via CLIP-like objective improve fine-grained biology classification, especially in low-data regimes?
• RQ3 How do different text types (taxonomic, scientific, common names) used during training affect zero-shot and few-shot generalization?
• RQ4 What is the impact of data diversity (TreeOfLife-10M vs iNat21) on downstream performance and generalization to unseen taxa?
• RQ5 Does BioCLIP learn hierarchical representations that reflect the tree of life, and how does this manifest in intrinsic analyses?

Key findings

• BioCLIP consistently outperforms baselines (by 17–20 percentage points in zero-shot) across 10 fine-grained biology datasets.
• BioCLIP achieves strong few-shot gains, with notable improvements over CLIP and OpenCLIP in 1-shot and 5-shot settings.
• Zero-shot performance is especially strong on unseen Rare Species, indicating good generalization to taxa not in training data.
• An intrinsic analysis shows BioCLIP learns a hierarchical feature structure that aligns with the tree of life, explaining improved generalization.
• Using taxonomic names in training significantly improves zero-shot accuracy over scientific names alone, and mixed text types enhance inference-time flexibility.
• TreeOfLife-10M diversity, including data from EOL, substantially boosts performance compared to using iNat21 alone.

(d) Onoclea sensibilis