[Paper Review] BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

Risa Shinoda, Kaede Shiohara+4 more|arXiv (Cornell University)|Mar 25, 2026

Animal Vocal Communication and Behavior0 citations

TL;DR

BioVITA introduces a million-scale tri-modal dataset (audio, image, text), a unified representation model trained in two stages, and a comprehensive cross-modal benchmark for retrieval across six directions and three taxonomic levels, advancing visual-textual-acoustic alignment in biodiversity research.

ABSTRACT

Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/

Motivation & Objective

• Construct BioVITATrain: a million-scale training dataset of audio, images, and taxonomic text annotations for 14k species and 34 ecological traits.
• Develop BioVITAModel: a unified audio-image-text representation model trained with a two-stage framework to align audio with visual and textual modalities.
• Create BioVITABench: a species-level cross-modal retrieval benchmark spanning six directions and three taxonomic levels for comprehensive evaluation.

Proposed method

• Use HTS-AT as the audio encoder to produce 768-d embeddings from mel-spectrograms.
• Adopt pre-trained BioCLIP 2 image and text encoders (ViT-L/14 and 12-layer Transformer) to produce 768-d embeddings.
• Implement a two-stage training strategy: Stage 1 aligns audio-text via audio-text contrastive loss (ATC); Stage 2 jointly aligns audio, image, and text via ATC, AIC (audio-image), and ITC (image-text) losses.
• Stage 1: train only audio-text with a batch of audio-label pairs and random text prompts; Stage 2: train all three encoders with a weighted sum of contrastive losses, gradually increasing the weight of L_AIC and L_ITC.

Experimental results

Research questions

• RQ1 How well can a unified VITA (visual-textual-acoustic) embedding support cross-modal retrieval across images, text, and audio for biodiversity data?
• RQ2 Does a two-stage training approach improve cross-modal alignment more than training with all modalities from the start?
• RQ3 How does BioVITA generalize to unseen species and perform at different taxonomic levels (Species, Genus, Family)?
• RQ4 What is the impact of using scientific vs common names in text prompts on retrieval performance?

Key findings

• BioVITA (Stage 2) achieves strong species-level cross-modal retrieval, with average Top-1 and Top-5 accuracies of 71.7% and 89.2% across six directions.
• BioVITA Stage 1 already improves audio-text alignment, and Stage 2 further enhances all directions by incorporating visual cues.
• On the unseen species subset, BioVITA achieves average Top-1 and Top-5 of 51.9% and 73.0%, respectively, demonstrating robust generalization.
• Taxonomy-aware prompting and using scientific names yield higher retrieval accuracy than common names in several directions.
• High-level (Genus/Family) retrieval remains more challenging, but BioVITA shows hierarchical structure capture, with meaningful genus/family-level consistency in misclassifications.
• Trait prediction results indicate ecological traits are better predicted in the audio modality for behavioral traits like migration and trohabitat.