[Paper Review] Exploring Fine-Grained Audiovisual Categorization with the SSW60 Dataset

Grant Van Horn, Rui Qian+4 more|arXiv (Cornell University)|Jul 21, 2022

Music and Audio Processing5 citations

TL;DR

This paper introduces SSW60, a new expert-curated dataset of 60 bird species with paired images, audio, and video for fine-grained audiovisual categorization. Using state-of-the-art transformer-based fusion methods, the authors demonstrate that audiovisual fusion consistently outperforms unimodal models, achieving 80.6% top-1 accuracy, and reveal that pretraining on unpaired audio significantly boosts performance, while image pretraining can sometimes hurt performance.

ABSTRACT

We present a new benchmark dataset, Sapsucker Woods 60 (SSW60), for advancing research on audiovisual fine-grained categorization. While our community has made great strides in fine-grained visual categorization on images, the counterparts in audio and video fine-grained categorization are relatively unexplored. To encourage advancements in this space, we have carefully constructed the SSW60 dataset to enable researchers to experiment with classifying the same set of categories in three different modalities: images, audio, and video. The dataset covers 60 species of birds and is comprised of images from existing datasets, and brand new, expert-curated audio and video datasets. We thoroughly benchmark audiovisual classification performance and modality fusion experiments through the use of state-of-the-art transformer methods. Our findings show that performance of audiovisual fusion methods is better than using exclusively image or audio based methods for the task of video classification. We also present interesting modality transfer experiments, enabled by the unique construction of SSW60 to encompass three different modalities. We hope the SSW60 dataset and accompanying baselines spur research in this fascinating area.

Motivation & Objective

• To address the lack of high-quality, multi-modal datasets for fine-grained audiovisual categorization in natural categories like birds.
• To enable systematic study of cross-modal learning and fusion in fine-grained recognition across images, audio, and video.
• To benchmark the effectiveness of audiovisual fusion and pretraining strategies on a shared, expert-curated dataset of 60 bird species.
• To provide a foundation for future research in multi-modal fine-grained categorization, especially for biodiversity monitoring and real-time deployment.

Proposed method

• Construction of SSW60, a dataset with 5,400 ten-second video clips, 3,861 ten-second audio recordings, and image data from NABirds and iNaturalist for 60 bird species.
• Use of state-of-the-art vision and audio transformers (e.g., MBT) as backbones for visual and audio feature extraction.
• Application of multiple fusion strategies: late fusion, score fusion, and mid-fusion (MBT) to combine audio and visual predictions.
• Pretraining on external datasets (ImageNet, NABirds, iNaturalist, and unpaired audio) followed by fine-tuning on SSW60 video data to evaluate transfer learning effects.
• Evaluation of modality-specific and fused models using top-1 accuracy on the 60-class classification task.
• Per-class analysis of fusion performance to identify species where audiovisual fusion improves or degrades accuracy.

Experimental results

Research questions

• RQ1 How does audiovisual fusion compare to unimodal classification in fine-grained bird species recognition?
• RQ2 Which fusion strategy—late, score, or mid-fusion—yields the best performance on the SSW60 dataset?
• RQ3 How does pretraining on external image and audio datasets affect downstream performance on SSW60 video classification?
• RQ4 Which modality (audio or visual) is more effective for specific bird species, and when does fusion improve or hurt performance?
• RQ5 What insights can be drawn from modality transfer experiments using unpaired images and audio for video classification?

Key findings

• Audiovisual fusion consistently outperforms unimodal models, with the best fusion method achieving 80.6% top-1 accuracy, surpassing any single modality.
• Late fusion and score fusion outperform mid-fusion (MBT) in different pretraining configurations, indicating no single fusion method is universally optimal.
• Pretraining on unpaired audio recordings significantly improves performance, with a ~20% top-1 accuracy gain when fine-tuned on SSW60 audio channels.
• Pretraining on high-quality images (e.g., NABirds or iNaturalist) can sometimes degrade performance, suggesting potential domain shift or overfitting risks.
• Fusion improved performance for 27 species, maintained accuracy for 27, and reduced it for 6, indicating that fusion is not universally beneficial and depends on species-specific modality strengths.
• The results confirm that audio and visual modalities are complementary: species visually confusable but acoustically distinct (e.g., American Crow vs. Common Raven) benefit from audio, while those acoustically similar but visually distinct (e.g., Yellow Warbler vs. Chestnut-sided Warbler) benefit from vision.