[Paper Review] Attention Bottlenecks for Multimodal Fusion

Arsha Nagrani, Shan Yang+4 more|arXiv (Cornell University)|Jun 30, 2021

Music and Audio Processing62 references264 citations

TL;DR

The paper introduces Multimodal Bottleneck Transformer (MBT) that restricts cross-modal fusion through bottleneck tokens, improving audio-visual video classification while reducing computation and achieving state-of-the-art results on AudioSet, Epic-Kitchens100, and VGGSound.

ABSTRACT

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

Motivation & Objective

• Motivate multimodal fusion beyond late fusion by enabling cross-modal interactions through structured bottlenecks.
• Propose MBT to constrain cross-modal information flow via fusion bottlenecks to reduce quadratic attention costs.
• Evaluate MBT across standard audio-visual video benchmarks and compare to state-of-the-art methods.
• Analyze the impact of fusion layer location (early/mid/late) and bottleneck size on performance and efficiency.

Proposed method

• Represent audio and visual inputs as patches and embed them into transformer tokens with separate modality tokens.
• Introduce fusion bottleneck tokens that force cross-modal information to pass through a small set of latent units within each layer.
• Allow either vanilla self-attention, cross-attention with modality-specific parameters, or bottleneck-based cross-modal attention, with bottlenecks restricting cross-modal flow inside a layer.
• Experiment with mid-fusion (fusion in later layers) rather than early or late fusion to balance unimodal feature learning and cross-modal interaction.
• Train and evaluate on AudioSet, Epic-Kitchens-100, and VGGSound using ViT-like backbones, with data augmentations and modality MixUp.

Experimental results

Research questions

• RQ1 Can restricting cross-modal attention to a small bottleneck improve multimodal fusion efficiency without sacrificing performance?
• RQ2 Does mid-fusion with bottlenecks outperform early or late fusion across audio-visual video benchmarks?
• RQ3 What is the impact of bottleneck size and fusion layer placement on accuracy and compute?
• RQ4 Do modality-specific weights and input sampling strategies affect multimodal fusion performance?

Key findings

• MBT with bottleneck fusion outperforms unrestricted cross-modal fusion while reducing computational cost.
• Mid-fusion with bottlenecks yields state-of-the-art performance on AudioSet, Epic-Kitchens-100, and VGGSound.
• On AudioSet, MBT outperforms the previous state of the art by 5.9 mAP (12.7% relative improvement).
• A small number of bottleneck tokens (B=4) provides substantial gains over late fusion with little extra compute.
• Attention maps show MBT localizes to semantically relevant regions tied to audio events, highlighting effective cross-modal conditioning.
• MBT maintains robust performance across varying clip spans and dataset sizes, outperforming single-modality baselines in all datasets.