[Paper Review] Audio Captioning Transformer

Xinhao Mei, Xubo Liu+3 more|arXiv (Cornell University)|Jul 21, 2021

Music and Audio Processing23 references32 citations

TL;DR

This paper introduces Audio Captioning Transformer (ACT), a convolution-free Transformer encoder–decoder for audio captioning, pre-trained on AudioSet, and evaluated on AudioCaps with competitive results.

ABSTRACT

Audio captioning aims to automatically generate a natural language description of an audio clip. Most captioning models follow an encoder-decoder architecture, where the decoder predicts words based on the audio features extracted by the encoder. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are often used as the audio encoder. However, CNNs can be limited in modelling temporal relationships among the time frames in an audio signal, while RNNs can be limited in modelling the long-range dependencies among the time frames. In this paper, we propose an Audio Captioning Transformer (ACT), which is a full Transformer network based on an encoder-decoder architecture and is totally convolution-free. The proposed method has a better ability to model the global information within an audio signal as well as capture temporal relationships between audio events. We evaluate our model on AudioCaps, which is the largest audio captioning dataset publicly available. Our model shows competitive performance compared to other state-of-the-art approaches.

Motivation & Objective

• Advance audio captioning by applying a pure Transformer encoder–decoder without convolution.
• Model global and fine-grained temporal information in audio using self-attention over time patches.
• Improve generalization with pre-training on AudioSet as an audio tagging task and leverage DeiT initialization.
• Evaluate ACT against state-of-the-art methods on AudioCaps and analyze data efficiency and hyper-parameter effects.

Proposed method

• Represent log-mel spectrograms as non-overlapping time patches and embed them into a Transformer encoder with a class token for global audio information.
• Use a standard Transformer encoder with multi-head self-attention and feed-forward layers, including layer normalization and residual connections.
• In the decoder, employ masked self-attention and an additional cross-attention layer attending to encoder outputs, producing word predictions via a linear+softmax layer.
• Pre-train the encoder on AudioSet as an audio tagging task to learn generalized audio patterns, with a class token for tagging outputs.
• Initialize the decoder with Word2Vec embeddings and experiment with three decoder variants differing in depth and heads.
• Train end-to-end with cross-entropy loss and teacher-forcing, using a beam search (beam size up to 5) during inference.

Experimental results

Research questions

• RQ1 Does a convolution-free Transformer encoder–decoder (ACT) effectively capture global and temporal audio information for captioning?
• RQ2 How does pre-training the encoder on a large audio tagging dataset (AudioSet) affect captioning performance?
• RQ3 What is the impact of decoder depth and attention heads on caption quality and evaluation metrics?
• RQ4 How does ACT compare to CNN-based and Transformer-based baseline methods on AudioCaps in terms of accuracy and efficiency?

Key findings

• ACT achieves competitive performance on AudioCaps compared with state-of-the-art methods.
• Pre-training the encoder on AudioSet significantly boosts performance, with DeiT initialization alone providing substantial gains.
• Encoder pre-training is crucial for Transformer-based audio captioning; training from scratch underperforms CNN+Transformer without pre-training.
• ACT_m (4 decoder layers) achieves the best machine translation metrics, while ACT_l improves CIDEr and SPIDEr scores.
• The ACT model trains faster than CNN+Transformer (less than five minutes per epoch vs. seven minutes).