[Paper Review] Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval

Yuma Koizumi, Yasunori Ohishi+3 more|arXiv (Cornell University)|Dec 14, 2020

Music and Audio Processing24 references21 citations

TL;DR

This paper proposes a novel audio captioning framework that leverages a pre-trained GPT-2 language model for caption generation by using audio-based retrieval of similar captions as guidance. By retrieving relevant captions from a training dataset based on audio similarity and conditioning GPT-2 on them, the method achieves superior performance over training from scratch, especially in n-gram accuracy, demonstrating the viability of adapting large-scale pre-trained language models to audio captioning despite modality mismatch.

ABSTRACT

The goal of audio captioning is to translate input audio into its description using natural language. One of the problems in audio captioning is the lack of training data due to the difficulty in collecting audio-caption pairs by crawling the web. In this study, to overcome this problem, we propose to use a pre-trained large-scale language model. Since an audio input cannot be directly inputted into such a language model, we utilize guidance captions retrieved from a training dataset based on similarities that may exist in different audio. Then, the caption of the audio input is generated by using a pre-trained language model while referring to the guidance captions. Experimental results show that (i) the proposed method has succeeded to use a pre-trained language model for audio captioning, and (ii) the oracle performance of the pre-trained model-based caption generator was clearly better than that of the conventional method trained from scratch.

Motivation & Objective

• To address the scarcity of audio-caption training data in audio captioning by leveraging large-scale pre-trained language models.
• To enable the use of powerful pre-trained language models like GPT-2 in audio captioning, despite their input modality mismatch (text-only).
• To improve caption generation quality by using retrieved, semantically relevant captions as conditioning context for the language model.
• To evaluate the effectiveness of pre-trained language models in cross-modal captioning tasks using a retrieval-augmented generation framework.

Proposed method

• The method employs a two-stage cascaded system: first, a retrieval module identifies the most similar training captions based on audio embeddings.
• Audio embeddings are learned using a contrastive learning objective with triplet loss to improve retrieval quality.
• The retrieved captions serve as conditioning context for a frozen GPT-2 language model, which generates the final caption autoregressively.
• The decoder generates each word by conditioning on both the retrieved captions and previously generated tokens, using the pre-trained language model's autoregressive prior.
• The system avoids direct audio input to GPT-2 by using retrieved captions as a bridge between audio input and text generation.
• The framework is trained end-to-end with only a few additional trainable layers on top of the frozen GPT-2 and embedding network.

Experimental results

Research questions

• RQ1 Can a pre-trained large-scale language model like GPT-2 be effectively adapted for audio captioning despite its text-only input design?
• RQ2 How can audio input be meaningfully conditioned into a pre-trained language model without direct audio input?
• RQ3 Does using retrieved, semantically similar captions as context improve the quality of generated audio captions?
• RQ4 How does the performance of a retrieval-augmented GPT-2 system compare to a conventional end-to-end model trained from scratch?

Key findings

• The proposed method successfully leverages a pre-trained GPT-2 model for audio captioning by using audio-based caption retrieval as conditioning, overcoming the modality mismatch.
• The oracle performance of the GPT-2-based generator significantly outperformed the conventional method trained from scratch, especially in BLEU-1 and BLEU-2 metrics.
• The method achieved near-human-level performance on most metrics, outperforming human captions on three of the evaluation metrics.
• The performance gap in BLEU-4 and CIDEr suggests a distribution shift between the web-trained GPT-2 and training captions, indicating room for improvement via fine-tuning.
• The retrieval module's accuracy strongly influences overall caption quality, as incorrect retrieval (e.g., 'vehicle' instead of 'bus') led to incorrect caption generation.
• The study confirms that audio-based guidance caption retrieval is the most promising component for future improvement, as it has the largest impact on final caption quality.