[Paper Review] Soundscape Captioning using Sound Affective Quality Network and Large Language Model

Yuanbo Hou, Qiaoqiao Ren+5 more|arXiv (Cornell University)|Jun 9, 2024

Noise Effects and Management4 citations

TL;DR

This paper introduces Soundscape Captioning (SoundSCap), an automated task that generates context-aware descriptions of audio scenes by integrating acoustic scene/event recognition with perceived affective qualities. The proposed SoundSCaper model combines a multi-scale acoustic model (SoundAQnet) and a large language model (LLM) to generate captions that match expert-level quality, with no statistically significant difference in human evaluation scores (mean difference of 0.21–0.25 out of 5).

ABSTRACT

We live in a rich and varied acoustic world, which is experienced by individuals or communities as a soundscape. Computational auditory scene analysis, disentangling acoustic scenes by detecting and classifying events, focuses on objective attributes of sounds, such as their category and temporal characteristics, ignoring their effects on people, such as the emotions they evoke within a context. To fill this gap, we propose the affective soundscape captioning (ASSC) task, which enables automated soundscape analysis, thus avoiding labour-intensive subjective ratings and surveys in conventional methods. With soundscape captioning, context-aware descriptions are generated for soundscape by capturing the acoustic scenes (ASs), audio events (AEs) information, and the corresponding human affective qualities (AQs). To this end, we propose an automatic soundscape captioner (SoundSCaper) system composed of an acoustic model, i.e. SoundAQnet, and a large language model (LLM). SoundAQnet simultaneously models multi-scale information about ASs, AEs, and perceived AQs, while the LLM describes the soundscape with captions by parsing the information captured with SoundAQnet. SoundSCaper is assessed by two juries of 32 people. In expert evaluation, the average score of SoundSCaper-generated captions is slightly lower than that of two soundscape experts on the evaluation set D1 and the external mixed dataset D2, but not statistically significant. In layperson evaluation, SoundSCaper outperforms soundscape experts in several metrics. In addition to human evaluation, compared to other automated audio captioning systems with and without LLM, SoundSCaper performs better on the ASSC task in several NLP-based metrics. Overall, SoundSCaper performs well in human subjective evaluation and various objective captioning metrics, and the generated captions are comparable to those annotated by soundscape experts.

Motivation & Objective

• To automate the labor-intensive process of subjective soundscape assessment traditionally reliant on surveys and expert ratings.
• To bridge the gap between objective audio event detection and subjective human affective responses in acoustic environments.
• To develop a unified framework that captures acoustic scenes, events, and perceived affective qualities (PAQ) for natural language captioning.
• To evaluate the performance of an automated captioning system against human experts using a jury of 16 soundscape experts.
• To enable scalable, context-aware, and emotionally informed descriptions of complex audio environments using deep learning and LLMs.

Proposed method

• Proposes SoundAQnet, a lightweight, multi-scale deep neural network that jointly models acoustic scenes (AS), audio events (AE), and perceived affective qualities (PAQ) from raw audio.
• Trained on a large-scale dataset with participant ratings to learn perceptual attributes such as pleasantness, eventfulness, and calmness using the 8D soundscape circumplex model (SCM).
• Uses multiscale representations to capture both short-term salient events (e.g., sirens) and long-term environmental context (e.g., urban vs. forest).
• Integrates SoundAQnet’s output with a general-purpose large language model (LLM) to generate natural language captions from three perspectives: AS, AE, and PAQ.
• Employs a zero-shot or few-shot prompting strategy to guide the LLM in synthesizing coherent, context-aware descriptions from the model’s latent representations.
• Validates the system using a jury-based human evaluation on both held-out test data and an external mixed dataset with diverse audio lengths and acoustic properties.

Experimental results

Research questions

• RQ1 Can an automated system generate context-aware soundscape captions that match the quality of expert-annotated descriptions?
• RQ2 How well does the integration of acoustic scene/event recognition with perceived affective qualities (PAQ) improve captioning performance compared to traditional AS/AE classification?
• RQ3 To what extent does the model generalize across diverse audio clips with varying lengths and acoustic characteristics?
• RQ4 How do the affective quality predictions of SoundAQnet compare to individual expert perceptions in terms of subjectivity and consistency?
• RQ5 Is the performance gap between SoundSCaper-generated captions and expert-annotated captions statistically significant?

Key findings

• SoundSCaper-generated captions achieved an average human evaluation score of 4.29 out of 5 on the test set, compared to 4.50 for expert-annotated captions, a difference of 0.21 points.
• On an external mixed dataset with 5 model-unknown datasets, the average score for SoundSCaper was 4.25, compared to 4.50 for expert-annotated captions, a difference of 0.25 points.
• The performance gap between SoundSCaper and expert-annotated captions was not statistically significant, indicating comparable quality in human perception.
• SoundSCaper excels in capturing dominant audio events and general acoustic scene characteristics but lacks fine-grained spatial and temporal details (e.g., direction of vehicles, presence of short sirens).
• SoundAQnet’s affective quality predictions are more moderate and general than individual expert responses, reducing personal bias but potentially losing nuanced emotional expression.
• The model demonstrates strong generalization across diverse audio clips with varying lengths and acoustic properties, confirming robustness in real-world conditions.