[Paper Review] From Visual to Acoustic Question Answering

Jerome Abdelnour, Giampiero Salvi+1 more|arXiv (Cornell University)|Feb 28, 2019

Multimodal Machine Learning Applications35 references4 citations

TL;DR

This paper introduces Acoustic Question Answering (AQA), a novel task that requires models to reason about elementary sounds in audio scenes and answer relational questions about their positions and properties. Using a synthetic dataset of 50-second audio scenes composed of musical instrument sounds, the authors evaluate visual reasoning models (FiLM and MAC) adapted for audio, achieving up to 90.3% accuracy on a test set of 7,500 scenes and 300,000 questions, demonstrating the feasibility of acoustic reasoning with minimal architectural changes.

ABSTRACT

We introduce the new task of Acoustic Question Answering (AQA) to promote research in acoustic reasoning. The AQA task consists of analyzing an acoustic scene composed by a combination of elementary sounds and answering questions that relate the position and properties of these sounds. The kind of relational questions asked, require that the models perform non-trivial reasoning in order to answer correctly. Although similar problems have been extensively studied in the domain of visual reasoning, we are not aware of any previous studies addressing the problem in the acoustic domain. We propose a method for generating the acoustic scenes from elementary sounds and a number of relevant questions for each scene using templates. We also present preliminary results obtained with two models (FiLM and MAC) that have been shown to work for visual reasoning.

Motivation & Objective

• To propose and formalize the task of Acoustic Question Answering (AQA) as a new benchmark for audio reasoning.
• To generate a synthetic dataset of audio scenes with controlled, compositional questions to minimize bias and enable systematic evaluation.
• To adapt visual question answering models (FiLM and MAC) for audio input and evaluate their performance on acoustic reasoning tasks.
• To investigate the transferability of visual reasoning models to audio and identify challenges in generalizing to real-world audio scenarios.

Proposed method

• Audio scenes are synthesized by combining elementary sounds (e.g., musical instruments) with controlled timing, pitch, loudness, and brightness.
• Questions are generated using functional program templates that encode reasoning steps, ensuring semantic control and reducing dataset bias.
• Spectrograms of the audio scenes are computed with 1024-sample Hanning windows and 512-sample shifts, then resampled to 480×320 pixels for input to deep learning models.
• The FiLM and MAC neural networks are fine-tuned on the audio spectrogram data using cross-entropy loss and Adam optimization.
• Data is split into training (70%), validation (15%), and test (15%) sets with no overlap between sets to ensure evaluation integrity.
• Model performance is evaluated on accuracy across different question types, including relative position, absolute position, loudness, and brightness.

Experimental results

Research questions

• RQ1 Can visual reasoning models like FiLM and MAC be effectively adapted to perform reasoning on audio data without architectural modifications?
• RQ2 How does model performance on AQA vary with training data size and model complexity?
• RQ3 Which types of questions (e.g., relative position, loudness) are most challenging for audio reasoning models?
• RQ4 To what extent does the synthetic data generation method reduce bias and enable reliable evaluation of reasoning capabilities?

Key findings

• The FiLM model with 4 ResBlocks achieved the highest test accuracy of 90.3% when trained on the largest data set of 50,000 scenes.
• The MAC model with 8 MacCells achieved 94.8% training accuracy and 88.0% validation accuracy, but only 44.8% on the test set, indicating potential overfitting.
• Question types involving relative and absolute positions were the most difficult, with accuracies ranging from 25% to 45%, while loudness and brightness questions were easier, achieving 60–70% accuracy.
• Models trained on smaller data sets (e.g., 3,500 scenes) showed significant overfitting, with large gaps between training and test performance.
• The best-performing model (FiLM with 4 ResBlocks) achieved 97.9% training accuracy and 96.4% validation accuracy on the largest data set, indicating strong generalization.
• The results demonstrate that visual reasoning models can be effectively transferred to audio with minimal adaptation, achieving high accuracy on a novel acoustic reasoning benchmark.