[Paper Review] Perceptual Evaluation on Audio-visual Dataset of 360 Content

Randy Frans Fela, Andréas Pastor+4 more|arXiv (Cornell University)|May 16, 2022

Image and Video Quality Assessment4 citations

TL;DR

This paper presents a novel open-source 360° audiovisual dataset featuring 8K equirectangular video and 4th-order ambisonic audio, along with subjective quality scores from audio, video, and audiovisual tests. The dataset enables robust evaluation of perceptual quality, with results showing VMAF 4K and ViSQOL aswb outperforming other objective metrics in correlation with subjective scores.

ABSTRACT

To open up new possibilities to assess the multimodal perceptual quality of omnidirectional media formats, we proposed a novel open source 360 audiovisual (AV) quality dataset. The dataset consists of high-quality 360 video clips in equirectangular (ERP) format and higher-order ambisonic (4th order) along with the subjective scores. Three subjective quality experiments were conducted for audio, video, and AV with the procedures detailed in this paper. Using the data from subjective tests, we demonstrated that this dataset can be used to quantify perceived audio, video, and audiovisual quality. The diversity and discriminability of subjective scores were also analyzed. Finally, we investigated how our dataset correlates with various objective quality metrics of audio and video. Evidence from the results of this study implies that the proposed dataset can benefit future studies on multimodal quality evaluation of 360 content.

Motivation & Objective

• To address the lack of high-quality, multimodal, subjective quality datasets for immersive 360° audiovisual content.
• To establish a benchmark dataset with controlled, high-fidelity recordings of 360° video and higher-order ambisonic audio.
• To evaluate the discriminability and reliability of subjective quality scores across audio, video, and audiovisual modalities.
• To assess the performance of existing objective quality metrics in predicting human perception on 360° AV content.
• To provide a foundation for future development of multimodal, perceptually accurate quality metrics for immersive media.

Proposed method

• Acquired 12 high-quality 360° video and audio scenes using professional equipment: Insta360 Pro2 for 8K equirectangular video (7680×3840, 30fps) and em32 Eigenmike for 4th-order ambisonic (B-format, 25 channels) audio.
• Collected subjective mean opinion scores (MOS) through three separate experiments: audio-only, video-only, and audiovisual, using MUSHRA methodology.
• Prepared processed video sequences by downsampling to 6144×3072 and encoding with H.265/HEVC at various bitrates and QP values.
• Generated audio sequences by downmixing from raw 32-channel A-format to 4th-order B-format (AmbiX), maintaining 24-bit, 48kHz, 1152 kbps per channel.
• Evaluated 12 audio and 12 video objective quality metrics (e.g., ViSQOL, VMAF, PEAQ, PSNR) against subjective DMOS scores using PLCC, SRCC, KRCC, and AUC metrics.
• Conducted statistical analysis on subjective scores, including overall mean-CI, SOS (Sensitivity of Subjective Scores), and assessors' number impact on accuracy.

Experimental results

Research questions

• RQ1 How do different encoding parameters affect perceived audio and video quality in 360° AV content?
• RQ2 What is the discriminability and reliability of subjective quality scores across audio, video, and audiovisual modalities?
• RQ3 How does the number of assessors influence the accuracy and stability of subjective quality scores?
• RQ4 Which objective quality metrics show the strongest correlation with subjective mean opinion scores for 360° audio and video?
• RQ5 To what extent does the proposed dataset support the development of new multimodal perceptual quality models for immersive content?

Key findings

• The proposed dataset demonstrates strong discriminability, with video experiments showing the highest score variance and audiovisual experiments showing the most consistent subjective ratings.
• Subjective scores achieved low confidence intervals (CI) and stable trends after 12 assessors, supporting the use of 20 assessors per task for reliable results.
• The audio portion of the dataset achieved an alpha value comparable to other MUSHRA-based studies, indicating good reliability in perceptual evaluation.
• Among audio metrics, ViSQOL aswb achieved the highest PLCC (0.924), SRCC (0.938), and AUC (0.995), outperforming PEAQ and ViSQOL wb.
• For video, VMAF 4K achieved the best performance with PLCC of 0.919, SRCC of 0.957, and AUC of 0.989, significantly outperforming PSNR, SSIM, and VMAF HD.
• Metrics like S-PSNR, WS-PSNR, and CPP-PSNR showed no perceptual advantage over their 2D counterparts when applied to 360° video, indicating limited relevance for equirectangular content.