[Paper Review] Visual Display and Retrieval of Music Information

Rafael Valle|arXiv (Cornell University)|Jul 26, 2018

Music and Audio Processing21 references3 citations

TL;DR

This paper presents computational methods for visualizing and retrieving music information using audio descriptors like spectrograms, MFCCs, CQT, and chromagrams, combined with data visualization techniques such as self-similarity matrices and t-SNE projections. It demonstrates how these tools enable systematic, quantitative analysis of musical structure, timbre, and similarity, supporting both musicological research and industrial applications in music information retrieval.

ABSTRACT

This paper describes computational methods for the visual display and analysis of music information. We provide a concise description of software, music descriptors and data visualization techniques commonly used in music information retrieval. Finally, we provide use cases where the described software, descriptors and visualizations are showcased.

Motivation & Objective

• To develop and demonstrate computational methods for visualizing and retrieving music information from audio and symbolic data.
• To bridge the gap between musicologists and computational musicology by providing accessible tools and descriptors.
• To showcase how visualization enhances annotation quality and reduces human labor in music analysis.
• To enable systematic, quantitative analysis of musical structure, timbre, and similarity through feature extraction and dimensionality reduction.
• To illustrate practical applications of MIR techniques in music research and industry, such as in Spotify and Gracenote.

Proposed method

• Uses Short-Time Fourier Transform (STFT) and Mel-Frequency Cepstral Coefficients (MFCC) to extract timbral features from audio signals.
• Applies Constant Q Transform (CQT) and chroma features to represent pitch content with octave-invariant pitch class representation.
• Employs self-similarity matrices to detect structural patterns in music by measuring frame-wise similarity using distance functions.
• Utilizes dimensionality reduction techniques like t-SNE and MDS to project high-dimensional feature frames into 2D visual spaces for similarity analysis.
• Visualizes feature projections in RGB color space by mapping CQT features to color channels for temporal and perceptual interpretation.
• Employs feature-based distance measures such as Euclidean distance and correlation to quantify similarity between musical entities.

Experimental results

Research questions

• RQ1 How can audio descriptors like MFCC, CQT, and chroma be used to represent and analyze musical timbre and pitch structure?
• RQ2 What role do visualizations such as spectrograms, self-similarity matrices, and t-SNE projections play in enhancing music information retrieval?
• RQ3 How does data visualization improve the accuracy and efficiency of music annotation in crowdsourcing contexts?
• RQ4 To what extent can computational methods quantify musical similarity in ways that align with human perception?
• RQ5 How can dimensionality reduction techniques like t-SNE reveal stylistic groupings in large music datasets?

Key findings

• Spectrogram visualizations significantly improve annotator agreement and reduce time and labor in music annotation tasks.
• MFCC and CQT features effectively capture perceptually relevant aspects of timbre and pitch, mimicking human auditory perception.
• Self-similarity matrices computed from beat-aligned CQT features successfully reveal structural patterns in complex musical works like Grisey’s Partiels.
• t-SNE projections of feature frames from thousands of songs clearly separate musical styles such as Hardcore Punk and Ambient, with inter-point distances reflecting perceptual similarity.
• Color mapping of CQT features onto 3D RGB space enables intuitive visual interpretation of temporal musical evolution and feature similarity.
• Distance measures such as correlation outperform Euclidean distance in capturing perceptual similarity when comparing pitch profiles with varying dynamics.