[Paper Review] Time-Frequency Audio Features for Speech-Music Classification

Mrinmoy Bhattacharjee, S. R. Mahadeva Prasanna+1 more|arXiv (Cornell University)|Nov 3, 2018

Music and Audio Processing4 citations

TL;DR

This paper proposes a novel two-stage time-frequency feature extraction method for speech-music classification, using spectral peak sequences (SPS) derived from prominent frequency peaks in audio frames. The SPS are treated as time series to extract periodicity, zero-crossing rate, and statistical features (SPS-SCG), which outperform baseline methods on four datasets, especially with SVM classifiers.

ABSTRACT

Distinct striation patterns are observed in the spectrograms of speech and music. This motivated us to propose three novel time-frequency features for speech-music classification. These features are extracted in two stages. First, a preset number of prominent spectral peak locations are identified from the spectra of each frame. These important peak locations obtained from each frame are used to form Spectral peak sequences (SPS) for an audio interval. In second stage, these SPS are treated as time series data of frequency locations. The proposed features are extracted as periodicity, average frequency and statistical attributes of these spectral peak sequences. Speech-music categorization is performed by learning binary classifiers on these features. We have experimented with Gaussian mixture models, support vector machine and random forest classifiers. Our proposal is validated on four datasets and benchmarked against three baseline approaches. Experimental results establish the validity of our proposal.

Motivation & Objective

• To address the challenge of distinguishing speech and music in audio segmentation by leveraging joint temporal-spectral characteristics.
• To exploit the distinct striation patterns in spectrograms—smooth arcs in speech and horizontal lines in music—through a feature extraction framework.
• To develop time-frequency features that capture the dynamic evolution of prominent spectral peaks across frames.
• To benchmark the proposed features against established baselines across diverse audio datasets.
• To demonstrate the effectiveness of statistical and periodicity features derived from spectral peak sequences (SPS) for improved classification performance.

Proposed method

• Detect p=20 prominent spectral peaks per audio frame using magnitude spectrum analysis.
• Construct Spectral Peak Sequences (SPS) by treating the frequency locations of these peaks across frames as time series.
• Extract three new features per SPS: periodicity (SPS-P), zero-crossing rate (SPS-ZCR), and a composite of standard deviation, centroid, and gradient (SPS-SCG).
• Use SPS-SCG as the primary feature set due to superior performance, combining statistical and temporal dynamics of peak sequences.
• Train and evaluate binary classifiers (GMM, SVM, Random Forest) on the extracted features using a 70:30 training-test split.
• Optimize SVM hyperparameters via grid search and repeat experiments 20 times to report mean F-scores with variances.

Experimental results

Research questions

• RQ1 Can spectral peak sequences derived from time-frequency representations improve speech-music classification beyond traditional spectral and temporal features?
• RQ2 How do periodicity and statistical attributes of spectral peak sequences differ between speech and music signals?
• RQ3 Does the proposed SPS-SCG feature set outperform established baselines such as MFCCs, Khonglah-FS, and Sell-FS in diverse audio environments?
• RQ4 Is the performance of the proposed method robust across varied datasets, including real-world broadcast and movie audio?
• RQ5 Can the two-stage feature extraction framework be generalized to other audio classification tasks beyond speech-music discrimination?

Key findings

• The SPS-SCG feature set achieved the best performance on three out of four datasets—GTZAN, Scheirer-Slaney, and TV News Broadcast—when used with SVM classifiers.
• On the Movie dataset, SPS-SCG achieved second-best performance, indicating strong generalization across diverse audio content.
• SPS-SCG significantly outperformed the best baseline (Khonglah-FS) on the GTZAN dataset, demonstrating its discriminative power.
• The proposed method showed consistent improvement over baseline approaches across all four datasets, validating its effectiveness in capturing time-frequency dynamics.
• No significant gain was observed from early or late feature fusion, indicating that SPS-SCG alone is a robust and sufficient feature representation.
• The use of spectral peak sequences as time series effectively captures the distinct temporal evolution of pitch and harmonic structures in speech and music.