[Paper Review] Investigating Multi-Feature Selection and Ensembling for Audio Classification

Muhammad Turab, Teerath Kumar+2 more|arXiv (Cornell University)|Jun 15, 2022

Music and Audio Processing4 citations

TL;DR

This paper investigates multi-feature ensembling for audio classification by combining Mel Spectrogram, Mel Frequency Cepstral Coefficients (MFCC), and Zero Crossing Rate (ZCR) across diverse deep learning models (CNN, EfficientNet, MobileNet) on three speech digit datasets. Results show that ensembling only high-performing individual features—primarily Mel Spectrogram and MFCC—yields superior accuracy and efficiency, outperforming both single features and full feature ensembles.

ABSTRACT

Deep Learning (DL) algorithms have shown impressive performance in diverse domains. Among them, audio has attracted many researchers over the last couple of decades due to some interesting patterns--particularly in classification of audio data. For better performance of audio classification, feature selection and combination play a key role as they have the potential to make or break the performance of any DL model. To investigate this role, we conduct an extensive evaluation of the performance of several cutting-edge DL models (i.e., Convolutional Neural Network, EfficientNet, MobileNet, Supper Vector Machine and Multi-Perceptron) with various state-of-the-art audio features (i.e., Mel Spectrogram, Mel Frequency Cepstral Coefficients, and Zero Crossing Rate) either independently or as a combination (i.e., through ensembling) on three different datasets (i.e., Free Spoken Digits Dataset, Audio Urdu Digits Dataset, and Audio Gujarati Digits Dataset). Overall, results suggest feature selection depends on both the dataset and the model. However, feature combinations should be restricted to the only features that already achieve good performances when used individually (i.e., mostly Mel Spectrogram, Mel Frequency Cepstral Coefficients). Such feature combination/ensembling enabled us to outperform the previous state-of-the-art results irrespective of our choice of DL model.

Motivation & Objective

• To investigate how different combinations of audio features impact deep learning model performance in audio classification.
• To evaluate the effectiveness of ensembling multiple audio features (Mel Spectrogram, MFCC, ZCR) across diverse deep learning architectures.
• To determine whether feature ensembling improves classification accuracy and inference efficiency compared to individual features.
• To identify optimal feature combinations that generalize across different datasets and models, particularly for speech digit classification.
• To release code and trained models to support future research in audio feature ensembling and deep learning for audio.

Proposed method

• Extracted three state-of-the-art audio features: Mel Spectrogram (MS), Mel Frequency Cepstral Coefficients (MFCC), and Zero Crossing Rate (ZCR) from raw audio signals.
• Trained and evaluated three deep learning models—Convolutional Neural Network (CNN), EfficientNetB0, and MobileNetV1—on each feature individually.
• Implemented feature ensembling by concatenating feature maps from multiple features (e.g., MS + MFCC, MS + MFCC + ZCR) as input to the same models.
• Used standard training and validation protocols across three benchmark datasets: Free Spoken Digits Dataset (FSD), Audio Urdu Digits Dataset (AUDD), and Audio Gujarati Digits Dataset (AGDD).
• Measured performance using accuracy (mean ± std) and inference time (ms) to assess both effectiveness and efficiency.
• Conducted ablation studies across all feature combinations and models to identify optimal configurations, with results visualized via validation accuracy curves over epochs.

Experimental results

Research questions

• RQ1 Which individual audio features (MS, MFCC, ZCR) yield the highest classification accuracy across different deep learning models and datasets?
• RQ2 Does ensembling multiple audio features consistently improve model performance, or can it degrade results?
• RQ3 Are there specific feature combinations that generalize well across different models and datasets in audio classification?
• RQ4 How does feature ensembling affect inference time and model efficiency?
• RQ5 Can a systematic approach to feature selection and ensembling outperform state-of-the-art results in audio digit classification?

Key findings

• Mel Spectrogram (MS) and Mel Frequency Cepstral Coefficients (MFCC) individually achieved the highest accuracy across all models and datasets, with MS consistently outperforming or matching MFCC.
• Zero Crossing Rate (ZCR) significantly degraded performance across all models, with accuracy dropping to as low as 0.321 for EfficientNetB0 on the Gujarati dataset.
• Ensembling all three features (MS, MFCC, ZCR) consistently underperformed compared to ensembling only MS and MFCC, indicating that adding low-performing features harms overall performance.
• The best-performing configuration was ensembling MS and MFCC, which achieved 0.987 ± 0.05 accuracy on the Free Spoken Digits Dataset using MobileNetV1, outperforming prior state-of-the-art results.
• MobileNetV1 with MS and MFCC combination achieved 0.987 ± 0.05 accuracy in 0.63 ms inference time, demonstrating high efficiency.
• Validation curves showed stable training dynamics for MS and MFCC, while ZCR caused instability, especially in larger models like EfficientNet.