[Paper Review] Multiclass Language Identification using Deep Learning on Spectral Images of Audio Signals

Shauna Revay, Matthew Teschke|arXiv (Cornell University)|May 10, 2019

Speech Recognition and Synthesis7 references38 citations

TL;DR

This paper proposes LIFAS, a CNN-based approach that uses on-the-fly mel-spectrograms of raw audio to identify language, achieving 97% binary accuracy and 89% multiclass accuracy across six languages using short 4-second clips.

ABSTRACT

The first step in any voice recognition software is to determine what language a speaker is using, and ideally this process would be automated. The technique described in this paper, language identification for audio spectrograms (LIFAS), uses spectrograms generated from audio signals as inputs to a convolutional neural network (CNN) to be used for language identification. LIFAS requires minimal pre-processing on the audio signals as the spectrograms are generated during each batch as they are input to the network during training. LIFAS utilizes deep learning tools that are shown to be successful on image processing tasks and applies it to audio signal classification. LIFAS performs binary language classification with an accuracy of 97\%, and multi-class classification with six languages at an accuracy of 89\% on 3.75 second audio clips.

Motivation & Objective

• Motivate automatic language identification as a step for voice assistants and transcription systems.
• Propose an end-to-end CNN-based pipeline that operates on spectrograms generated from raw audio without external feature extraction.
• Demonstrate viability with short audio segments and open VoxForge data across six languages.

Proposed method

• Use spectrograms (mel-scale) generated on-the-fly from raw audio as CNN input.
• Adopt a pretrained ResNet-50 backbone with residual connections for robust feature learning.
• Train with 1-cycle learning rate policy to optimize convergence over 8 epochs.
• Leverage the fast.ai/PyTorch ecosystem for implementation and data handling.
• Evaluate both binary and multiclass language identification performance.

Experimental results

Research questions

• RQ1 Can a CNN operating on on-the-fly spectrogram representations of raw audio accurately identify language across multiple languages?
• RQ2 What is the impact of audio clip length on language identification accuracy for binary and multiclass tasks?
• RQ3 How does a ResNet-based architecture perform on spectrogram inputs for language recognition compared to prior feature-extraction approaches?

Key findings

• Binary language identification (English vs. Russian) achieves 97% accuracy with 60k-sample clips (3.75s).
• Binary identification with longer clips (100k samples) reaches 97% accuracy, indicating limited gain beyond 60k samples in this setup.
• Multiclass identification across six languages achieves 89% accuracy on 60k-sample clips.
• Binary results show robust performance across English and Russian, with little misclassification into other languages.
• The study uses VoxForge data with careful train/validation separation to avoid cross-contamination.