[Paper Review] Speech-based Age and Gender Prediction with Transformers

Felix Burkhardt, Johannes Wagner+3 more|arXiv (Cornell University)|Jun 29, 2023

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a speech-based age and gender prediction system using fine-tuned wav2vec 2.0 transformers on curated datasets, achieving 7.1–10.8 MAE for age and ≥91.1% accuracy for gender. It outperforms traditional hand-crafted feature models by 9% UAR for age and 4% for gender, and releases the best model and data splits for reproducibility.

ABSTRACT

We report on the curation of several publicly available datasets for age and gender prediction. Furthermore, we present experiments to predict age and gender with models based on a pre-trained wav2vec 2.0. Depending on the dataset, we achieve an MAE between 7.1 years and 10.8 years for age, and at least 91.1% ACC for gender (female, male, child). Compared to a modelling approach built on handcrafted features, our proposed system shows an improvement of 9% UAR for age and 4% UAR for gender. To make our findings reproducible, we release the best performing model to the community as well as the sample lists of the data splits.

Motivation & Objective

• To address the lack of standardized, publicly available datasets for age and gender prediction in speech by curating and releasing curated train/dev/test splits.
• To improve performance in speech-based age and gender prediction using self-supervised pre-trained transformer models (wav2vec 2.0) over traditional hand-crafted feature approaches.
• To investigate the generalization of models across domains (in-domain vs. cross-corpus), model architecture (multi-head vs. single-head), and depth (number of transformer layers).
• To provide a reproducible, open-source baseline model and data splits for future research in speech-based age and gender estimation.

Proposed method

• Fine-tuned a pre-trained wav2vec 2.0 model on four publicly available datasets: aGender, CommonVoice, VoxCeleb2, and Timit, after curating and standardizing their age and gender labels.
• Used a multi-task learning setup with shared encoder and separate heads for age (regression) and gender (classification), enabling joint prediction.
• Evaluated both single-head and multi-head architectures, comparing performance across in-domain and cross-corpus settings.
• Systematically varied the number of transformer layers (from 1 to 24) to determine optimal trade-off between accuracy and inference speed.
• Applied age label mapping to discrete classes (child, youth, adult, senior) for comparison with the 2010 Paralinguistic Challenge baseline.
• Released the best-performing model and data splits to the community via a GitHub repository to ensure reproducibility.

Figure 1: Distribution of speaker age (#samples) in the datasets for the three splits ( CommonVoice age in mid-decades).

Experimental results

Research questions

• RQ1 How does a fine-tuned wav2vec 2.0 transformer model compare to traditional hand-crafted feature systems in age and gender prediction?
• RQ2 What is the optimal number of transformer layers for age and gender prediction, balancing accuracy and inference efficiency?
• RQ3 How does model performance vary between in-domain and cross-corpus settings across different datasets?
• RQ4 To what extent does emotional speech degrade the performance of a model trained on neutral speech data?
• RQ5 Can a unified model jointly predict age and gender more effectively than separate models, and how does it compare to task-specific baselines?

Key findings

• The proposed wav2vec 2.0-based model achieves a mean absolute error (MAE) of 7.1 to 10.8 years for age prediction across different datasets, significantly outperforming the 2010 Paralinguistic Challenge baseline.
• For gender classification (female, male, child), the model achieves at least 91.1% accuracy, with 96.04% UAR on neutral samples from the Berlin emotional database.
• The model improves UAR by 9 percentage points for age and 4 percentage points for gender compared to the baseline system using hand-crafted features.
• Using six transformer layers provides the best trade-off between performance and inference speed, reducing parameters by a factor of 3.5 and inference time by a factor of 3.
• Performance degrades on emotional speech, with MAE rising to 8.35 (vs. 5.94 on neutral samples), indicating sensitivity to emotional prosody.
• The model trained on all four datasets (aGender, CommonVoice, VoxCeleb2, Timit) shows a small but consistent performance boost over models trained on individual datasets.

Figure 2: Proposed architecture built on wav2vec 2.0.