[Paper Review] Attention-Guided Generative Adversarial Network for Whisper to Normal Speech Conversion

Teng Gao, Jian Zhou+3 more|arXiv (Cornell University)|Nov 2, 2021

Speech and Audio Processing25 references4 citations

TL;DR

This paper proposes an attention-guided generative adversarial network (AGAN-W2SC) for whisper-to-normal speech conversion that bypasses the need for fundamental frequency estimation and dynamic time warping (DTW) alignment by using a self-attention mechanism for adaptive time alignment at the frame level. The model achieves superior speech quality and intelligibility, with a P.563 score of 1.872, outperforming DTW-based baselines like GMM, BLSTM, and Cycle-GAN.

ABSTRACT

Whispered speech is a special way of pronunciation without using vocal cord vibration. A whispered speech does not contain a fundamental frequency, and its energy is about 20dB lower than that of a normal speech. Converting a whispered speech into a normal speech can improve speech quality and intelligibility. In this paper, a novel attention-guided generative adversarial network model incorporating an autoencoder, a Siamese neural network, and an identity mapping loss function for whisper to normal speech conversion (AGAN-W2SC) is proposed. The proposed method avoids the challenge of estimating the fundamental frequency of the normal voiced speech converted from a whispered speech. Specifically, the proposed model is more amendable to practical applications because it does not need to align speech features for training. Experimental results demonstrate that the proposed AGAN-W2SC can obtain improved speech quality and intelligibility compared with dynamic-time-warping-based methods.

Motivation & Objective

• To address the challenge of converting whispered speech to normal speech without requiring explicit fundamental frequency (F0) estimation.
• To eliminate the need for time alignment via dynamic time warping (DTW) in training data by enabling adaptive, frame-level alignment through a self-attention mechanism.
• To improve speech quality and intelligibility in whisper-to-normal speech conversion by learning a robust mapping from whispered to normal speech spectra.
• To develop a flexible, end-to-end deep learning model that operates directly on mel-spectrograms without multi-parameter feature mapping or forced duration alignment.

Proposed method

• The model employs a GAN framework with a U-Net-like encoder-decoder generator and a patch-based discriminator for adversarial training.
• A self-attention module is integrated into the encoder to adaptively weight local spectral features and enable implicit time alignment across whisper and normal speech frames.
• An autoencoder component is used to preserve content information during the conversion process, enhancing reconstruction fidelity.
• A Siamese neural network is employed to compare embedded features of whispered and normal speech, aiding in discriminative representation learning.
• An identity mapping loss is applied to preserve the original speech content during the conversion process, reducing distortion.
• The generator is trained with a 3:1 ratio relative to the discriminator, using a learning rate between 0.0001 and 0.0002 to ensure stable convergence.

Experimental results

Research questions

• RQ1 Can a deep learning model convert whispered speech to normal speech without relying on explicit fundamental frequency estimation?
• RQ2 Can self-attention mechanisms effectively replace dynamic time warping for temporal alignment in whisper-to-normal speech conversion?
• RQ3 Does an end-to-end GAN-based approach with frame-level training outperform traditional DTW-aligned methods in speech quality and intelligibility?
• RQ4 Can the proposed model generate perceptually natural speech with competitive F0 contours despite the absence of vocal cord excitation in whispers?

Key findings

• The proposed AGAN-W2SC model achieves a P.563 score of 1.872, significantly outperforming baseline models such as GMM (1.163), BLSTM (1.247), and Cycle-GAN (1.107), indicating superior objective speech quality.
• The F0 contour of the converted speech generated by AGAN-W2SC closely matches the reference normal speech, demonstrating effective implicit F0 generation.
• The RMSE of the fundamental frequency between the converted and reference speech is comparable to or better than baseline methods, with the processed F0 RMSE showing improved robustness after silence frame removal.
• Subjective perception favors AGAN-W2SC, as it produces smoother and more natural-sounding speech compared to baseline models, despite not always achieving the lowest RMSE.
• The model operates at the frame level without requiring fixed-length input, making it more flexible and suitable for variable-duration whisper and normal speech pairs.