[Paper Review] Vocal Tract Area Estimation by Gradient Descent

David Südholt, Mateo Cámara+2 more|arXiv (Cornell University)|Jul 10, 2023

Speech and Audio ProcessingComputer Science3 citations

TL;DR

This paper proposes a white-box, gradient-based optimization method to estimate vocal tract area functions and glottal source parameters from audio, enabling accurate sound matching in the Pink Trombone articulatory synthesizer. By backpropagating error gradients through a differentiable vocal tract model, the method outperforms black-box baselines like genetic algorithms and neural networks in subjective listening tests.

ABSTRACT

Articulatory features can provide interpretable and flexible controls for the synthesis of human vocalizations by allowing the user to directly modify parameters like vocal strain or lip position. To make this manipulation through resynthesis possible, we need to estimate the features that result in a desired vocalization directly from audio recordings. In this work, we propose a white-box optimization technique for estimating glottal source parameters and vocal tract shapes from audio recordings of human vowels. The approach is based on inverse filtering and optimizing the frequency response of a wave\-guide model of the vocal tract with gradient descent, propagating error gradients through the mapping of articulatory features to the vocal tract area function. We apply this method to the task of matching the sound of the Pink Trombone, an interactive articulatory synthesizer, to a given vocalization. We find that our method accurately recovers control functions for audio generated by the Pink Trombone itself. We then compare our technique against evolutionary optimization algorithms and a neural network trained to predict control parameters from audio. A subjective evaluation finds that our approach outperforms these black-box optimization baselines on the task of reproducing human vocalizations.

Motivation & Objective

• To develop a differentiable, white-box optimization approach for estimating articulatory parameters from audio recordings.
• To enable accurate sound matching of human vocalizations using the Pink Trombone articulatory synthesizer.
• To overcome limitations of black-box methods such as genetic algorithms and neural networks in terms of convergence speed and interpretability.
• To demonstrate that gradient-based optimization can effectively recover control parameters for time-varying vocalizations.
• To lay the groundwork for end-to-end differentiable articulatory synthesis using neural networks.

Proposed method

• The method uses inverse filtering to separate the glottal source waveform and the vocal tract's all-pole filter response from target audio.
• It formulates the vocal tract's transfer function analytically based on the area function and uses gradient descent to minimize the difference between this and the estimated filter.
• A differentiable implementation of the mapping from articulatory control parameters to the vocal tract area function enables backpropagation of error gradients to the parameters.
• The optimization is performed frame-by-frame, with initialization from the previous frame’s solution to ensure temporal continuity.
• The method is applied to the Pink Trombone synthesizer, which models the glottal source and vocal tract using physical principles.
• For comparison, black-box baselines include genetic algorithms, particle swarm optimization, and a CNN-based neural network for parameter prediction.

Figure 1: The user interface of the Pink Trombone articulatory synthesizer.

Experimental results

Research questions

• RQ1 Can gradient-based optimization effectively recover vocal tract area functions from audio without requiring labeled training data?
• RQ2 How does a white-box, differentiable optimization method compare to black-box baselines in reproducing human vocalizations?
• RQ3 Can the method accurately estimate time-varying articulatory parameters for dynamic vocalizations?
• RQ4 Does gradient-based optimization yield more perceptually accurate reproductions than evolutionary or neural network-based approaches?
• RQ5 Can this method be extended to end-to-end differentiable articulatory synthesis with neural networks?

Key findings

• The proposed gradient-based method accurately recovers vocal tract area functions for audio generated by the Pink Trombone synthesizer itself.
• In a subjective listening test, the method significantly outperformed all black-box baselines, with ratings higher than those of genetic algorithms, particle swarm optimization, and a trained CNN (p < 0.001).
• The method achieved superior perceptual quality without requiring any training data or model fine-tuning.
• The use of frame-wise initialization and gradient propagation ensured smooth, temporally coherent parameter trajectories for time-varying vocalizations.
• Friedman’s test confirmed significant differences in perceptual quality across methods (p < 0.001), and post-hoc tests confirmed the superiority of the gradient-based approach.
• The results demonstrate the feasibility of using differentiable physical models for inverse acoustic modeling in articulatory synthesis.

Figure 2: Illustration of the proposed sound matching method. Target audio is inverse filtered to obtain a source waveform and the transfer function of a filter. For resynthesis, the glottal control parameters $F_{0}$ and Tenseness are estimated from the source waveform. The vocal tract area functio