[Paper Review] Selective Listening by Synchronizing Speech with Lips

Zexu Pan, Ruijie Tao+2 more|arXiv (Cornell University)|Jun 14, 2021

Speech and Audio Processing4 citations

TL;DR

This paper proposes a self-supervised audio-visual speaker extraction model that uses lip movements as a visual attractor to enable selective listening without requiring a pre-enrolled speech reference. By leveraging speech-lip synchronization as a self-supervised pre-training signal, the reentry model achieves state-of-the-art performance in target speaker extraction, especially under visual occlusion and cross-domain conditions.

ABSTRACT

A speaker extraction algorithm seeks to extract the speech of a target speaker from a multi-talker speech mixture when given a cue that represents the target speaker, such as a pre-enrolled speech utterance, or an accompanying video track. Visual cues are particularly useful when a pre-enrolled speech is not available. In this work, we don't rely on the target speaker's pre-enrolled speech, but rather use the target speaker's face track as the speaker cue, that is referred to as the auxiliary reference, to form an attractor towards the target speaker. We advocate that the temporal synchronization between the speech and its accompanying lip movements is a direct and dominant audio-visual cue. Therefore, we propose a self-supervised pre-training strategy, to exploit the speech-lip synchronization cue for target speaker extraction, which allows us to leverage abundant unlabeled in-domain data. We transfer the knowledge from the pre-trained model to the attractor encoder of the speaker extraction network. We show that the proposed speaker extraction network outperforms various competitive baselines in terms of signal quality, perceptual quality, and intelligibility, achieving state-of-the-art performance.

Motivation & Objective

• To address the cocktail party problem in single-channel speech separation without requiring a pre-enrolled speech reference.
• To leverage visual cues—specifically lip movements—as a robust, real-time attractor for auditory attention in speaker extraction.
• To develop a self-supervised pre-training strategy based on speech-lip synchronization to exploit large-scale unlabeled in-domain data.
• To improve generalization under visual occlusion and cross-dataset scenarios where visual input is unreliable or mismatched.
• To emulate human reentry-based cross-modal attention by synchronizing audio and visual streams in a deep learning framework.

Proposed method

• Proposes a self-supervised pre-training strategy using speech-lip synchronization as a supervisory signal to learn robust audio-visual representations from unpaired, unlabeled data.
• Designs a reentry model inspired by Edelman’s reentry theory, where audio and visual streams are dynamically synchronized through a bidirectional attention mechanism.
• Uses the target speaker’s face track as an auxiliary reference (attractor) instead of a pre-enrolled speech signal, enabling real-time, self-enrollment in human-robot interaction.
• Transfers knowledge from the pre-trained audio-visual model to the attractor encoder of the speaker extraction network to improve target speaker isolation.
• Employs time-domain speech separation with a U-Net-like architecture to reconstruct the target speaker’s speech while preserving perceptual quality.
• Introduces a data augmentation strategy to simulate visual occlusion by randomly blacking out face tracks during training, improving robustness to tracking failures.

Experimental results

Research questions

• RQ1 Can speech-lip synchronization be used as a self-supervised signal to pre-train a speaker extraction model without requiring labeled speech references?
• RQ2 How effective is a visual face track as an attractor for target speaker extraction when no pre-enrolled speech is available?
• RQ3 To what extent does the reentry model maintain performance under visual occlusion, where the face is partially or fully hidden?
• RQ4 How does the model generalize across domains, such as from studio-recorded to wild, unconstrained video data?
• RQ5 Can the proposed method outperform existing speaker extraction baselines in terms of signal quality, intelligibility, and perceptual quality?

Key findings

• The reentry model achieves 10.27 dB SI-SDRi and 0.909 PESQ on the VoxCeleb2-2mix-occl test set when trained with occlusion data, outperforming all baselines.
• With only 10% effective visual duration (i.e., 10% of the utterance with unoccluded face), the reentry model achieves near 7 dB SI-SDRi, demonstrating robustness to partial occlusion.
• On the LRS2 and LRS3 wild video datasets, the reentry model achieves 12.66 dB SI-SDRi and 1.036 PESQ, surpassing TDSE-I and MuSE-I across all metrics.
• On the Grid and TCD-TIMIT studio datasets, the reentry model achieves 14.96 dB SI-SDRi and 1.118 PESQ, showing strong cross-domain generalization, though MuSE-I slightly outperforms it on Grid for SI-SDRi and PESQ.
• The model trained with occlusion data generalizes significantly better than those without, and the reentry model maintains the highest performance margin across all evaluation settings.