[Paper Review] Lip-reading with Hierarchical Pyramidal Convolution and Self-Attention

Hang Chen, Jun Du+4 more|arXiv (Cornell University)|Dec 28, 2020

Speech and Audio Processing25 references4 citations

TL;DR

This paper proposes a novel deep learning model for word-level lip-reading that replaces standard convolution with hierarchical pyramidal convolution (HPConv) for multi-scale spatial feature extraction and replaces average pooling consensus with self-attention-based temporal fusion. The model achieves 86.83% accuracy on the LRW dataset, a 1.53% absolute improvement over the prior state-of-the-art, demonstrating enhanced detection of fine-grained lip movements and improved attention to relevant video frames.

ABSTRACT

In this paper, we propose a novel deep learning architecture to improving word-level lip-reading. On the one hand, we first introduce the multi-scale processing into the spatial feature extraction for lip-reading. Specially, we proposed hierarchical pyramidal convolution (HPConv) to replace the standard convolution in original module, leading to improvements over the model's ability to discover fine-grained lip movements. On the other hand, we merge information in all time steps of the sequence by utilizing self-attention, to make the model pay more attention to the relevant frames. These two advantages are combined together to further enhance the model's classification power. Experiments on the Lip Reading in the Wild (LRW) dataset show that our proposed model has achieved 86.83% accuracy, yielding 1.53% absolute improvement over the current state-of-the-art. We also conducted extensive experiments to better understand the behavior of the proposed model.

Motivation & Objective

• To improve word-level lip-reading performance by enhancing spatial feature extraction of fine-grained lip movements.
• To address the limitation of average pooling consensus by learning dynamic, attention-based frame weighting.
• To integrate multi-scale processing into the frontend of lip-reading models for better representation of viseme variations.
• To validate the effectiveness of hierarchical pyramidal convolution and self-attention consensus through ablation and comparative experiments on the LRW benchmark.

Proposed method

• Proposes hierarchical pyramidal convolution (HPConv) to process input features at multiple spatial resolutions, enabling better capture of local and global lip motion patterns.
• Replaces standard 2D convolutions in ResNet-18 with HPConv to extract multi-scale spatial features while maintaining hierarchical feature learning.
• Introduces a self-attention-based consensus module that dynamically weights temporal frames based on their relevance to the target word.
• Uses a learnable attention mechanism to generate soft word boundaries, improving frame selection over fixed or uniform averaging.
• Employs a multi-scale temporal convolutional network (MS-TCN) as the backend to model long-range temporal dependencies.
• Combines HPConv in the frontend with self-attention consensus to jointly improve spatial perception and temporal relevance modeling.

Experimental results

Research questions

• RQ1 Can multi-scale spatial feature extraction via hierarchical pyramidal convolution improve recognition of subtle lip movements in word-level lip-reading?
• RQ2 Does self-attention-based temporal consensus outperform average pooling in focusing on relevant video frames during classification?
• RQ3 How do HPConv and self-attention consensus individually and jointly affect performance on the LRW dataset?
• RQ4 To what extent can the model learn accurate word boundaries through attention mechanisms compared to manual annotations?

Key findings

• The proposed model achieves 86.83% top-1 accuracy on the LRW dataset, representing a 1.53% absolute improvement over the previous state-of-the-art.
• HPConv significantly improves classification accuracy on words with few visemes by enhancing detection of subtle lip movements.
• The self-attention-based consensus outperforms average pooling, especially when word boundaries are ambiguous, by learning more precise frame relevance.
• Ablation studies confirm that both HPConv and self-attention consensus contribute independently and synergistically to performance gains.
• The model’s learned attention weights correlate with manual word boundaries, and accuracy decreases with increasing edit distance, confirming the model learns meaningful temporal structure.