[Paper Review] Learn an Effective Lip Reading Model without Pains

Dalu Feng, Shuang Yang+2 more|arXiv (Cornell University)|Nov 15, 2020

Speech and Audio Processing30 references51 citations

TL;DR

The paper performs a comprehensive quantitative study of training factors for lip reading and shows that simple refinements to a baseline pipeline can achieve state-of-the-art results on LRW and LRW-1000 without substantial model changes.

ABSTRACT

Lip reading, also known as visual speech recognition, aims to recognize the speech content from videos by analyzing the lip dynamics. There have been several appealing progress in recent years, benefiting much from the rapidly developed deep learning techniques and the recent large-scale lip-reading datasets. Most existing methods obtained high performance by constructing a complex neural network, together with several customized training strategies which were always given in a very brief description or even shown only in the source code. We find that making proper use of these strategies could always bring exciting improvements without changing much of the model. Considering the non-negligible effects of these strategies and the existing tough status to train an effective lip reading model, we perform a comprehensive quantitative study and comparative analysis, for the first time, to show the effects of several different choices for lip reading. By only introducing some easy-to-get refinements to the baseline pipeline, we obtain an obvious improvement of the performance from 83.7% to 88.4% and from 38.2% to 55.7% on two largest public available lip reading datasets, LRW and LRW-1000, respectively. They are comparable and even surpass the existing state-of-the-art results.

Motivation & Objective

• Motivate and quantify the impact of training choices and data processing on lip-reading performance.
• Identify effective refinements that improve a baseline lip-reading pipeline.
• Show that a simple, refined training pipeline can achieve competitive results on LRW and LRW-1000 without major architectural changes.

Proposed method

• Use a baseline lip-reading pipeline with ResNet-18 frontend and GRU backend.
• Explore data processing (face alignment, word boundary input), training tweaks (MixUp, label smoothing, learning-rate schedules).
• Incorporate SE (squeeze-and-excitation) modules and cosine learning-rate scheduling into a refined pipeline.
• Evaluate on LRW (English) and LRW-1000 (Mandarin) datasets.
• Compare frontend/backbone modules and document effects of different configurations.
• Report results in tabular form to identify effective combinations.

Experimental results

Research questions

• RQ1 Which training strategies and data processing steps most improve lip-reading performance without changing the model architecture?
• RQ2 Do refinements such as face alignment, word boundary inputs, MixUp, label smoothing, and cosine LR scheduling yield consistent gains on LRW and LRW-1000?
• RQ3 What is the relative impact of frontend/backend choices when combined with these training tricks?
• RQ4 Can a simple, refined pipeline reach or surpass state-of-the-art results on large lip-reading datasets?

Key findings

• Baseline accuracy on LRW improves from 83.7% to 88.4% on LRW with refinements.
• Baseline accuracy on LRW-1000 improves from 46.5% to 55.7% with refinements.
• SE-ResNet-18 frontend with BiGRU backend and aligned lip data yields competitive results; adding word boundary further improves to 88.4% (LRW) and 55.7% (LRW-1000).
• Face alignment and word boundary inputs provide notable gains over the baseline data processing.
• Cosine learning-rate scheduling yields small but positive improvements over plateaus-based scheduling; MixUp and label smoothing contribute to better generalization.
• Among backend modules, GRU-based backends tend to outperform Temporal ConvNET and Transformer in this study under the same conditions.