[Paper Review] Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading

Xinshuo Weng, Kris Kitani|arXiv (Cornell University)|May 4, 2019

Video Surveillance and Tracking MethodsComputer Science48 references45 citations

TL;DR

The paper presents a two-stream deep 3D CNN lipreading framework (I3D front-end with grayscale video and optical flow) pre-trained on ImageNet and Kinetics, followed by a Bi-LSTM back-end, achieving state-of-the-art word-level lipreading on LRW with an absolute improvement of 5.3%.

ABSTRACT

We focus on the word-level visual lipreading, which requires recognizing the word being spoken, given only the video but not the audio. State-of-the-art methods explore the use of end-to-end neural networks, including a shallow (up to three layers) 3D convolutional neural network (CNN) + a deep 2D CNN (e.g., ResNet) as the front-end to extract visual features, and a recurrent neural network (e.g., bidirectional LSTM) as the back-end for classification. In this work, we propose to replace the shallow 3D CNNs + deep 2D CNNs front-end with recent successful deep 3D CNNs --- two-stream (i.e., grayscale video and optical flow streams) I3D. We evaluate different combinations of front-end and back-end modules with the grayscale video and optical flow inputs on the LRW dataset. The experiments show that, compared to the shallow 3D CNNs + deep 2D CNNs front-end, the deep 3D CNNs front-end with pre-training on the large-scale image and video datasets (e.g., ImageNet and Kinetics) can improve the classification accuracy. Also, we demonstrate that using the optical flow input alone can achieve comparable performance as using the grayscale video as input. Moreover, the two-stream network using both the grayscale video and optical flow inputs can further improve the performance. Overall, our two-stream I3D front-end with a Bi-LSTM back-end results in an absolute improvement of 5.3% over the previous art on the LRW dataset.

Motivation & Objective

• Advance word-level visual lipreading by leveraging deep 3D CNN front-ends beyond three layers.
• Investigate the benefits of pre-training deep 3D CNNs on large-scale datasets (ImageNet and Kinetics) for lipreading.
• Assess the utility of optical flow as an input and of a two-stream architecture for lipreading.
• Demonstrate end-to-end trainability and end-to-end performance gains over prior two-stage and shallow-front-end methods.

Proposed method

• Use a two-stream I3D front-end (grayscale video and optical flow) to learn spatial-temporal features.
• Inflate 2D ImageNet weights to 3D and perform two-stage pre-training: ImageNet inflation followed by fine-tuning on Kinetics.
• Back-end consists of a two-layer Bi-LSTM to model temporal dependencies and produce word scores.
• Train end-to-end with a softmax layer for word probabilities.
• Compare with single-stream I3D and shallow 3D CNN front-ends to isolate contributions of depth, pre-training, and two-stream inputs.

Experimental results

Research questions

• RQ1 Does a deep 3D CNN front-end outperform shallow 3D plus deep 2D front-ends for lipreading?
• RQ2 Does two-stage pre-training (ImageNet inflation + Kinetics fine-tuning) improve lipreading accuracy?
• RQ3 Is optical flow a viable or complementary input for lipreading, and does a two-stream setup yield improvements?
• RQ4 What is the impact of the back-end choice (Bi-LSTM vs 1D temporal conv nets) on word classification performance?

Key findings

• Two-stream I3D front-end with Bi-LSTM back-end achieves 84.07% test accuracy on LRW, surpassing prior state-of-the-art by 5.3 percentage points.
• Two-round pre-training (ImageNet-inflated 3D weights + Kinetics fine-tuning) is crucial for good performance of deep 3D front-ends.
• Optical flow alone yields comparable performance to grayscale video, and combining both streams yields further improvements.
• Deep 3D CNN front-ends outperform shallow 3D + deep 2D front-ends on LRW.
• Single-stream inputs (grayscale or flow) are effective, but two-stream inputs consistently improve results.