[Paper Review] Unsupervised Learning of View-invariant Action Representations

Junnan Li, Yongkang Wong+2 more|arXiv (Cornell University)|Sep 6, 2018

Human Pose and Action RecognitionComputer Science56 references70 citations

TL;DR

The paper proposes an unsupervised framework that learns view-invariant video representations by predicting cross-view 3D motion from a source view, aided by view-adversarial training, and demonstrates strong cross-view and cross-subject action recognition performance.

ABSTRACT

The recent success in human action recognition with deep learning methods mostly adopt the supervised learning paradigm, which requires significant amount of manually labeled data to achieve good performance. However, label collection is an expensive and time-consuming process. In this work, we propose an unsupervised learning framework, which exploits unlabeled data to learn video representations. Different from previous works in video representation learning, our unsupervised learning task is to predict 3D motion in multiple target views using video representation from a source view. By learning to extrapolate cross-view motions, the representation can capture view-invariant motion dynamics which is discriminative for the action. In addition, we propose a view-adversarial training method to enhance learning of view-invariant features. We demonstrate the effectiveness of the learned representations for action recognition on multiple datasets.

Motivation & Objective

• Motivate reducing reliance on labeled data for action recognition by leveraging unlabeled multi-view video data.
• Learn a view-invariant motion representation that can predict 3D flows for multiple views from a source view.
• Encourage view invariance with a view-adversarial training objective.
• Demonstrate effectiveness on cross-subject and cross-view action recognition across RGB, depth, and flow modalities.

Proposed method

• Encode multi-view video frames with a CNN+BiLSTM-based encoder to extract motion-sensitive features.
• Predict 3D flow sequences for other views using a cross-view decoder that uses view-specific depth anchors.
• Reconstruct flows for the same view with a reconstruction decoder to reinforce motion dynamics capture.
• Apply a view classifier with a Gradient Reversal Layer to promote view-invariant representations (adversarial training).
• Combine losses from cross-view flow prediction, flow reconstruction, and view classification to train end-to-end.

Experimental results

Research questions

• RQ1 Can an unsupervised model learn representations that are robust to viewpoint changes by predicting cross-view motion?
• RQ2 How does view-adversarial training affect the invariance and discriminative power of learned representations?
• RQ3 Do the learned representations transfer effectively to cross-subject and cross-view action recognition across multiple modalities?

Key findings

• Flow-based cross-view prediction is strongest among RGB, depth, and flow, indicating flow contains more view-invariant motion information.
• Incorporating flow reconstruction and view-adversarial training improves cross-view flow prediction across modalities.
• Unsupervised learned representations enable competitive action recognition, especially with flow input, and benefit from fine-tuning or fixing the encoder during supervised evaluation.
• The approach yields superior cross-subject and cross-view accuracy compared to several baselines and reaches competitive results with state-of-the-art methods using other modalities.
• Transfer to other datasets (MSR-DailyActivity3D and Northwestern-UCLA MultiviewAction3D) shows the learned representations generalize across domains when fine-tuned.