[Paper Review] Learning Human Optical Flow

Anurag Ranjan, Javier Romero+1 more|arXiv (Cornell University)|Jun 14, 2018

Human Pose and Action Recognition30 references3 citations

TL;DR

This paper proposes HumanFlow, a deep learning-based optical flow method specifically trained for human motion using a large-scale synthetic dataset of 146,020 human motion sequences with ground-truth flow. By fine-tuning SpyNet on this dataset and making it end-to-end trainable, the method achieves 30% better accuracy than state-of-the-art generic flow methods on human motion and generalizes well to real-world scenes while running in real time at 32 fps with only 7.8 MB model size.

ABSTRACT

The optical flow of humans is well known to be useful for the analysis of human action. Given this, we devise an optical flow algorithm specifically for human motion and show that it is superior to generic flow methods. Designing a method by hand is impractical, so we develop a new training database of image sequences with ground truth optical flow. For this we use a 3D model of the human body and motion capture data to synthesize realistic flow fields. We then train a convolutional neural network to estimate human flow fields from pairs of images. Since many applications in human motion analysis depend on speed, and we anticipate mobile applications, we base our method on SpyNet with several modifications. We demonstrate that our trained network is more accurate than a wide range of top methods on held-out test data and that it generalizes well to real image sequences. When combined with a person detector/tracker, the approach provides a full solution to the problem of 2D human flow estimation. Both the code and the dataset are available for research.

Motivation & Objective

• To address the lack of specialized optical flow methods for human motion, which is complex due to articulation, self-occlusion, and variable appearance.
• To create a large-scale, realistic synthetic dataset with ground-truth optical flow for human motion, overcoming the difficulty of capturing accurate real-world human flow.
• To train a deep neural network specifically for human optical flow that generalizes to real-world video sequences.
• To develop a lightweight, real-time model suitable for mobile and embedded applications.
• To provide open-source data, code, and trained models to advance research in human motion understanding.

Proposed method

• The authors synthesize a dataset of 146,020 image pairs using the SMPL body model and motion capture data to simulate diverse human shapes, poses, and movements in realistic indoor environments.
• Ground-truth optical flow is generated synthetically by rendering the 3D motion of the human body into 2D image sequences with accurate pixel-level motion vectors.
• A convolutional neural network based on SpyNet is fine-tuned on this Human Flow dataset, with architectural modifications to improve performance on human motion.
• The network is extended to be end-to-end trainable, enhancing feature learning and flow estimation accuracy.
• The model is optimized for efficiency, resulting in a compact 7.8 MB network with 4.2 million parameters that runs at 32 fps on standard hardware.
• For real-world evaluation, a DPM person detector is used to crop human regions from videos, and the model is applied to estimate flow on these cropped sequences.

Experimental results

Research questions

• RQ1 Can a deep learning model trained on a large-scale synthetic dataset of human motion outperform generic optical flow methods on human-specific motion estimation?
• RQ2 Does a model fine-tuned on human motion generalize effectively to real-world video sequences with complex backgrounds and motion?
• RQ3 Can a compact, real-time optical flow network be trained specifically for human motion without sacrificing accuracy?
• RQ4 How does the performance of a human-specific flow model compare to state-of-the-art generic flow methods on human motion tasks?
• RQ5 To what extent does end-to-end training improve flow estimation accuracy for human motion compared to pre-trained backbones?

Key findings

• The HumanFlow model achieves a 30% improvement in accuracy over previous state-of-the-art optical flow methods on the Human Flow dataset, measured by average end-point error (AEPE).
• The model generalizes well to real-world scenes, producing visually superior results in resolving fine human body parts such as hands, legs, and head, even under occlusion.
• HumanFlow runs at 32 frames per second on a single GPU, making it suitable for real-time applications on mobile and embedded devices.
• The model is compact, requiring only 7.8 MB of memory, with 4.2 million learnable parameters, enabling efficient deployment.
• Visual comparisons show that HumanFlow produces more coherent and accurate flow fields than methods like SPyNet, LDOF, EpicFlow, and FlowFields, especially in regions with complex motion.
• The use of a synthetic dataset with realistic textures and backgrounds enables high-quality training without the need for expensive real-world annotations.