[Paper Review] Hallucinating Optical Flow Features for Video Classification

Yongyi Tang, Lin Ma+1 more|arXiv (Cornell University)|May 28, 2019

Advanced Image Processing TechniquesComputer Science31 references3 citations

TL;DR

This paper proposes MoNet, a motion hallucination network that generates optical flow features from appearance features without relying on computationally expensive optical flow estimation. By modeling temporal and contextual relationships concurrently, MoNet enables two-stream video classification with up to 50% reduction in FLOPs and storage, achieving consistent performance gains on Kinetics-400 and YouTube-8M with minimal accuracy drop compared to ground-truth flows.

ABSTRACT

Appearance and motion are two key components to depict and characterize the video content. Currently, the two-stream models have achieved state-of-the-art performances on video classification. However, extracting motion information, specifically in the form of optical flow features, is extremely computationally expensive, especially for large-scale video classification. In this paper, we propose a motion hallucination network, namely MoNet, to imagine the optical flow features from the appearance features, with no reliance on the optical flow computation. Specifically, MoNet models the temporal relationships of the appearance features and exploits the contextual relationships of the optical flow features with concurrent connections. Extensive experimental results demonstrate that the proposed MoNet can effectively and efficiently hallucinate the optical flow features, which together with the appearance features consistently improve the video classification performances. Moreover, MoNet can help cutting down almost a half of computational and data-storage burdens for the two-stream video classification. Our code is available at: https://github.com/YongyiTang92/MoNet-Features.

Motivation & Objective

• To eliminate the high computational and storage costs of optical flow estimation in two-stream video classification.
• To develop a method that hallucinates motion representations from appearance features without relying on optical flow computation.
• To improve two-stream video classification performance using hallucinated optical flow features that complement appearance features.
• To enable efficient large-scale video classification by bypassing optical flow extraction and feature extraction pipelines.

Proposed method

• MoNet uses a concurrent architecture to model temporal dependencies in appearance features and contextual relationships in optical flow features simultaneously.
• The network employs recursive expansion with residual connections to propagate information across time and spatial contexts.
• It formulates feature hallucination as a sequence-to-sequence translation problem, leveraging RNN-like long-range modeling with improved context aggregation.
• The model is pre-trained on Kinetics-400 using appearance features and I3D-extracted optical flow features, then transferred to YouTube-8M.
• Hallucinated optical flow features are fused with appearance features in a two-stream classification framework.
• The method avoids optical flow computation and storage by directly generating motion features from appearance features via a learned transformation.

Experimental results

Research questions

• RQ1 Can optical flow features be effectively hallucinated from appearance features without explicit optical flow computation?
• RQ2 Can a concurrent modeling approach outperform standard RNNs in capturing complex motion representations for video classification?
• RQ3 To what extent can hallucinated optical flow features improve two-stream video classification accuracy while reducing computational and storage costs?
• RQ4 How well does the hallucinated motion representation generalize to large-scale video datasets like YouTube-8M?

Key findings

• The hallucinated two-stream model achieves 71.32% top-1 accuracy on Kinetics-400 using a linear classifier, and 72.40% with NeXtVLAD, showing consistent improvement over single-stream appearance models.
• On YouTube-8M, the hallucinated two-stream model reaches 88.62% Hit@1, 80.41% GAP@20, 79.39% MAP@20, and 46.79% PERR, outperforming the single appearance stream across all metrics.
• The computational cost is reduced from 425 GFLOPs (with real optical flow) to 224 GFLOPs (hallucinated), cutting FLOPs by nearly half.
• The model saves approximately 50% in data storage by eliminating the need to store optical flow images.
• Performance improves with deeper MoNet architectures, confirming the effectiveness of the recursive expansion strategy.
• The hallucinated optical flow features are sufficiently representative to complement appearance features, achieving ~2% accuracy drop compared to models using ground-truth optical flows.