[Paper Review] Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

He Zhao, Richard P. Wildes|arXiv (Cornell University)|Jul 11, 2021

Human Pose and Action Recognition168 references4 citations

TL;DR

This survey reviews video predictive understanding with a focus on early action recognition and future action prediction, synthesizing decades of research using Markov chains, Gaussian processes, and deep learning models like LSTMs and transformers. It identifies key challenges such as data scarcity, poor generalization, and lack of interpretability, while advocating for improved long-term modeling, joint feature-prediction learning, and self-supervised representation learning to advance robust, scalable video prediction systems.

ABSTRACT

Video predictive understanding encompasses a wide range of efforts that are concerned with the anticipation of the unobserved future from the current as well as historical video observations. Action prediction is a major sub-area of video predictive understanding and is the focus of this review. This sub-area has two major subdivisions: early action recognition and future action prediction. Early action recognition is concerned with recognizing an ongoing action as soon as possible. Future action prediction is concerned with the anticipation of actions that follow those previously observed. In either case, the extbf{ extit{causal}} relationship between the past, current, and potential future information is the main focus. Various mathematical tools such as Markov Chains, Gaussian Processes, Auto-Regressive modeling, and Bayesian recursive filtering are widely adopted jointly with computer vision techniques for these two tasks. However, these approaches face challenges such as the curse of dimensionality, poor generalization, and constraints from domain-specific knowledge. Recently, structures that rely on deep convolutional neural networks and recurrent neural networks have been extensively proposed for improving the performance of existing vision tasks, in general, and action prediction tasks, in particular. However, they have their own shortcomings, \eg reliance on massive training data and lack of strong theoretical underpinnings. In this survey, we start by introducing the major sub-areas of the broad area of video predictive understanding, which recently have received intensive attention and proven to have practical value. Next, a thorough review of various early action recognition and future action prediction algorithms are provided with suitably organized divisions. Finally, we conclude our discussion with future research directions.

Motivation & Objective

• To provide a systematic review of early action recognition and future action prediction as core sub-areas of video predictive understanding.
• To analyze the strengths and limitations of existing mathematical tools (e.g., Markov chains, Bayesian filtering) and deep learning architectures (e.g., RNNs, CNNs) in video prediction tasks.
• To identify critical challenges such as data dependency, poor generalization, and lack of interpretability in current models.
• To propose future research directions, including improved long-term modeling, joint feature-prediction learning, and self-supervised representation learning.
• To evaluate the impact of observation ratio and temporal modeling on prediction performance, and to advocate for better datasets and evaluation protocols.

Proposed method

• Categorizes and reviews 12 major algorithmic families across early action recognition and future action prediction, including one-shot mapping, knowledge distillation, propagation-based methods, and teacher-student learning.
• Analyzes the use of probabilistic models such as Gaussian processes, Kalman filtering, and Poisson processes for temporal reasoning and uncertainty estimation.
• Examines deep learning frameworks like Encouraging-LSTM, deep Kalman filtering, and VAE-Poisson processes for end-to-end feature learning and sequence modeling.
• Evaluates multi-task and joint prediction models that simultaneously predict action labels and temporal durations using structured output layers.
• Reviews ego-centric video prediction methods that leverage first-person perspectives to improve action anticipation in daily activity scenarios.
• Proposes integration of semantic feature extraction with future prediction to enhance robustness under noisy or partial observations.

Experimental results

Research questions

• RQ1 How do different mathematical tools (e.g., Markov chains, Gaussian processes) compare in modeling temporal dependencies for video prediction?
• RQ2 What are the key limitations of current deep learning-based approaches in early action recognition and future action prediction, particularly regarding data efficiency and interpretability?
• RQ3 To what extent does the quality of semantic feature extraction affect long-term prediction performance in video understanding?
• RQ4 How can long-term correlations and remote historical cues be effectively modeled to improve action anticipation accuracy?
• RQ5 What role can self-supervised and unsupervised representation learning play in reducing reliance on large-scale annotated datasets for video prediction?

Key findings

• Deep learning models such as LSTMs and transformers have significantly improved prediction accuracy but often require massive labeled data and lack strong theoretical grounding.
• Joint prediction of action and time duration using multi-task models improves performance on datasets like Epic-Kitchen, with accuracy reaching 78.5% for future action prediction within 1 second.
• Models that integrate feature extraction with prediction, such as deep Kalman filtering, show improved robustness under noisy or partial observations.
• Long-term filtering mechanisms, such as long-term filter banks, demonstrate potential for capturing remote historical cues that influence future action prediction.
• The performance of early action recognition models is highly sensitive to observation ratio; even 50% of video input can yield significant performance drops if feature quality is poor.
• Current evaluation metrics often fail to reflect real-world constraints, and there is a need to re-evaluate models using practical observation ratios and correlation analysis between observation and action evolution rates.