[论文解读] Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

He Zhao, Richard P. Wildes|arXiv (Cornell University)|Jul 11, 2021

Human Pose and Action Recognition参考文献 168被引用 4

一句话总结

本综述聚焦于早期动作识别与未来动作预测，系统回顾了视频预测理解领域的研究进展，涵盖基于马尔可夫链、高斯过程以及LSTM、Transformer等深度学习模型的数十年研究。文章识别出数据稀缺、泛化能力差、可解释性不足等关键挑战，同时倡导改进长期建模、联合特征-预测学习以及自监督表征学习，以推动更鲁棒、可扩展的视频预测系统发展。

ABSTRACT

Video predictive understanding encompasses a wide range of efforts that are concerned with the anticipation of the unobserved future from the current as well as historical video observations. Action prediction is a major sub-area of video predictive understanding and is the focus of this review. This sub-area has two major subdivisions: early action recognition and future action prediction. Early action recognition is concerned with recognizing an ongoing action as soon as possible. Future action prediction is concerned with the anticipation of actions that follow those previously observed. In either case, the extbf{ extit{causal}} relationship between the past, current, and potential future information is the main focus. Various mathematical tools such as Markov Chains, Gaussian Processes, Auto-Regressive modeling, and Bayesian recursive filtering are widely adopted jointly with computer vision techniques for these two tasks. However, these approaches face challenges such as the curse of dimensionality, poor generalization, and constraints from domain-specific knowledge. Recently, structures that rely on deep convolutional neural networks and recurrent neural networks have been extensively proposed for improving the performance of existing vision tasks, in general, and action prediction tasks, in particular. However, they have their own shortcomings, \eg reliance on massive training data and lack of strong theoretical underpinnings. In this survey, we start by introducing the major sub-areas of the broad area of video predictive understanding, which recently have received intensive attention and proven to have practical value. Next, a thorough review of various early action recognition and future action prediction algorithms are provided with suitably organized divisions. Finally, we conclude our discussion with future research directions.

研究动机与目标

• 系统回顾早期动作识别与未来动作预测作为视频预测理解核心子领域的研究进展。
• 分析现有数学工具（如马尔可夫链、贝叶斯滤波）与深度学习架构（如RNN、CNN）在视频预测任务中的优势与局限。
• 识别当前模型在数据依赖性、泛化能力差以及可解释性不足等方面的关键挑战。
• 提出未来研究方向，包括改进长期建模、联合特征-预测学习以及自监督表征学习。
• 评估观测比例与时间建模对预测性能的影响，并倡导构建更优的数据集与评估协议。

提出的方法

• 对早期动作识别与未来动作预测中的12类主要算法家族进行分类与综述，包括零样本映射、知识蒸馏、基于传播的方法以及教师-学生学习。
• 分析高斯过程、卡尔曼滤波与泊松过程等概率模型在时间推理与不确定性估计中的应用。
• 研究端到端特征学习与序列建模的深度学习框架，如鼓励性LSTM、深度卡尔曼滤波与VAE-泊松过程。
• 评估多任务与联合预测模型，通过结构化输出层同时预测动作标签与时间持续时长。
• 综述第一人称视角视频预测方法，利用第一视角信息提升日常活动场景中的动作预判能力。
• 提出将语义特征提取与未来预测相结合，以增强在噪声或部分观测条件下的鲁棒性。

实验结果

研究问题

• RQ1 不同数学工具（如马尔可夫链、高斯过程）在建模视频预测的时间依赖性方面有何比较优势？
• RQ2 当前基于深度学习的方法在早期动作识别与未来动作预测中的关键局限是什么，特别是在数据效率与可解释性方面？
• RQ3 语义特征提取的质量在多大程度上影响视频理解中的长期预测性能？
• RQ4 如何有效建模长期相关性与远程历史线索，以提升动作预判的准确性？
• RQ5 自监督与无监督表征学习在减少对大规模标注数据集依赖方面可发挥何种作用？

主要发现

• LSTM与Transformer等深度学习模型显著提升了预测准确率，但通常需要大量标注数据，且缺乏坚实的理论基础。
• 采用多任务模型联合预测动作与持续时间，在Epic-Kitchen等数据集上实现未来动作预测准确率达78.5%（1秒内预测）。
• 将特征提取与预测相结合的模型（如深度卡尔曼滤波）在噪声或部分观测条件下表现出更强的鲁棒性。
• 长期滤波机制（如长期滤波器组）在捕捉影响未来动作预测的远程历史线索方面展现出潜力。
• 早期动作识别模型的性能对观测比例高度敏感；即使仅使用50%的视频输入，若特征质量差，性能仍可能显著下降。
• 当前评估指标常无法反映真实世界约束，亟需基于实际观测比例与观测与动作演化速率之间的相关性分析，重新评估模型性能。