[论文解读] Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation

Zhenguang Liu, Runyang Feng+5 more|arXiv (Cornell University)|Mar 29, 2022

Human Pose and Action Recognition被引用 4

一句话总结

本文提出FAMI-Pose，一种用于基于视频的人体姿态估计的新框架，通过自粗到精的特征对齐同步跨帧的时间特征，并利用信息论损失最大化与任务相关的互信息。该方法在SOTA性能上表现优异，在PoseTrack2017上排名第一，并在Sub-JHMDB和PoseTrack2018上创下新SOTA，通过改进的特征对齐和互补线索提取，有效处理快速运动和遮挡问题。

ABSTRACT

Multi-frame human pose estimation has long been a compelling and fundamental problem in computer vision. This task is challenging due to fast motion and pose occlusion that frequently occur in videos. State-of-the-art methods strive to incorporate additional visual evidences from neighboring frames (supporting frames) to facilitate the pose estimation of the current frame (key frame). One aspect that has been obviated so far, is the fact that current methods directly aggregate unaligned contexts across frames. The spatial-misalignment between pose features of the current frame and neighboring frames might lead to unsatisfactory results. More importantly, existing approaches build upon the straightforward pose estimation loss, which unfortunately cannot constrain the network to fully leverage useful information from neighboring frames. To tackle these problems, we present a novel hierarchical alignment framework, which leverages coarse-to-fine deformations to progressively update a neighboring frame to align with the current frame at the feature level. We further propose to explicitly supervise the knowledge extraction from neighboring frames, guaranteeing that useful complementary cues are extracted. To achieve this goal, we theoretically analyzed the mutual information between the frames and arrived at a loss that maximizes the task-relevant mutual information. These allow us to rank No.1 in the Multi-frame Person Pose Estimation Challenge on benchmark dataset PoseTrack2017, and obtain state-of-the-art performance on benchmarks Sub-JHMDB and Pose-Track2018. Our code is released at https://github. com/Pose-Group/FAMI-Pose, hoping that it will be useful to the community.

研究动机与目标

• 解决基于视频的人体姿态估计中关键帧与支持帧特征之间的空间错位问题。
• 克服现有方法依赖未对齐的上下文聚合与简单MSE监督的局限性，这些方法未能有效利用有用的时序线索。
• 通过信息论目标显式监督从邻近帧中提取与任务相关的信息。
• 通过合理的特征对齐与知识蒸馏，提升在快速运动和姿态遮挡等挑战性场景下的鲁棒性。
• 建立统一且理论基础坚实的框架，为多帧人体姿态估计设立新的SOTA基准。

提出的方法

• 该框架采用自粗到精的形变策略：首先应用全局变换模块（GTM）校正关键帧与支持帧特征之间的大尺度空间位移。
• 随后，局部校准模块（LCM）执行自适应的像素级特征调制，以在局部层面进一步优化对齐。
• 引入一种信息论损失，以最大化关键帧特征与对齐后的支持帧特征之间的互信息，确保有效提取互补的时序线索。
• 互信息目标基于理论分析推导得出，作为中间监督信号，引导网络学习与任务相关的表征。
• 该方法采用端到端训练，结合标准姿态估计损失与所提出的基于互信息的损失，实现对齐与特征学习的联合优化。
• 该框架支持可变大小的时间窗口作为支持帧，实现灵活的上下文建模。

Figure 1 : State-of-the-art methods like PoseWarper and DCPose directly aggregate unaligned contexts from neighboring frames, which may fail for scenes with fast motion or pose occlusion. We perform temporal feature alignment between each supporting frame and the key frame, delivering robust pose es

实验结果

研究问题

• RQ1 自粗到精的特征对齐是否能显著提升基于视频的人体姿态估计的准确性？
• RQ2 通过互信息最大化显式监督邻近帧的信息增益，是否能优于标准的MSE监督？
• RQ3 关键帧与支持帧之间的特征错位在快速运动或遮挡场景下如何影响姿态估计？
• RQ4 增加支持帧数量在多大程度上提升性能，且该提升是否可通过对齐与互信息监督进一步增强？
• RQ5 基于理论基础的信息论目标是否能优于启发式或隐式的运动建模方法？

主要发现

• FAMI-Pose在PoseTrack2017验证集上达到84.8的mAP，位列多帧人体姿态估计挑战赛第一。
• 仅使用全局变换模块，mAP相比HRNet-W48提升5.6个百分点，并达到此前SOTA方法DCPose的82.8 mAP水平。
• 加入局部校准模块后，mAP提升至84.0，证实细粒度对齐的优势。
• 整合互信息损失后，mAP进一步提升0.8个百分点，证实其作为辅助监督的有效性。
• 随着支持帧数量增加，性能单调提升：mAP从N={-1}时的82.8提升至N={-2,-1,1,2}时的84.8，表明扩展时序上下文具有显著价值。
• 可视化对比显示，FAMI-Pose在处理运动模糊与姿态遮挡方面优于HRNet-W48、PoseWarper与DCPose，腕部与踝部关键点检测错误更少。

Figure 2 : Overall pipeline of our FAMI-Pose framework. The goal is to detect the pose of person $i$ in the key frame $I_{t}^{i}$ , with the assistance of its supporting frames. For clarity of illustration, we only show a single supporting frame $I_{t+\delta}^{i}$ in this figure. We first extract th