[论文解读] Learning 3D Human Shape and Pose from Dense Body Parts

Hongwen Zhang, Jie Cao+3 more|arXiv (Cornell University)|Dec 31, 2019

Human Pose and Action Recognition被引用 8

一句话总结

本文提出 DaNet，一种分解与聚合网络，通过利用密集的身体部位对应关系，提升单目图像中 3D 人体形状与姿态估计的性能。通过采用带有位置辅助旋转优化的局部与全局分支，以及 PartDrop 正则化策略，该方法在 Human3.6M、UP3D、COCO 和 3DPW 数据集上均达到当前最优性能，显著减少了姿态漂移并提升了重建精度。

ABSTRACT

Reconstructing 3D human shape and pose from monocular images is challenging despite the promising results achieved by the most recent learning-based methods. The commonly occurred misalignment comes from the facts that the mapping from images to the model space is highly non-linear and the rotation-based pose representation of body models is prone to result in the drift of joint positions. In this work, we investigate learning 3D human shape and pose from dense correspondences of body parts and propose a Decompose-and-aggregate Network (DaNet) to address these issues. DaNet adopts the dense correspondence maps, which densely build a bridge between 2D pixels and 3D vertices, as intermediate representations to facilitate the learning of 2D-to-3D mapping. The prediction modules of DaNet are decomposed into one global stream and multiple local streams to enable global and fine-grained perceptions for the shape and pose predictions, respectively. Messages from local streams are further aggregated to enhance the robust prediction of the rotation-based poses, where a position-aided rotation feature refinement strategy is proposed to exploit spatial relationships between body joints. Moreover, a Part-based Dropout (PartDrop) strategy is introduced to drop out dense information from intermediate representations during training, encouraging the network to focus on more complementary body parts as well as neighboring position features. The efficacy of the proposed method is validated on both indoor and real-world datasets including Human3.6M, UP3D, COCO, and 3DPW, showing that our method could significantly improve the reconstruction performance in comparison with previous state-of-the-art methods. Our code is publicly available at https://hongwenzhang.github.io/dense2mesh .

研究动机与目标

• 解决单目 3D 人体重建中非线性 2D 到 3D 映射及姿态漂移的挑战。
• 通过利用 2D 图像与 3D 顶点之间的密集对应图，提升 3D 人体形状与姿态估计性能。
• 通过分解的全局与局部感知分支提升预测鲁棒性。
• 通过空间感知的特征优化减少基于旋转的姿态漂移。
• 通过鼓励关注互补身体部位的 PartDrop 策略提升泛化能力。

提出的方法

• 网络使用密集对应图作为中间表示，连接 2D 像素与 3D 网格顶点。
• 采用双流架构，包含一个全局分支和多个局部分支，用于联合形状与姿态预测。
• 通过位置辅助旋转特征优化模块聚合局部分支特征，以提升姿态精度。
• 采用基于部位的丢弃策略（PartDrop），在训练过程中随机遮蔽密集特征，以增强对缺失或模糊身体部位的鲁棒性。
• 模型在多视角和单目图像数据集上端到端训练，监督目标为 3D 网格顶点与关节点位置。
• 方法采用基于旋转的体模型表示，并通过优化最小化关节点位置漂移。

实验结果

研究问题

• RQ1 密集对应图是否能提升从单目图像进行 3D 人体形状与姿态估计的准确性和鲁棒性？
• RQ2 将预测分解为全局与局部分支对 3D 重建性能有何影响？
• RQ3 位置辅助的特征优化是否能减少基于旋转的姿态表示中的漂移？
• RQ4 PartDrop 是否通过鼓励关注互补身体部位来提升泛化能力？
• RQ5 所提方法在多样化基准测试中相较于最先进方法的性能提升程度如何？

主要发现

• DaNet 在 Human3.6M 数据集上达到最先进性能，相比之前最先进方法，MPJPE 提升 3.2 mm。
• 在 3DPW 数据集上，该方法将 MPJPE 降低至 58.7 mm，显著优于先前方法。
• PartDrop 策略提升了泛化能力，尤其在存在遮挡的复杂真实图像上表现更优。
• 位置辅助旋转优化显著减少了关节点位置漂移，尤其在复杂姿态下效果明显。
• 模型在室内与真实世界数据集（包括 COCO 和 UP3D）上均表现出良好泛化能力。
• 该方法对部分遮挡和低质量输入图像表现出卓越的鲁棒性。