[Paper Review] Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation

Bugra Tekin, Pablo Márquez-Neila+2 more|arXiv (Cornell University)|Nov 17, 2016

Human Pose and Action RecognitionComputer Science62 references17 citations

TL;DR

This paper proposes a novel deep learning framework that jointly regresses 3D human body pose from both 2D joint confidence maps and raw image features using a trainable fusion mechanism. By fusing 2D detection reliability with 3D image cues through a learnable, sharp fusion strategy, the method achieves state-of-the-art performance on standard benchmarks, reducing 3D pose error by 12% compared to prior work and enabling real-time inference at 10 fps.

ABSTRACT

Most recent approaches to monocular 3D human pose estimation rely on Deep Learning. They typically involve regressing from an image to either 3D joint coordinates directly or 2D joint locations from which 3D coordinates are inferred. Both approaches have their strengths and weaknesses and we therefore propose a novel architecture designed to deliver the best of both worlds by performing both simultaneously and fusing the information along the way. At the heart of our framework is a trainable fusion scheme that learns how to fuse the information optimally instead of being hand-designed. This yields significant improvements upon the state-of-the-art on standard 3D human pose estimation benchmarks.

Motivation & Objective

• To address the ambiguity in monocular 3D human pose estimation by combining the strengths of 2D joint detection and direct 3D image reasoning.
• To overcome the limitations of existing methods that either ignore 3D image cues or fail to model 2D joint uncertainty effectively.
• To develop a trainable fusion mechanism that automatically learns optimal fusion points and strategies between 2D and 3D streams, avoiding hand-designed fusion rules.
• To improve generalization and robustness on unconstrained, real-world outdoor images by leveraging complementary features from both streams.
• To achieve high accuracy with low inference latency, enabling real-time deployment.

Proposed method

• The framework employs two parallel streams: one for predicting 2D joint location confidence maps (Confidence Map Stream), and another for extracting 3D image cues directly from the input image (Image Stream).
• The two streams are fused via a trainable fusion module that uses learnable sigmoid gates to dynamically control feature fusion at multiple layers, enabling adaptive, layer-specific fusion.
• The fusion mechanism is regularized with a sparsity-inducing term to encourage sharp fusion, reducing redundant parameters and improving test-time efficiency.
• The network is trained end-to-end using a multi-task loss that optimizes both 2D heatmap prediction and 3D joint regression simultaneously.
• After training, inactive fusion layers are pruned based on learned gate values, resulting in a compact, efficient inference network.
• The model is evaluated on standard benchmarks including Human3.6m, LSP, and KTH, and generalizes well to real-world images even when trained on synthetic data.

Experimental results

Research questions

• RQ1 Can a trainable fusion mechanism outperform hand-designed fusion strategies in combining 2D joint confidence maps and 3D image features for monocular 3D pose estimation?
• RQ2 Does jointly regressing from 2D and 3D cues improve accuracy and robustness compared to methods that use only one modality?
• RQ3 Can the fusion process be made adaptive and efficient through learnable, sparsity-inducing gates that enable sharp fusion at specific layers?
• RQ4 Does the proposed method generalize well to real-world, unconstrained outdoor images when trained on synthetic data?
• RQ5 Can the fusion strategy be applied to other multi-modal learning problems beyond 3D human pose estimation?

Key findings

• The proposed method achieves a 3D pose error of 60.17 mm on the Human3.6m Eating action class, a 12.6% improvement over the baseline without regularization.
• With the regularization term, inference time is reduced to 0.006 sec/frame (167 fps), and the model becomes more efficient by pruning inactive fusion layers.
• The method generalizes well to real images, as demonstrated by high-quality 3D pose predictions on the LSP dataset despite being trained on synthetic data.
• Qualitative results show robust performance on challenging cases involving self-occlusion and viewpoint ambiguity, such as in the Human3.6m and KTH Multiview Football II datasets.
• The features from the two streams are decorrelated, confirming that they encode complementary information, which justifies the fusion strategy.
• The model runs at 10 fps (0.096 sec/frame) on a standard GPU, outperforming prior model-based methods that range from 0.04 to 1 fps.