[Paper Review] MetaFuse: A Pre-trained Fusion Model for Human Pose Estimation

Rongchang Xie, Chunyu Wang+1 more|arXiv (Cornell University)|Mar 30, 2020

Human Pose and Action Recognition38 references4 citations

TL;DR

MetaFuse proposes a pre-trained cross-view feature fusion model for human pose estimation that enables rapid adaptation to new camera configurations using only a few labeled images. By factorizing the fusion network into a generic, meta-learned model and lightweight camera-specific transformations, MetaFuse achieves state-of-the-art performance on H36M, Total Capture, and CMU Panoptic datasets with minimal fine-tuning data.

ABSTRACT

Cross view feature fusion is the key to address the occlusion problem in human pose estimation. The current fusion methods need to train a separate model for every pair of cameras making them difficult to scale. In this work, we introduce MetaFuse, a pre-trained fusion model learned from a large number of cameras in the Panoptic dataset. The model can be efficiently adapted or finetuned for a new pair of cameras using a small number of labeled images. The strong adaptation power of MetaFuse is due in large part to the proposed factorization of the original fusion model into two parts (1) a generic fusion model shared by all cameras, and (2) lightweight camera-dependent transformations. Furthermore, the generic model is learned from many cameras by a meta-learning style algorithm to maximize its adaptation capability to various camera poses. We observe in experiments that MetaFuse finetuned on the public datasets outperforms the state-of-the-arts by a large margin which validates its value in practice.

Motivation & Objective

• To address the scalability and inflexibility of existing multi-view pose estimation methods that require retraining for each new camera pair.
• To enable efficient adaptation of feature fusion models to new camera configurations using only a small number of labeled images.
• To improve robustness to occlusion by leveraging cross-view feature fusion across multiple cameras.
• To develop a generic fusion model with strong generalization capability through meta-learning on diverse camera configurations.
• To reduce the data and training cost for deploying multi-view pose estimation in new environments.

Proposed method

• Factorizing the NaiveFuse model into a generic fusion component shared across all cameras and lightweight, camera-specific affine transformations.
• Training the generic fusion model via a meta-learning algorithm to maximize its adaptation performance across diverse camera poses.
• Fine-tuning only the camera-dependent affine transformations on a small number of labeled images in the target domain.
• Using the pre-trained generic model as a strong inductive bias to improve few-shot adaptation performance.
• Integrating MetaFuse with any 2D pose estimation network for end-to-end multi-view 3D pose estimation.
• Leveraging the CMU Panoptic dataset for pre-training on thousands of camera pairs to enhance generalization.

Experimental results

Research questions

• RQ1 Can a pre-trained fusion model be effectively adapted to new camera pairs with minimal labeled data?
• RQ2 How does meta-learning improve the generalization capability of a cross-view feature fusion model across diverse camera configurations?
• RQ3 Can a factorized fusion architecture reduce model complexity while maintaining or improving performance?
• RQ4 Does MetaFuse outperform existing fusion methods in terms of accuracy and data efficiency on benchmark datasets?
• RQ5 Can MetaFuse achieve state-of-the-art 3D pose estimation performance with significantly fewer training images than prior methods?

Key findings

• MetaFuse finetuned on just 50 images outperforms the No-Fusion baseline by 2.6% in joint detection rate (86.3% vs. 83.7%) on H36M.
• On H36M, MetaFuse with 500 finetuning images achieves a 3D MPJPE of 32.7 mm, outperforming state-of-the-art methods that use the full dataset.
• On the Total Capture dataset, MetaFuse achieves the lowest 3D error of 32.4 mm, outperforming recent methods that use temporal modeling.
• MetaFuse reduces 3D error from 40.47 mm (No-Fusion) to 31.78 mm on the Panoptic dataset with only 200 finetuning images.
• MetaFuse with 500 finetuned images achieves a 3D error of 32.4 mm on Total Capture, outperforming LSTM-AE (34.1 mm) despite not using temporal information.
• The Savitzky-Golay filter reduces MetaFuse’s 3D error on the "Walking2" action by ~5 mm, indicating strong potential for post-processing enhancement.