[Paper Review] Pose and Shape Estimation with Discriminatively Learned Parts

Menglong Zhu, Xiaowei Zhou+1 more|arXiv (Cornell University)|Feb 1, 2015

Human Pose and Action Recognition32 references3 citations

TL;DR

This paper proposes a unified framework for 3D pose and shape estimation from a single image by learning discriminative 2D part detectors aligned with 3D landmarks and optimizing for geometric consistency and appearance compatibility simultaneously. Using a sparse shape basis and ADMM-based convex optimization, it achieves state-of-the-art performance on the FG3D Car dataset with mean APD of 3.4° in viewpoint estimation and lower fitting errors than prior methods.

ABSTRACT

We introduce a new approach for estimating the 3D pose and the 3D shape of an object from a single image. Given a training set of view exemplars, we learn and select appearance-based discriminative parts which are mapped onto the 3D model from the training set through a facil- ity location optimization. The training set of 3D models is summarized into a sparse set of shapes from which we can generalize by linear combination. Given a test picture, we detect hypotheses for each part. The main challenge is to select from these hypotheses and compute the 3D pose and shape coefficients at the same time. To achieve this, we optimize a function that minimizes simultaneously the geometric reprojection error as well as the appearance matching of the parts. We apply the alternating direction method of multipliers (ADMM) to minimize the resulting convex function. We evaluate our approach on the Fine Grained 3D Car dataset with superior performance in shape and pose errors. Our main and novel contribution is the simultaneous solution for part localization, 3D pose and shape by maximizing both geometric and appearance compatibility.

Motivation & Objective

• To address the challenge of jointly estimating 3D pose and shape from a single image, avoiding reliance on discrete pose clustering or initial pose estimates.
• To learn a compact, discriminative representation of 2D parts that correspond to 3D landmarks on a 3D model, improving generalization and robustness.
• To simultaneously optimize for part localization, 3D pose, and shape coefficients by balancing geometric reprojection error and appearance matching.
• To enable accurate 3D reconstruction and viewpoint estimation without requiring prior knowledge of the object class or explicit 3D instance matching.
• To overcome limitations of RANSAC and nonlinear optimization by using a convex formulation that avoids local minima.

Proposed method

• Learns discriminative 2D part descriptors from training images, where parts are centered on 3D landmarks projected from a 3D model.
• Uses a facility location optimization to select a minimal set of representative 3D landmarks that maximize discriminative power and coverage.
• Represents intra-class shape variation using a sparse linear combination of basis shapes derived from the training set.
• Detects multiple part hypotheses in test images using learned part detectors, followed by visibility- and proximity-based pruning.
• Optimizes for pose and shape jointly by formulating a convex objective that minimizes both geometric reprojection error and appearance matching, solved via ADMM.
• Joins shape coefficients and 3D rotation parameters into a single matrix variable for joint optimization.

Experimental results

Research questions

• RQ1 Can a discriminatively trained part model be effectively mapped to 3D landmarks to enable joint 3D pose and shape estimation?
• RQ2 How can geometric consistency and appearance compatibility be optimized simultaneously without relying on pairwise constraints or discrete pose initialization?
• RQ3 Can a sparse shape basis effectively model intra-class shape variation while enabling generalization to unseen instances?
• RQ4 Does a convex optimization framework with ADMM outperform nonlinear, non-convex alternatives in avoiding local minima and improving accuracy?
• RQ5 To what extent does the method generalize to novel object instances not seen during training?

Key findings

• The method achieves a mean viewpoint estimation error of 3.4° on the FG3DCar dataset, significantly outperforming V-DPM in both 20° and 40° discretization schemes.
• On 52 selected discriminative landmarks, the method achieves a mean APD of 14.6 pixels using the shape space basis, outperforming both mean shape and class mean baselines.
• The method reduces fitting error on the full set of 64 dataset-provided landmarks compared to the FG3D method, even though it was trained on only 52 automatically selected landmarks.
• The method significantly outperforms FG3D on pickup trucks, demonstrating robustness to challenging object categories with high shape variation.
• Qualitative results show robust 3D wireframe projection and textured 3D reconstruction at novel viewpoints, confirming accurate pose and shape recovery.
• The use of visibility and proximity pruning improves hypothesis selection, enhancing the stability and accuracy of the final pose and shape estimate.