[Paper Review] A Unified Framework for Multi-View Multi-Class Object Pose Estimation

Chi Li, Jin Bai+1 more|arXiv (Cornell University)|Mar 21, 2018

Robotics and Sensor-Based Localization35 references12 citations

TL;DR

This paper proposes a unified deep learning framework for scalable, multi-class, multi-view 6-DoF object pose estimation. It introduces a single-output CNN with SE(3) pose representation via bin-and-delta encoding, tiled class maps for class prior fusion, and deep supervision with object masks, achieving state-of-the-art performance on YCB-Video, JHUScene-50, and ObjectNet-3D benchmarks with improved robustness and generalization.

ABSTRACT

One core challenge in object pose estimation is to ensure accurate and robust performance for large numbers of diverse foreground objects amidst complex background clutter. In this work, we present a scalable framework for accurately inferring six Degree-of-Freedom (6-DoF) pose for a large number of object classes from single or multiple views. To learn discriminative pose features, we integrate three new capabilities into a deep Convolutional Neural Network (CNN): an inference scheme that combines both classification and pose regression based on a uniform tessellation of the Special Euclidean group in three dimensions (SE(3)), the fusion of class priors into the training process via a tiled class map, and an additional regularization using deep supervision with an object mask. Further, an efficient multi-view framework is formulated to address single-view ambiguity. We show that this framework consistently improves the performance of the single-view network. We evaluate our method on three large-scale benchmarks: YCB-Video, JHUScene-50 and ObjectNet-3D. Our approach achieves competitive or superior performance over the current state-of-the-art methods.

Motivation & Objective

• Address the scalability and robustness challenges in 6-DoF object pose estimation for large numbers of diverse object classes under cluttered backgrounds.
• Overcome single-view ambiguity caused by symmetry, occlusion, and appearance variation in complex scenes.
• Develop a unified deep learning architecture that generalizes to unseen object instances while maintaining high accuracy across multiple classes.
• Formulate an efficient multi-view fusion framework that selects reliable pose hypotheses using symmetry-aware voting, reducing reliance on single-view accuracy.
• Improve generalization from synthetic to real-world data through deep supervision with object masks and class-aware feature learning.

Proposed method

• Introduces a bin-and-delta (BD) representation for 6-DoF pose in SE(3), enabling discrete, differentiable, and discriminative pose regression via uniform tessellation of the group.
• Fuses object class labels into the network via a tiled class map, which is concatenated with feature maps to enable class-aware feature learning within a single-branch architecture.
• Applies deep supervision using ground-truth object segmentation masks at intermediate layers to improve feature learning and generalization from synthetic to real images.
• Employs a multi-view fusion framework that aligns pose hypotheses from multiple views to a canonical coordinate system and selects the most consistent pose via symmetry-robust distance metrics.
• Uses a voting-based hypothesis selection mechanism to resolve single-view ambiguities, especially for symmetric objects, improving final pose accuracy.
• Trains the model end-to-end using RGB and RGB-D inputs, with ablation studies to validate each component’s contribution.

Experimental results

Research questions

• RQ1 Can a single deep CNN architecture with a unified output stream achieve state-of-the-art performance across hundreds of object classes while remaining scalable and generalizable?
• RQ2 How does incorporating class priors via a tiled class map improve pose estimation accuracy and generalization compared to per-class or per-branch networks?
• RQ3 To what extent does deep supervision with object masks enhance generalization from synthetic to real-world images in 6-DoF pose estimation?
• RQ4 Can a multi-view fusion framework effectively resolve single-view ambiguities caused by object symmetry and partial occlusion?
• RQ5 How do the proposed components—bin-and-delta representation, tiled class map, and mask supervision—contribute individually and collectively to performance on large-scale benchmarks?

Key findings

• The proposed MCN framework achieves 80.2% mPCK on YCB-Video, significantly outperforming the baseline (61.0%) and the separate-branch baseline (73.8%), demonstrating superior accuracy and scalability.
• On JHUScene-50, MCN achieves 33.9% mPCK, a 8.3% improvement over the baseline (25.0%) and 5.2% over the separate-branch baseline (29.3%), showing strong performance in cluttered scenes.
• On ObjectNet-3D, MCN achieves 90.8% AOS and 78.9% AVP, outperforming the large-scale baseline [42] even though the latter was optimized on the validation set, indicating strong generalization to unseen instances.
• Ablation studies confirm that removing any of the three core components—bin-and-delta, tiled class map, or mask supervision—leads to consistent performance drops, with bin-and-delta being the most critical.
• The separate-branch baseline with shared features still underperforms MCN, and the separate-network baseline is infeasible for large-scale settings due to high training cost (e.g., 100 GPUs for ObjectNet-3D).
• The multi-view framework improves single-view performance by resolving ambiguities through voting, with MV5-MCN achieving higher accuracy than single-view variants on all benchmarks.