[Paper Review] Object Pose Estimation using Mid-level Visual Representations

Negar Nejatishahidin, Pooya Fayyazsanavi+1 more|arXiv (Cornell University)|Mar 2, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a novel 6D object pose estimation method that leverages pre-trained mid-level visual representations—surface normals and re-shading—for improved generalization with minimal labeled data. By fine-tuning a lightweight CNN on object masks and silhouette retrieval, the model achieves 35% higher accuracy than prior methods when trained on only 25% of the Pix3D dataset, and demonstrates strong zero-shot transfer to the challenging Active Vision Dataset (AVD).

ABSTRACT

This work proposes a novel pose estimation model for object categories that can be effectively transferred to previously unseen environments. The deep convolutional network models (CNN) for pose estimation are typically trained and evaluated on datasets specifically curated for object detection, pose estimation, or 3D reconstruction, which requires large amounts of training data. In this work, we propose a model for pose estimation that can be trained with small amount of data and is built on the top of generic mid-level representations \cite{taskonomy2018} (e.g. surface normal estimation and re-shading). These representations are trained on a large dataset without requiring pose and object annotations. Later on, the predictions are refined with a small CNN neural network that exploits object masks and silhouette retrieval. The presented approach achieves superior performance on the Pix3D dataset \cite{pix3d} and shows nearly 35\% improvement over the existing models when only 25\% of the training data is available. We show that the approach is favorable when it comes to generalization and transfer to novel environments. Towards this end, we introduce a new pose estimation benchmark for commonly encountered furniture categories on challenging Active Vision Dataset \cite{Ammirato2017ADF} and evaluated the models trained on the Pix3D dataset.

Motivation & Objective

• Address the challenge of poor generalization in deep learning-based 6D object pose estimation when deployed in unseen environments.
• Reduce reliance on large-scale, pose-annotated datasets by leveraging generic mid-level representations trained on proxy tasks.
• Enable effective pose estimation with minimal supervision by combining mid-level features with a refinement stage using object masks and silhouette retrieval.
• Establish a new benchmark for real-world, challenging indoor environments using the Active Vision Dataset (AVD) to evaluate zero-shot transfer performance.

Proposed method

• Pre-train mid-level representation networks (surface normal estimation and re-shading) on large-scale datasets without pose or object annotations.
• Use the pre-trained mid-level features as a backbone for pose estimation, avoiding end-to-end training from raw pixels.
• Fine-tune a lightweight CNN on top of mid-level features using only object masks and silhouette maps as supervision.
• Implement a retrieval-based refinement stage that leverages learned similarity between predicted and ground-truth silhouettes to improve pose classification.
• Train the full pipeline in a two-stage fashion: first mid-level features are extracted, then a classifier predicts discrete azimuth and elevation bins.
• Evaluate zero-shot transfer by testing the Pix3D-trained model directly on the AVD dataset without any adaptation or fine-tuning.

Figure 1 : Active Vision Dataset [ 1 ] pose estimation benchmark challenges; many objects are highly occluded (e.g. the first two rows), large lighting variations, glassy objects (the third row), and truncated viewpoints (last row). These challenges affect both computation of mid-level representatio

Experimental results

Research questions

• RQ1 Can mid-level visual representations trained on proxy tasks (e.g., surface normals, re-shading) serve as effective, generalizable features for 6D object pose estimation?
• RQ2 To what extent can a lightweight refinement network improve pose estimation accuracy when trained only on object masks and silhouettes?
• RQ3 How does the proposed method perform under low-data regimes, particularly when only 25% of the training data is available?
• RQ4 Can the model generalize effectively to novel, real-world environments with challenging conditions such as occlusions, lighting variations, and truncated views?

Key findings

• The proposed model achieves 35% higher accuracy than existing models when trained on only 25% of the Pix3D training data, demonstrating strong data efficiency.
• On the Pix3D dataset, the model outperforms Mousavian et al. [17] and achieves superior results compared to the Pix3D baseline, especially in the 13-bin setting with 1.5° overlap.
• The model achieves 46.22% mean accuracy on the AVD benchmark, outperforming the ResNet_baseline by 10.56 percentage points, demonstrating strong zero-shot generalization.
• The model’s performance is significantly affected by object mask quality, with degradation observed in highly occluded or truncated views.
• The refinement stage using silhouette retrieval improves performance over the base mid-level feature model, confirming the effectiveness of the two-stage design.
• The approach generalizes well to novel environments, as evidenced by strong performance on the Active Vision Dataset despite no training on this data.

Figure 2 : Network Architecture. First stage: up-sampled and fused mid-level feature maps (not the actual normal and re-shading images) are used to predict azimuth and elevation. Second stage: The object mask and mid-level features are compared with the top three object D-masks corresponding to most