[Paper Review] Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains

Bowen Wen, Chaitanya Mitash+1 more|arXiv (Cornell University)|May 29, 2021

Robot Manipulation and Learning29 references4 citations

TL;DR

This paper proposes $se(3)$-TrackNet, a data-driven 6D object pose tracking method that uses a novel neural network with disentangled feature encoding and Lie algebra-based 3D orientation representation to enable robust sim-to-real transfer. Trained exclusively on synthetic data with domain randomization, it achieves real-time performance at 90.9Hz and state-of-the-art accuracy on YCB-Video and a new YCBInEOAT benchmark, outperforming methods trained on real data and eliminating the need for re-initialization.

ABSTRACT

Tracking the 6D pose of objects in video sequences is important for robot manipulation. This work presents se(3)-TrackNet, a data-driven optimization approach for long term, 6D pose tracking. It aims to identify the optimal relative pose given the current RGB-D observation and a synthetic image conditioned on the previous best estimate and the object's model. The key contribution in this context is a novel neural network architecture, which appropriately disentangles the feature encoding to help reduce domain shift, and an effective 3D orientation representation via Lie Algebra. Consequently, even when the network is trained solely with synthetic data can work effectively over real images. Comprehensive experiments over multiple benchmarks show se(3)-TrackNet achieves consistently robust estimates and outperforms alternatives, even though they have been trained with real images. The approach runs in real time at 90.9Hz. Code, data and supplementary video for this project are available at https://github.com/wenbowen123/iros20-6d-pose-tracking

Motivation & Objective

• To develop a robust, real-time 6D object pose tracking system that minimizes reliance on costly real-world annotated data.
• To address domain shift between synthetic training data and real-world test images through a novel disentangled feature encoding architecture.
• To enable long-term, stable pose tracking under challenging conditions such as occlusion, motion, and viewpoint changes.
• To achieve state-of-the-art performance on benchmark datasets without requiring re-initialization after tracking failure.
• To introduce a new benchmark dataset, YCBInEOAT, for evaluating 6D pose tracking in dynamic robotic manipulation scenarios.

Proposed method

• The method employs a dual-branch neural network with separate feature encoders for the current RGB-D observation and a synthetic rendering of the object based on the previous pose estimate.
• It uses a Lie algebra representation for 3D rotations to enable effective learning of residual pose transformations with a differentiable loss function.
• The network is trained exclusively on synthetic data generated via a physics-based simulation pipeline with domain randomization to reduce domain shift.
• Feature encoding is disentangled to separate content and transformation information, improving generalization to real images.
• The system predicts a relative 6D pose transform ($\Delta T_{\tau}$) between consecutive frames, which is applied iteratively to refine the object's pose over time.
• A shape-matching loss is used during training to improve robustness to symmetry and object shape variations.

Experimental results

Research questions

• RQ1 Can a 6D object pose tracking network trained solely on synthetic data achieve comparable or better performance than real-data-trained baselines on real-world benchmarks?
• RQ2 How effective is a Lie algebra-based 3D orientation representation in enabling accurate and differentiable learning of residual pose transformations?
• RQ3 To what extent does disentangled feature encoding reduce domain shift in sim-to-real 6D pose tracking?
• RQ4 Can the proposed method maintain long-term tracking accuracy without re-initialization under challenging conditions like occlusion and large motion?
• RQ5 How does the method perform on a new benchmark dataset designed for robotic manipulation tasks with dynamic object interactions?

Key findings

• On the YCB-Video benchmark, $se(3)$-TrackNet achieves SOTA performance with 94.71% AUC on ADD and 96.93% on ADD-S, outperforming real-data-trained baselines without any re-initialization.
• The method runs at 90.9Hz, significantly faster than prior state-of-the-art approaches, enabling real-time application in robotic manipulation.
• On the YCBInEOAT dataset, $se(3)$-TrackNet achieves 92.66% AUC on ADD and 95.53% on ADD-S across all 5 YCB objects, demonstrating robustness to dynamic manipulation and slippage.
• Ablation studies show that removing physics simulation during data generation reduces performance to 91.88% AUC (ADD), indicating the importance of realistic dynamics in synthetic data.
• Removing depth modality during training and inference degrades performance to 75.65% AUC (ADD), highlighting the value of depth supervision.
• Using a shared encoder for both current and rendered images results in near-zero performance (0.28% AUC), proving that separate feature learning is essential for tracking.