[Paper Review] OrcVIO: Object residual constrained Visual-Inertial Odometry

Mo Shan, Vikas Dhiman+3 more|arXiv (Cornell University)|Jul 29, 2020

Robotics and Sensor-Based Localization97 references4 citations

TL;DR

OrcVIO proposes a tightly coupled visual-inertial odometry system that jointly estimates sensor trajectory and 3D object poses using semantic residuals from object detectors and semantic keypoints. By integrating object-feature and bounding-box measurements into a modified MSCKF framework, it achieves accurate, real-time sensor localization and large-scale semantic mapping with improved robustness and consistency over purely geometric SLAM approaches.

ABSTRACT

Introducing object-level semantic information into simultaneous localization and mapping (SLAM) system is critical. It not only improves the performance but also enables tasks specified in terms of meaningful objects. This work presents OrcVIO, for visual-inertial odometry tightly coupled with tracking and optimization over structured object models. OrcVIO differentiates through semantic feature and bounding-box reprojection errors to perform batch optimization over the pose and shape of objects. The estimated object states aid in real-time incremental optimization over the IMU-camera states. The ability of OrcVIO for accurate trajectory estimation and large-scale object-level mapping is evaluated using real data.

Motivation & Objective

• To bridge the gap between geometric SLAM and semantic scene understanding by integrating object-level priors into visual-inertial odometry.
• To address the lack of online, globally consistent object mapping in real-time SLAM systems using deep learning-based object detection and tracking.
• To improve sensor trajectory estimation accuracy by leveraging semantic residuals from object features and bounding boxes.
• To enable real-time, incremental estimation of both IMU-camera states and object states (pose, orientation, coarse and fine shape) in a single optimization framework.

Proposed method

• Introduces object states that model position, orientation, coarse ellipsoidal shape, and fine-grained semantic-keypoint-based shape for detected objects.
• Defines two novel residual terms: one linking object semantic keypoints to visual features, and another linking object bounding-box detections to the estimated object shape.
• Derives closed-form Jacobians for all semantic residuals to enable efficient, real-time optimization within a modified multi-state constraint Kalman filter (MSCKF).
• Extends MSCKF to support online, tightly coupled optimization of IMU-camera states and object states over multiple views.
• Uses a coarse category-agnostic ellipsoid model and a fine category-specific semantic keypoint model to represent object shape and pose.
• Employs zero-velocity updates based on pseudo-inertial measurements to reduce drift during static periods.

Experimental results

Research questions

• RQ1 Can object-level semantic priors improve the accuracy and robustness of visual-inertial odometry in real-time systems?
• RQ2 How can object-feature and bounding-box measurements be effectively integrated into a factor graph optimization framework for joint sensor and object state estimation?
• RQ3 To what extent does incorporating semantic residuals enhance trajectory estimation and object mapping performance compared to purely geometric SLAM?
• RQ4 Can a tightly coupled, incremental estimation framework maintain real-time performance while optimizing over both sensor and object states?
• RQ5 How does the use of semantic keypoints and shape priors improve the consistency and interpretability of large-scale semantic maps?

Key findings

• OrcVIO achieves significantly improved sensor trajectory accuracy compared to baseline geometric-only SLAM methods, especially in environments with repeated structures.
• The integration of object semantic residuals reduces drift and improves loop closure performance by providing semantically meaningful constraints.
• The system produces geometrically consistent and semantically rich maps, with estimated object shapes and poses closely matching ground truth in both simulated and real-world experiments.
• The use of coarse ellipsoidal and fine semantic-keypoint models enables accurate object representation without requiring detailed CAD models.
• The algorithm runs in real time and supports online incremental estimation, making it suitable for deployment on mobile robotic platforms.
• Zero-velocity updates effectively reduce drift during static periods, further enhancing trajectory accuracy.