[Paper Review] Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression

Felix Ott, Nisha Lakshmana Raichur+5 more|arXiv (Cornell University)|Aug 1, 2022

Robotics and Sensor-Based Localization4 citations

TL;DR

This paper benchmarks deep visual-inertial multimodal fusion for relative pose regression (RPR) and odometry-aided absolute pose regression (APR), proposing a novel fusion framework using pose graph optimization (PGO), multi-modal transformation modules (MMTM), and auxiliary/Bayesian learning. It achieves state-of-the-art accuracy on EuRoC MAV, PennCOSYVIO, and a new large-scale IndustryVI dataset, with MMTM-based fusion and PGO showing superior generalization and robustness under image corruption and sensor noise.

ABSTRACT

Visual-inertial localization is a key problem in computer vision and robotics applications such as virtual reality, self-driving cars, and aerial vehicles. The goal is to estimate an accurate pose of an object when either the environment or the dynamics are known. Absolute pose regression (APR) techniques directly regress the absolute pose from an image input in a known scene using convolutional and spatio-temporal networks. Odometry methods perform relative pose regression (RPR) that predicts the relative pose from a known object dynamic (visual or inertial inputs). The localization task can be improved by retrieving information from both data sources for a cross-modal setup, which is a challenging problem due to contradictory tasks. In this work, we conduct a benchmark to evaluate deep multimodal fusion based on pose graph optimization and attention networks. Auxiliary and Bayesian learning are utilized for the APR task. We show accuracy improvements for the APR-RPR task and for the RPR-RPR task for aerial vehicles and hand-held devices. We conduct experiments on the EuRoC MAV and PennCOSYVIO datasets and record and evaluate a novel industry dataset.

Motivation & Objective

• To evaluate and compare deep multimodal fusion techniques for visual-inertial pose estimation in challenging real-world environments.
• To improve absolute pose regression (APR) by fusing visual APR with inertial RPR, leveraging odometry for trajectory consistency.
• To enhance robustness against image degradation and sensor noise through adaptive fusion mechanisms and uncertainty estimation.
• To establish a comprehensive benchmark across diverse datasets, including a novel large-scale industrial indoor dataset (IndustryVI).
• To investigate the impact of sensor bias and modality reliability under corrupted inputs using activation analysis and uncertainty quantification.

Proposed method

• Proposes a multi-stage fusion framework combining visual APR (PoseNet), inertial RPR (IMUNet), and visual RPR (FlowNet) as baseline models.
• Employs pose graph optimization (PGO) to enforce global consistency in trajectory estimation across fused visual and inertial sequences.
• Introduces a multi-modal transformation module (MMTM) for intermediate fusion, enabling learnable, dynamic attention-based feature weighting between modalities.
• Applies late fusion via concatenation and soft fusion with BiLSTM layers to compare fusion strategies across modalities.
• Integrates auxiliary learning to improve main task performance and Bayesian neural networks (BNNs) for aleatoric uncertainty estimation.
• Uses image corruption techniques (blur, noise, occlusion) to evaluate modality reliability and fusion adaptability under degradation.

Experimental results

Research questions

• RQ1 How does visual-inertial multimodal fusion improve absolute pose regression (APR) accuracy compared to unimodal baselines?
• RQ2 Which fusion strategy—late, intermediate, or attention-based—yields the most robust and accurate pose estimation under image degradation?
• RQ3 To what extent does pose graph optimization (PGO) enhance trajectory smoothness and global consistency in APR and RPR tasks?
• RQ4 How do auxiliary learning and Bayesian uncertainty estimation improve model robustness and reliability in challenging conditions?
• RQ5 How does modality reliability shift under image corruption, and can the fusion network adaptively prioritize inertial data in degraded visual conditions?

Key findings

• The APR_V-RPR_I + PGO fusion method achieves the highest performance on the EuRoC MAV dataset, producing smoother and more accurate trajectories than other fusion strategies.
• Intermediate fusion using MMTM modules outperforms late fusion and concatenation, especially on the PennCOSYVIO and IndustryVI datasets, due to better modality interaction and representation learning.
• Fusing three MMTM modules yields superior generalization compared to one or two, indicating deeper fusion benefits for complex motion dynamics.
• Auxiliary learning significantly improves main task performance across all datasets, demonstrating its value in multi-task learning for pose regression.
• Bayesian uncertainty estimation effectively captures model confidence, with higher aleatoric uncertainty observed in degraded visual inputs, validating its use for robustness analysis.
• Under image corruption, the network shifts attention toward inertial features, as shown by increased activation weights on the RPR_I branch, confirming adaptive fusion under challenging conditions.