[Paper Review] Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry

Kashmira Shinde, Jong‐Seok Lee+3 more|arXiv (Cornell University)|Jul 15, 2020

Robotics and Sensor-Based Localization34 references5 citations

TL;DR

This paper proposes an end-to-end Bayesian neural network for monocular visual-inertial odometry that learns multiplicative interactions via multi-head self-attention to enhance sensor fusion robustness. By incorporating scalable Laplace approximation for uncertainty estimation, the method achieves state-of-the-art accuracy and superior resilience to sensor degradation on the KITTI dataset, demonstrating that multiplicative interactions serve as a powerful inductive bias for robust VIO.

ABSTRACT

This paper presents an end-to-end multi-modal learning approach for monocular Visual-Inertial Odometry (VIO), which is specifically designed to exploit sensor complementarity in the light of sensor degradation scenarios. The proposed network makes use of a multi-head self-attention mechanism that learns multiplicative interactions between multiple streams of information. Another design feature of our approach is the incorporation of the model uncertainty using scalable Laplace Approximation. We evaluate the performance of the proposed approach by comparing it against the end-to-end state-of-the-art methods on the KITTI dataset and show that it achieves superior performance. Importantly, our work thereby provides an empirical evidence that learning multiplicative interactions can result in a powerful inductive bias for increased robustness to sensor failures.

Motivation & Objective

• Address the challenge of robust visual-inertial odometry under sensor degradation, especially in safety-critical applications like autonomous driving.
• Overcome limitations of existing additive fusion strategies (e.g., concatenation or weighted fusion) that fail to model complex, complementary interactions between visual and inertial modalities.
• Investigate whether learning multiplicative interactions through attention mechanisms improves robustness and generalization in end-to-end VIO.
• Integrate model uncertainty estimation via scalable Laplace approximation to improve reliability and interpretability of predictions.
• Demonstrate that multiplicative fusion provides a stronger inductive bias than additive fusion, particularly under adverse sensor conditions.

Proposed method

• Employ a multi-head self-attention mechanism to model multiplicative interactions between visual and inertial features, enabling dynamic, attention-driven fusion.
• Use a FlowNet-based architecture to extract geometric features from consecutive monocular images, and a bidirectional LSTM to encode raw IMU measurements.
• Formulate the fusion function as $ f(\mathbf{u}, \mathbf{v}) = \mathbf{v}^T \mathcal{W} \mathbf{u} + \mathbf{v}^T \mathbf{W}_v + \mathbf{W}_u \mathbf{u} + \mathbf{a} $, emphasizing the learnable product term $ \mathbf{v}^T \mathcal{W} \mathbf{u} $ as the core of multiplicative interaction.
• Apply scalable Laplace approximation to approximate the posterior distribution over network weights, enabling Bayesian uncertainty estimation with $ T = 30 $ Monte Carlo samples.
• Regularize the Hessian approximation using $ NF + \tau I $, where $ F $ is the Fisher information approximation, to stabilize uncertainty estimation.
• Train the end-to-end network end-to-end with uncertainty-aware loss, using the mean for prediction and variance for uncertainty quantification.

Experimental results

Research questions

• RQ1 Can multiplicative interactions learned via multi-head self-attention improve the robustness of end-to-end visual-inertial odometry under sensor degradation?
• RQ2 Does incorporating model uncertainty via scalable Laplace approximation enhance the reliability and interpretability of VIO predictions?
• RQ3 How does the proposed method compare to state-of-the-art additive fusion methods (e.g., FC-fusion, soft/hard feature selection) in terms of accuracy and robustness?
• RQ4 To what extent does the multiplicative fusion mechanism serve as a stronger inductive bias than additive mechanisms in real-world degradation scenarios?
• RQ5 Can the predicted uncertainty reliably reflect the actual error magnitude in degraded sensor conditions?

Key findings

• The proposed method achieves a mean translation error of 11.42 on the nominal KITTI sequences, outperforming all baselines including VINet, DeepVO, and VIO Soft.
• Under IMU degradation, the method reduces mean translation error to 12.48, significantly outperforming VIO Soft (14.05) and VINet (13.65).
• In the all-vision-degradation scenario, the method achieves a mean translation error of 12.61, surpassing VIO Soft (14.59) and VINet (13.65).
• Under full sensor degradation, the method maintains a mean translation error of 12.47, demonstrating robustness where other models fail or degrade significantly.
• The predicted uncertainty correlates well with actual errors, with higher variance observed in degraded conditions, validating the reliability of the uncertainty estimation.
• Visualizations show that the predicted trajectories remain close to the ground truth even under severe degradation, confirming the model’s robustness and generalization capability.