[Paper Review] HyperPose: Camera Pose Localization using Attention Hypernetworks

Ron Ferens, Yosi Keller|arXiv (Cornell University)|Mar 5, 2023

Robotics and Sensor-Based Localization4 citations

TL;DR

This paper proposes HyperPose, a novel absolute pose regression method that uses a Transformer-encoder as an attention hypernetwork to dynamically generate adaptive regression head weights based on input image features. By enabling dynamic weight modulation in response to visual variations, HyperPose achieves state-of-the-art accuracy on both indoor and outdoor benchmarks, outperforming existing APR methods in handling domain shifts due to lighting, viewpoint, and scene changes.

ABSTRACT

In this study, we propose the use of attention hypernetworks in camera pose localization. The dynamic nature of natural scenes, including changes in environment, perspective, and lighting, creates an inherent domain gap between the training and test sets that limits the accuracy of contemporary localization networks. To overcome this issue, we suggest a camera pose regressor that integrates a hypernetwork. During inference, the hypernetwork generates adaptive weights for the localization regression heads based on the input image, effectively reducing the domain gap. We also suggest the use of a Transformer-Encoder as the hypernetwork, instead of the common multilayer perceptron, to derive an attention hypernetwork. The proposed approach achieves superior results compared to state-of-the-art methods on contemporary datasets. To the best of our knowledge, this is the first instance of using hypernetworks in camera pose regression, as well as using Transformer-Encoders as hypernetworks. We make our code publicly available.

Motivation & Objective

• To address the domain gap in camera pose localization caused by dynamic environmental changes such as lighting, viewpoint, and scene variations.
• To improve the robustness and accuracy of absolute pose regression (APR) models by enabling dynamic, input-dependent adaptation of regression head weights.
• To explore the use of hypernetworks in camera pose estimation, a novel application not previously investigated in this context.
• To evaluate the effectiveness of using a Transformer-Encoder as a hypernetwork, compared to traditional MLP-based hypernetworks, for generating regression head parameters.
• To achieve state-of-the-art performance on standard outdoor and indoor localization benchmarks through adaptive, attention-based weight generation.

Proposed method

• The model uses a dual-branch architecture where intermediate activation maps from a EfficientNet-B0 backbone are processed by separate Transformer-Encoder heads for translation and rotation prediction.
• An attention hypernetwork, implemented as a Transformer-Encoder, takes the backbone’s latent representation and generates dynamic, task-specific weights for the regression heads based on the input image.
• The hypernetwork outputs residual weights that are added to the main network’s regression heads, enabling adaptive modulation of the regression process.
• The architecture treats pose regression as two separate sequence-to-one problems, allowing the model to focus on informative visual cues for each pose component independently.
• The use of 4D-Norm rotation representation with a rotation loss based on rotation matrices improves orientation estimation accuracy over quaternions or 6D representations.
• The entire system is trained end-to-end using backpropagation, with the hypernetwork and main network jointly optimized.

Experimental results

Research questions

• RQ1 Can a hypernetwork architecture improve generalization in camera pose regression under varying environmental conditions?
• RQ2 Does using a Transformer-Encoder as a hypernetwork outperform traditional MLP-based hypernetworks in generating regression weights for pose estimation?
• RQ3 Can dynamic, input-dependent weight modulation via a hypernetwork reduce the domain gap between training and test data in camera localization?
• RQ4 How does the residual-based hypernetwork design compare to direct regression in terms of pose estimation accuracy?
• RQ5 What is the optimal configuration of hypernetwork depth, embedding dimension, and output layer structure for pose regression?

Key findings

• HyperPose achieves state-of-the-art median position and orientation errors on the Cambridge Landmarks dataset, with 0.53 meters and 3.55 degrees, respectively, on the ShopFacade scene.
• The residual hypernetwork architecture outperforms direct regression, reducing position error by 0.02 meters and orientation error by 0.51 degrees.
• Using a Transformer-Encoder as the hypernetwork yields superior performance (0.57m position, 4.06° orientation) compared to an MLP-based hypernetwork (0.68m, 4.29°), demonstrating the benefit of attention mechanisms.
• The ablation study shows that a 256/512 embedding dimension configuration yields the best balance, with 0.53m and 3.55° errors, outperforming 256/256 and 512/512 configurations.
• The 4D-Norm rotation representation with matrix-based loss achieves the lowest orientation error (4.08°), outperforming quaternions (3.55°) and 6D representations (4.54°), indicating its suitability for this task.
• The model demonstrates robust adaptation to visual variations, as shown by the hypernetwork’s ability to generate distinct, context-aware weights for different image inputs.