[Paper Review] Privacy-Preserving Visual Localization with Event Cameras

Junho Kim, Young‐Min Kim+5 more|arXiv (Cornell University)|Dec 4, 2022

Random lasers and scattering media4 citations

TL;DR

This paper presents a privacy-preserving visual localization system using event cameras that enables robust camera pose estimation under challenging conditions such as low lighting and fast motion. By converting events to images and applying lightweight sensor- and network-level privacy filters, the method preserves localization accuracy while effectively obscuring facial details and full scene views, making it suitable for practical AR/VR and location-based services.

ABSTRACT

We consider the problem of client-server localization, where edge device users communicate visual data with the service provider for locating oneself against a pre-built 3D map. This localization paradigm is a crucial component for location-based services in AR/VR or mobile applications, as it is not trivial to store large-scale 3D maps and process fast localization on resource-limited edge devices. Nevertheless, conventional client-server localization systems possess numerous challenges in computational efficiency, robustness, and privacy-preservation during data transmission. Our work aims to jointly solve these challenges with a localization pipeline based on event cameras. By using event cameras, our system consumes low energy and maintains small memory bandwidth. Then during localization, we propose applying event-to-image conversion and leverage mature image-based localization, which achieves robustness even in low-light or fast-moving scenes. To further enhance privacy protection, we introduce privacy protection techniques at two levels. Network level protection aims to hide the entire user's view in private scenes using a novel split inference approach, while sensor level protection aims to hide sensitive user details such as faces with light-weight filtering. Both methods involve small client-side computation and localization performance loss, while significantly mitigating the feeling of insecurity as revealed in our user study. We thus project our method to serve as a building block for practical location-based services using event cameras. Project page including the code is available through this link: https://82magnolia.github.io/event\_localization/.

Motivation & Objective

• Address privacy concerns in client-server visual localization where users share visual data with service providers.
• Enable robust visual localization in low-lighting and fast-motion scenarios using event cameras’ high dynamic range and temporal resolution.
• Develop lightweight, privacy-preserving mechanisms at both sensor and network levels to protect sensitive visual information without sacrificing localization performance.
• Provide a practical pipeline for event-based visual localization that supports real-world deployment in AR/VR and private environments.

Proposed method

• Apply event-to-image conversion on the service provider side to leverage powerful image-based localization methods, enabling robust pose estimation despite domain gaps between event and image data.
• Implement sensor-level privacy protection via a lightweight filtering technique that blurs facial landmarks without explicit detection, preserving key features for localization.
• Introduce a network-level privacy mechanism that splits neural network inference, keeping intermediate features on the service provider side while preventing image reconstruction from events.
• Use a dual-stage inference pipeline where the user performs initial processing and the service provider handles computationally heavy steps, ensuring privacy through obfuscation of intermediate representations.
• Leverage event camera data—capturing only brightness changes—to inherently reduce visual fidelity and limit sensitive detail exposure.
• Train and evaluate on new datasets (EvRooms and EvHumans) to assess performance across diverse conditions including moving people, low light, and fast motion.

Experimental results

Research questions

• RQ1 Can event cameras enable robust visual localization under low-lighting and fast camera motion, where conventional image-based methods fail?
• RQ2 To what extent can sensor-level filtering obscure facial details without degrading localization accuracy?
• RQ3 Can a network-level privacy mechanism prevent image reconstruction from events while maintaining localization performance?
• RQ4 How do users perceive the privacy of facial and scene-level details in event-based visual localization systems?
• RQ5 What is the performance trade-off between privacy protection and localization accuracy in real-world event-based scenarios?

Key findings

• The proposed event-based localization pipeline achieves robust performance in low-lighting and fast-motion scenarios, outperforming conventional image-based methods under these conditions.
• Sensor-level privacy filtering effectively obscures facial details while maintaining localization accuracy with only a small performance drop, as validated by user studies and qualitative results.
• Network-level privacy protection successfully prevents image reconstruction from events, even under targeted retraining attacks, demonstrating strong resistance to privacy leakage.
• User studies indicate strong user confidence in privacy protection: most participants reported feeling secure with facial details hidden and full scene views obscured in private settings.
• The method incurs minimal performance degradation—less than 5% drop in localization accuracy—while achieving significant privacy gains, making it suitable for real-world deployment.
• The authors collected and partially released two new datasets, EvRooms and EvHumans, to support future research in event-based visual localization under challenging conditions.