[Paper Review] Feature Map Filtering: Improving Visual Place Recognition with Convolutional Calibration

Stephen Hausler, Adam Jacobson+1 more|arXiv (Cornell University)|Oct 30, 2018

Robotics and Sensor-Based Localization23 references4 citations

TL;DR

This paper proposes a computationally efficient method called Feature Map Filtering that improves visual place recognition by calibrating a pre-trained CNN using only 50 environment-specific images. By selecting a subset of feature maps that remain consistent across appearance changes (e.g., day/night, season), the method enhances recognition accuracy—especially in early layers—while reducing inference time by up to 64%, without retraining network weights.

ABSTRACT

Convolutional Neural Networks (CNNs) have recently been shown to excel at performing visual place recognition under changing appearance and viewpoint. Previously, place recognition has been improved by intelligently selecting relevant spatial keypoints within a convolutional layer and also by selecting the optimal layer to use. Rather than extracting features out of a particular layer, or a particular set of spatial keypoints within a layer, we propose the extraction of features using a subset of the channel dimensionality within a layer. Each feature map learns to encode a different set of weights that activate for different visual features within the set of training images. We propose a method of calibrating a CNN-based visual place recognition system, which selects the subset of feature maps that best encodes the visual features that are consistent between two different appearances of the same location. Using just 50 calibration images, all collected at the beginning of the current environment, we demonstrate a significant and consistent recognition improvement across multiple layers for two different neural networks. We evaluate our proposal on three datasets with different types of appearance changes - afternoon to morning, winter to summer and night to day. Additionally, the dimensionality reduction approach improves the computational processing speed of the recognition system.

Motivation & Objective

• Address the challenge of visual place recognition under significant appearance changes such as day-night, season, and time-of-day variations.
• Overcome the high computational cost of retraining deep CNNs online for new environments.
• Develop a fast, lightweight calibration method that improves recognition performance without modifying network weights.
• Enable real-time adaptation of pre-trained CNNs to new environmental conditions using minimal calibration data.
• Demonstrate that filtering redundant or inconsistent feature maps leads to both improved accuracy and faster inference.

Proposed method

• Calibrate a pre-trained CNN by selecting a subset of feature maps that minimize L2 distance between same-location images under different appearances.
• Maximize the L2 distance between different locations that appear visually similar due to shared environmental conditions (e.g., same lighting).
• Use a greedy selection algorithm to iteratively remove feature maps that degrade performance under appearance variation.
• Apply the calibration procedure using only 50 images collected at the start of a new environment, without retraining the network.
• Filter feature maps across multiple layers of two CNN architectures: HybridNet and AlexNet (ImageNet-pretrained).
• Evaluate the filtered feature maps for localization performance and computational efficiency on real-world datasets with diverse appearance changes.

Experimental results

Research questions

• RQ1 Can a lightweight calibration procedure improve visual place recognition performance without retraining CNN weights?
• RQ2 Does filtering feature maps based on consistency across appearance variations enhance recognition accuracy in challenging environments?
• RQ3 How effective is the method across different CNN architectures and network layers, especially early vs. late layers?
• RQ4 To what extent does feature map filtering reduce computational cost while maintaining or improving recognition performance?
• RQ5 Can the method generalize across diverse appearance changes such as day-night, winter-summer, and morning-afternoon transitions?

Key findings

• Feature map filtering improved the F1 score from 0.56 to 0.81 on the Oxford RobotCar dataset using HybridNet’s Conv3 layer for night-to-day recognition.
• On AlexNet’s Conv2 layer, the F1 score increased from 0.41 to 0.69 on the same dataset, demonstrating significant accuracy gains in challenging lighting conditions.
• The method reduced inference time by 64% on HybridNet’s Conv3 (from 68ms to 43.9ms per frame) and halved processing time on AlexNet’s Conv2 (from 81ms to 41ms).
• On average, 51% of feature maps were filtered out when using HybridNet and 61% when using AlexNet, indicating substantial dimensionality reduction.
• The greatest performance improvements were observed in early convolutional layers, where low-level features like edges and textures are more sensitive to appearance changes.
• Visualizations confirmed that removed feature maps often responded to transient features like shadows and lighting effects, while retained maps focused on stable cues like road markings.