[Paper Review] FGPrompt: Fine-grained Goal Prompting for Image-goal Navigation

Xinyu Sun, Peihao Chen+4 more|arXiv (Cornell University)|Oct 11, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

FGPrompt proposes a fine-grained goal prompting method for image-goal navigation that enhances performance by using high-resolution, intermediate feature maps from the goal image as prompts to condition the observation encoder. By employing FiLM-based mid-fusion or early fusion with joint modeling, it improves attention to goal-relevant regions, achieving a 90.4% success rate on Gibson with 1/50th the model size of SOTA methods.

ABSTRACT

Learning to navigate to an image-specified goal is an important but challenging task for autonomous systems. The agent is required to reason the goal location from where a picture is shot. Existing methods try to solve this problem by learning a navigation policy, which captures semantic features of the goal image and observation image independently and lastly fuses them for predicting a sequence of navigation actions. However, these methods suffer from two major limitations. 1) They may miss detailed information in the goal image, and thus fail to reason the goal location. 2) More critically, it is hard to focus on the goal-relevant regions in the observation image, because they attempt to understand observation without goal conditioning. In this paper, we aim to overcome these limitations by designing a Fine-grained Goal Prompting (FGPrompt) method for image-goal navigation. In particular, we leverage fine-grained and high-resolution feature maps in the goal image as prompts to perform conditioned embedding, which preserves detailed information in the goal image and guides the observation encoder to pay attention to goal-relevant regions. Compared with existing methods on the image-goal navigation benchmark, our method brings significant performance improvement on 3 benchmark datasets (i.e., Gibson, MP3D, and HM3D). Especially on Gibson, we surpass the state-of-the-art success rate by 8% with only 1/50 model size. Project page: https://xinyusun.github.io/fgprompt-pages

Motivation & Objective

• Address the challenge of image-goal navigation where agents must locate a goal based on a reference image.
• Overcome limitations in existing methods that lose fine-grained details in goal images and fail to condition observation encoding with goal context.
• Improve navigation performance by enabling the agent to focus on goal-relevant regions in the observation through goal-conditioned feature modulation.
• Design a parameter-efficient, generalizable architecture suitable for real-world robot deployment with limited computational resources.

Proposed method

• Leverage high-resolution, intermediate feature maps from the goal image as fine-grained prompts to guide observation encoding.
• Implement a mid-fusion mechanism using FiLM layers to apply affine transformations that condition the observation encoder on goal features.
• Design an early-fusion variant (FGPrompt-EF) that concatenates goal and observation images at the pixel level for implicit feature exchange.
• Use a shared ResNet-based encoder in the early fusion setup to jointly model goal and observation features from the start.
• Apply EigenCAM visualization to analyze attention shifts, confirming that the model focuses on goal-relevant objects even at a distance.
• Optimize for model efficiency by minimizing parameters while maximizing performance, enabling deployment on resource-constrained robots.

Experimental results

Research questions

• RQ1 Can fine-grained, high-resolution features from the goal image improve navigation accuracy by guiding attention in the observation?
• RQ2 How does goal-conditioned feature modulation via FiLM layers affect the agent’s ability to identify relevant objects in complex environments?
• RQ3 Does early fusion with joint modeling outperform mid-fusion in terms of efficiency and generalization across unseen environments?
• RQ4 Can the proposed prompting mechanism generalize beyond image-goal navigation to other embodied AI tasks like visual rearrangement?
• RQ5 To what extent does reducing model size impact performance, and can a small model achieve SOTA results in image-goal navigation?

Key findings

• FGPrompt achieves a 90.4% success rate on the Gibson dataset, surpassing the previous SOTA by 8% while using only 1/50th of the model size.
• The mid-fusion variant (FGPrompt-MF) generalizes better than early fusion in scenarios with mismatched camera parameters, indicating stronger inductive bias.
• The early-fusion variant (FGPrompt-EF) achieves competitive performance with a simpler architecture and is highly parameter-efficient.
• On the ai2thor-rearrangement challenge, FGPrompt-EF delivers a 400% relative improvement over the baseline, demonstrating transferability to visual rearrangement.
• Visualization shows that the model successfully focuses on goal-relevant regions (e.g., kitchen bars) even when the agent is far from the goal.
• The method maintains strong performance across seen (Gibson) and unseen environments (MP3D, HM3D), indicating robust generalization.