[Paper Review] Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill

Wenzhe Cai, Siyuan Huang+5 more|arXiv (Cornell University)|Sep 19, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Pixel-guided Navigation (PixNav), a pure RGB-based navigation policy that uses pixels as goal specifications to bridge foundation models and zero-shot object navigation, enabling versatile, low-cost, and generalizable robot navigation. PixNav achieves 80%+ success rate in local path planning and, when combined with an LLM-based planner, enables effective long-horizon navigation in both simulated and real-world environments without requiring depth sensing or map-based planning.

ABSTRACT

Zero-shot object navigation is a challenging task for home-assistance robots. This task emphasizes visual grounding, commonsense inference and locomotion abilities, where the first two are inherent in foundation models. But for the locomotion part, most works still depend on map-based planning approaches. The gap between RGB space and map space makes it difficult to directly transfer the knowledge from foundation models to navigation tasks. In this work, we propose a Pixel-guided Navigation skill (PixNav), which bridges the gap between the foundation models and the embodied navigation task. It is straightforward for recent foundation models to indicate an object by pixels, and with pixels as the goal specification, our method becomes a versatile navigation policy towards all different kinds of objects. Besides, our PixNav is a pure RGB-based policy that can reduce the cost of home-assistance robots. Experiments demonstrate the robustness of the PixNav which achieves 80+% success rate in the local path-planning task. To perform long-horizon object navigation, we design an LLM-based planner to utilize the commonsense knowledge between objects and rooms to select the best waypoint. Evaluations across both photorealistic indoor simulators and real-world environments validate the effectiveness of our proposed navigation strategy. Code and video demos are available at https://github.com/wzcai99/Pixel-Navigator.

Motivation & Objective

• To address the challenge of integrating foundation models with embodied navigation by eliminating the need for map-based planning and depth sensing.
• To develop a universal, RGB-based navigation policy that generalizes across diverse objects without requiring explicit semantic annotations.
• To enable long-horizon navigation in unseen environments using a hierarchical policy with an LLM planner.
• To reduce data collection cost by leveraging pixel-level supervision instead of object-level annotations.
• To validate the robustness of the navigation policy across varying camera configurations and real-world deployment.

Proposed method

• Propose a pixel-guided navigation policy (PixNav) that treats any object pixel as a goal, enabling zero-shot generalization to unseen objects.
• Use off-the-shelf 3D coordinate estimation to convert pixel goals into 3D waypoints for dataset collection via point-goal navigation.
• Design a transformer-based policy network with tracking prediction, temporal distance prediction, and goal fusion to improve long-horizon navigation stability.
• Integrate a large language model (LLM) with a step-by-step prompt template to perform hierarchical planning across rooms using commonsense knowledge.
• Train the PixNav policy end-to-end on large-scale, pixel-annotated trajectories collected from diverse object regions without explicit object labels.
• Enable sim-to-real transfer using only RGB observations from a standard RGB camera, avoiding depth or localization dependencies.

Experimental results

Research questions

• RQ1 Can a pure RGB-based navigation policy achieve high generalization and robustness across diverse objects and camera configurations?
• RQ2 Can pixel-level goal specification effectively bridge foundation models and embodied navigation without map-based planning?
• RQ3 Can an LLM-based planner with structured prompting enable effective long-horizon navigation in unseen environments?
• RQ4 How does the PixNav policy perform under distribution shift in camera height and field of view?
• RQ5 What components are most critical for maintaining performance in long-distance navigation (3–5m)?
• RQ6 What is the extent of spatial reasoning capability in LLMs for room localization and environmental mapping?

Key findings

• PixNav achieves a 88% success rate (SR) and 88% SPL on 1–3m navigation tasks in the HM3D simulator, demonstrating strong local path-planning performance.
• The policy generalizes across camera configurations, maintaining over 70% success rate even with 50% change in camera height and 25% change in HFOV.
• Ablation studies show that tracking prediction and goal fusion are critical for long-horizon navigation (3–5m), improving SR from 0.34 to 0.42.
• The LLM planner achieves 68.4% accuracy in room localization and 47.1% in room clustering, indicating measurable spatial reasoning capability.
• The system successfully transfers to real-world deployment using only RGB input on an iRobot Create3 robot, validating sim-to-real generalization.
• The method reduces data collection cost by eliminating the need for semantic annotations, enabling large-scale trajectory generation via pixel-level supervision.