[Paper Review] ALT-Pilot: Autonomous navigation with Language augmented Topometric maps

Mohammad Omama, Pranav Inani+5 more|arXiv (Cornell University)|Oct 3, 2023

Autonomous Vehicle Technology and Safety4 citations

TL;DR

ALT-Pilot enables autonomous navigation on road networks without HD LiDAR maps by using language-augmented topometric maps—OSM road networks enriched with natural language descriptions of landmarks. It leverages vision-language models for multimodal localization and achieves 3× higher localization accuracy and 5× better goal reachability than state-of-the-art systems like MapLite.

ABSTRACT

We present an autonomous navigation system that operates without assuming HD LiDAR maps of the environment. Our system, ALT-Pilot, relies only on publicly available road network information and a sparse (and noisy) set of crowdsourced language landmarks. With the help of onboard sensors and a language-augmented topometric map, ALT-Pilot autonomously pilots the vehicle to any destination on the road network. We achieve this by leveraging vision-language models pre-trained on web-scale data to identify potential landmarks in a scene, incorporating vision-language features into the recursive Bayesian state estimation stack to generate global (route) plans, and a reactive trajectory planner and controller operating in the vehicle frame. We implement and evaluate ALT-Pilot in simulation and on a real, full-scale autonomous vehicle and report improvements over state-of-the-art topometric navigation systems by a factor of 3x on localization accuracy and 5x on goal reachability

Motivation & Objective

• Address the scalability and cost limitations of HD LiDAR maps in autonomous driving systems.
• Enable accurate localization and navigation to arbitrary destinations on road networks without relying on metrically precise maps.
• Introduce a practical, low-footprint map representation using only language-annotated landmarks on topological road networks.
• Develop a robust, multimodal localization system that fuses LiDAR, RGB, and language modalities for real-time vehicle pose estimation.
• Enable open-vocabulary navigation using natural language goals, such as 'find a place to sit,' via language-landmark matching.

Proposed method

• Construct language-augmented topometric maps by enriching OpenStreetMap (OSM) road networks with geotagged, text-only descriptions of persistent landmarks (e.g., signs, signals).
• Precompute CLIP embeddings for each language landmark to enable cross-attention between vision and language modalities.
• At inference, extract per-pixel CLIP image embeddings from onboard RGB cameras and use them in a recursive Bayesian estimation framework to localize the vehicle.
• Fuse LiDAR-based road segmentation and costmaps with vision-language features to improve pose estimation robustness.
• Integrate the localization output into a global planning and reactive trajectory control stack for end-to-end navigation.
• Use a large language model (GPT-4) to parse open-ended natural language goals into structured landmark queries for goal-directed navigation.

Figure 2 : Pipeline : ALT-Pilot uses OSM maps augmented with language descriptions of landmarks. We precompute the CLIP (language) descriptors for each of these landmarks. At inference time, the LiDAR and Camera information is used to generate per-pixel CLIP (image) descriptors used in our multimoda

Experimental results

Research questions

• RQ1 Can autonomous navigation be achieved without HD LiDAR maps by leveraging only topological road networks and sparse language landmarks?
• RQ2 How effectively can vision-language models bridge visual and linguistic modalities to improve localization accuracy in real-world driving scenarios?
• RQ3 To what extent is the system robust to errors in landmark location, false positives, and incorrect language descriptions?
• RQ4 Can language-augmented maps enable open-vocabulary navigation to semantically described goals (e.g., 'a place to sit')?
• RQ5 How does multimodal fusion of LiDAR, RGB, and language improve localization precision and goal reachability compared to map-only or vision-only baselines?

Key findings

• ALT-Pilot achieves 3× higher localization accuracy than MapLite, with a mean absolute pose error (APE) of 3.72 m compared to MapLite’s 10.3 m on the CARLA1 map.
• In closed-loop navigation, ALT-Pilot reduces the average distance to goal by 5× compared to MapLite, achieving 1.57 m for non-intersection goals versus 14.37 m on CARLA1.
• The system remains robust to up to 50% false negative or false positive landmarks, maintaining APE below 5.89 m, while performance degrades significantly beyond 80% error.
• ALT-Pilot localizes and reaches goals faster, converging to the correct pose in 23.32 m on real-world data, compared to 73.1 m for MapLite.
• Language-guided navigation enables successful path planning to open-vocabulary goals like 'a place to sit' by matching them to relevant landmarks via LLM parsing.
• The addition of language landmarks significantly improves localization performance in grid-like urban environments, where landmark diversity increases along longer routes.

Figure 3 : Topological and Topometric localization on two CARLA maps. ALT-Pilot outperforms MapLite in both topological and topometric localization. Further, it’s more robust to noise in the map while MapLite is adversely affected by it. The impact of noise is more noticeable in the larger CARLA1 ma