[Paper Review] GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks

Muhammad Sohail Danish, Muhammad Akhtar Munir+6 more|arXiv (Cornell University)|Nov 28, 2024

Semantic Web and Ontologies4 citations

TL;DR

GEOBench-VLM introduces a specialized benchmark with over 10,000 manually verified instructions to evaluate Vision-Language Models (VLMs) on geospatial tasks such as scene understanding, object counting, localization, and temporal analysis. Despite strong performance on generic tasks, state-of-the-art VLMs like LLaVA-OneVision achieve only 41.7% accuracy on multiple-choice questions, indicating significant room for improvement in geospatial reasoning and perception.

ABSTRACT

While numerous recent benchmarks focus on evaluating generic Vision-Language Models (VLMs), they do not effectively address the specific challenges of geospatial applications. Generic VLM benchmarks are not designed to handle the complexities of geospatial data, an essential component for applications such as environmental monitoring, urban planning, and disaster management. Key challenges in the geospatial domain include temporal change detection, large-scale object counting, tiny object detection, and understanding relationships between entities in remote sensing imagery. To bridge this gap, we present GEOBench-VLM, a comprehensive benchmark specifically designed to evaluate VLMs on geospatial tasks, including scene understanding, object counting, localization, fine-grained categorization, segmentation, and temporal analysis. Our benchmark features over 10,000 manually verified instructions and spanning diverse visual conditions, object types, and scales. We evaluate several state-of-the-art VLMs to assess performance on geospatial-specific challenges. The results indicate that although existing VLMs demonstrate potential, they face challenges when dealing with geospatial-specific tasks, highlighting the room for further improvements. Notably, the best-performing LLaVa-OneVision achieves only 41.7% accuracy on MCQs, slightly more than GPT-4o, which is approximately double the random guess performance. Our benchmark is publicly available at https://github.com/The-AI-Alliance/GEO-Bench-VLM .

Motivation & Objective

• Address the lack of specialized benchmarks for evaluating Vision-Language Models (VLMs) on geospatial applications.
• Identify and quantify the performance gaps of existing VLMs on geospatial-specific challenges such as tiny object detection, large-scale counting, and temporal change detection.
• Provide a standardized, diverse, and manually verified benchmark to enable systematic evaluation and advancement of VLMs in remote sensing and geospatial AI.
• Highlight the limitations of current VLMs when applied to real-world geospatial tasks, despite strong performance on generic vision-language benchmarks.

Proposed method

• Design a comprehensive benchmark with over 10,000 manually verified instructions covering diverse visual conditions, object types, and scales.
• Structure the benchmark around core geospatial tasks: scene understanding, object counting, localization, fine-grained categorization, segmentation, and temporal analysis.
• Incorporate complex geospatial challenges such as detecting small objects, tracking changes over time, and reasoning about spatial relationships in satellite imagery.
• Use a variety of VLMs—including LLaVA-OneVision, GPT-4o, and others—for evaluation across the benchmark's tasks.
• Define standardized evaluation protocols with metrics like accuracy for multiple-choice questions and task-specific scores for localization and segmentation.
• Ensure data diversity by including varied geographic regions, image resolutions, and temporal sequences to reflect real-world geospatial data complexity.

Experimental results

Research questions

• RQ1 How well do existing Vision-Language Models generalize to geospatial-specific tasks such as object counting and localization in remote sensing imagery?
• RQ2 What are the key failure modes of current VLMs when confronted with geospatial challenges like tiny object detection and temporal change analysis?
• RQ3 To what extent do state-of-the-art VLMs outperform random baseline performance on geospatial reasoning tasks?
• RQ4 How does the performance of VLMs vary across different geospatial tasks, and which tasks pose the greatest challenges?
• RQ5 Can a standardized benchmark improve the evaluation and development of VLMs tailored for geospatial applications?

Key findings

• The best-performing VLM, LLaVA-OneVision, achieved only 41.7% accuracy on multiple-choice questions, indicating substantial performance gaps in geospatial reasoning.
• GPT-4o performed slightly better than random guessing, achieving approximately double the random baseline performance, highlighting the difficulty of geospatial VLM tasks.
• Existing VLMs struggle significantly with tiny object detection and large-scale object counting, common challenges in remote sensing imagery.
• Temporal analysis and fine-grained categorization tasks showed particularly low performance, suggesting limitations in long-range and relational reasoning.
• The benchmark reveals that generic VLMs are not directly transferable to geospatial applications without significant adaptation and task-specific fine-tuning.
• The results underscore the need for new architectural and training strategies tailored to the unique demands of geospatial vision-language understanding.