[Paper Review] VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis

Chao Pang, Xingxing Weng+9 more|arXiv (Cornell University)|Mar 29, 2024

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes H2RSVLM, a versatile and honest remote sensing vision-language model enhanced by two key innovations: HqDC-1.4M, a large-scale, high-quality dataset with 1.4 million detailed image-caption pairs that improve spatial perception and object understanding, and RSSA, the first self-awareness dataset designed to reduce hallucinations by training models to recognize unanswerable questions. H2RSVLM achieves state-of-the-art performance on multiple RS benchmarks while demonstrating strong refusal capabilities on unanswerable queries.

ABSTRACT

This paper develops a Versatile and Honest vision language Model (VHM) for remote sensing image analysis. VHM is built on a large-scale remote sensing image-text dataset with rich-content captions (VersaD), and an honest instruction dataset comprising both factual and deceptive questions (HnstD). Unlike prevailing remote sensing image-text datasets, in which image captions focus on a few prominent objects and their relationships, VersaD captions provide detailed information about image properties, object attributes, and the overall scene. This comprehensive captioning enables VHM to thoroughly understand remote sensing images and perform diverse remote sensing tasks. Moreover, different from existing remote sensing instruction datasets that only include factual questions, HnstD contains additional deceptive questions stemming from the non-existence of objects. This feature prevents VHM from producing affirmative answers to nonsense queries, thereby ensuring its honesty. In our experiments, VHM significantly outperforms various vision language models on common tasks of scene classification, visual question answering, and visual grounding. Additionally, VHM achieves competent performance on several unexplored tasks, such as building vectorizing, multi-label classification and honest question answering. We will release the code, data and model weights at https://github.com/opendatalab/VHM .

Motivation & Objective

• To address the poor performance of generic large vision-language models (VLMs) in remote sensing (RS) due to domain-specific image characteristics and limited spatial perception.
• To overcome the limitations of existing RS-specific VLMs (RSVLMs), which suffer from insufficient training data and lack of robustness to unanswerable questions.
• To develop a large-scale, high-quality RS vision-language dataset (HqDC-1.4M) with detailed, attribute-rich captions to enhance model understanding and spatial reasoning.
• To introduce RSSA, the first dataset specifically designed to improve model self-awareness and reduce hallucinations in VLMs through unanswerable question examples.
• To build and evaluate H2RSVLM, a model that achieves both high helpfulness and honesty in RS image analysis tasks.

Proposed method

• Constructed HqDC-1.4M, a 1.4-million-image-caption dataset using Gemini-Vision to generate rich, detailed captions covering object types, attributes (color, shape, location), and dynamic features like seasonal changes.
• Designed prompt engineering strategies to elicit comprehensive, fluent, and attribute-rich descriptions from the VLM, enhancing spatial perception and localization capabilities.
• Created RSSA, a self-awareness dataset for RS VLMs by injecting unanswerable questions into standard visual question-answering tasks to train models to refuse incorrect generations.
• Fine-tuned the vision encoder (ViT from CLIP) and language head using a multi-task instruction dataset (RS-ClsQaGrd-Instruct) covering classification, VQA, and visual grounding tasks.
• Integrated HqDC-1.4M and RSSA into a unified training pipeline to jointly optimize for helpfulness and honesty in the H2RSVLM architecture.
• Evaluated the model on multiple public RS benchmarks, including RS-ClsQaGrd-Instruct, RS-Specialized-Instruct, and RSSA, using task-specific metrics.

Experimental results

Research questions

• RQ1 Can a large-scale, high-quality vision-language dataset significantly improve the spatial perception and object understanding capabilities of remote sensing VLMs?
• RQ2 To what extent can self-awareness training via unanswerable questions reduce hallucinations in RS VLMs?
• RQ3 How does the combination of detailed captioning and self-awareness data improve both helpfulness and honesty in a single VLM?
• RQ4 Can H2RSVLM generalize across diverse RS tasks, including classification, counting, visual grounding, and vectorization?
• RQ5 What is the performance of H2RSVLM on unanswerable questions compared to baseline models, and does it demonstrate reliable refusal behavior?

Key findings

• H2RSVLM achieved state-of-the-art performance on multiple RS benchmarks, including F1-score of 85.77% on GID and 72.20% on FBP for classification tasks.
• On the visual grounding task, H2RSVLM achieved a C-IoU of 70.41% on CrowdAI-val, demonstrating strong localization accuracy.
• The model achieved a mean absolute error (MAE) of 6.22 on the counting task, indicating strong object counting capability.
• H2RSVLM demonstrated robust refusal behavior on unanswerable questions from the RSSA benchmark, effectively reducing hallucinations.
• Visualizations show that H2RSVLM performs well on large objects in visual grounding but struggles with small objects, likely due to input resolution constraints (336×336).
• The model showed strong performance in multi-label landcover classification, image type identification, and ground sampling distance estimation, as validated by qualitative results.