[Paper Review] TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data
TEOChat is a vision-language model designed for temporal Earth observation (EO) data, enabling reasoning over sequences of satellite images to perform tasks like change detection and damage assessment. It outperforms prior models—including Video-LLaVA and GeoChat—on temporal reasoning tasks and achieves state-of-the-art zero-shot performance on change detection and question-answering benchmarks, rivaling specialist models.
Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than several specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image instruction-following model on scene classification, visual question answering, and captioning. We publicly release our data, model, and code at https://github.com/ermongroup/TEOChat .
Motivation & Objective
- To develop a generalist vision-language model capable of reasoning over temporal sequences of Earth observation (EO) imagery, addressing limitations of existing models that handle only single images or natural videos.
- To create a large-scale, instruction-following dataset for temporal EO tasks to enable effective fine-tuning of multimodal models.
- To demonstrate that a single unified model can match or exceed the performance of specialist models on diverse EO tasks requiring spatial and temporal reasoning.
- To enable zero-shot generalization on unseen temporal EO benchmarks, including change detection and question answering, without task-specific fine-tuning.
- To release the model, dataset (TEOChatlas), and code publicly to foster community research and application in Earth observation.
Proposed method
- TEOChat is built on a LLaVA-like architecture, combining a vision encoder (ViT-CLIP), a vision-language connector (MLP), and a large language model (Llama 2), fine-tuned on a curated instruction-following dataset.
- The model processes input as a sequence of EO images with natural language prompts, enabling it to generate context-aware, multimodal responses to complex temporal queries.
- A novel dataset, TEOChatlas, was constructed with 554,071 examples across 50+ temporal EO tasks, including change detection, damage assessment, and temporal scene classification, to train and evaluate the model.
- The training process uses instruction tuning with diverse natural language instructions, ensuring robust zero-shot generalization across tasks.
- Image identifiers (e.g., 'Image 1: <image>') are incorporated into prompts to improve alignment and performance on temporal referring tasks.
- Evaluation includes both zero-shot and few-shot benchmarks on real-world datasets like xBD, S2Looking, and QFabric, using metrics such as IoU, F1, and accuracy.
Experimental results
Research questions
- RQ1Can a single vision-language model effectively perform a wide range of temporal Earth observation tasks, including change detection and damage assessment, without task-specific fine-tuning?
- RQ2How does TEOChat's performance on temporal reasoning tasks compare to prior vision-language models trained on natural images or single EO images?
- RQ3To what extent can TEOChat generalize to unseen temporal EO benchmarks in zero-shot settings?
- RQ4What impact do image identifiers in prompts have on the model's ability to reason about temporal sequences and refer to specific images?
- RQ5Can a generalist vision-language model achieve performance comparable to specialist models on specific EO tasks like building change detection or damage classification?
Key findings
- TEOChat achieves state-of-the-art zero-shot performance on the xBD change detection and change question-answering benchmark, outperforming both GPT-4o and Gemini 1.5 Pro.
- On the QFabric dataset, TEOChat achieves 71.7% accuracy in region-based temporal question answering and 74.9% in temporal referring expression tasks when image identifiers are included in prompts.
- TEOChat outperforms GeoChat, a specialist single-image EO VLM, on multiple zero-shot single-image tasks, including visual question answering and scene classification.
- TEOChat rivals or exceeds the performance of specialist models on building change detection (F1: 0.85), damage classification (F1: 0.82), and semantic change detection (IoU: 0.78).
- Including image identifiers in prompts improves performance on temporal reasoning tasks by over 12 percentage points, demonstrating their importance for temporal grounding.
- TEOChat achieves strong performance on both temporal and single-image tasks, indicating it is a robust generalist model for Earth observation applications.
Better researchstarts right now
From reading papers to final review, dramatically reduce your research time.
No credit card · Free plan available
This review was created by AI and reviewed by human editors.