[论文解读] Remote Sensing ChatGPT: Solving Remote Sensing Tasks with ChatGPT and Visual Models

Haonan Guo, Xin Su+4 more|arXiv (Cornell University)|Jan 17, 2024

Topic Modeling被引用 5

一句话总结

本文提出 Remote Sensing ChatGPT，一种由大模型驱动的代理，利用 ChatGPT 来规划任务并编排可视化遥感模型，通过自然语言反馈实现对遥感图像的解释。

ABSTRACT

Recently, the flourishing large language models(LLM), especially ChatGPT, have shown exceptional performance in language understanding, reasoning, and interaction, attracting users and researchers from multiple fields and domains. Although LLMs have shown great capacity to perform human-like task accomplishment in natural language and natural image, their potential in handling remote sensing interpretation tasks has not yet been fully explored. Moreover, the lack of automation in remote sensing task planning hinders the accessibility of remote sensing interpretation techniques, especially to non-remote sensing experts from multiple research fields. To this end, we present Remote Sensing ChatGPT, an LLM-powered agent that utilizes ChatGPT to connect various AI-based remote sensing models to solve complicated interpretation tasks. More specifically, given a user request and a remote sensing image, we utilized ChatGPT to understand user requests, perform task planning according to the tasks' functions, execute each subtask iteratively, and generate the final response according to the output of each subtask. Considering that LLM is trained with natural language and is not capable of directly perceiving visual concepts as contained in remote sensing images, we designed visual cues that inject visual information into ChatGPT. With Remote Sensing ChatGPT, users can simply send a remote sensing image with the corresponding request, and get the interpretation results as well as language feedback from Remote Sensing ChatGPT. Experiments and examples show that Remote Sensing ChatGPT can tackle a wide range of remote sensing tasks and can be extended to more tasks with more sophisticated models such as the remote sensing foundation model. The code and demo of Remote Sensing ChatGPT is publicly available at https://github.com/HaonanGuo/Remote-Sensing-ChatGPT .

研究动机与目标

• 激励为用户（包括非专家）实现遥感任务规划与解读的自动化。
• 使 ChatGPT 能理解用户请求并将其分解为可执行的遥感子任务。
• 通过在 ChatGPT 中的视觉提示，将自然语言处理与视觉遥感模型连接起来。
• 展示一个灵活的框架，能够通过更先进的遥感基础模型扩展到更多任务。

提出的方法

• 使用 ChatGPT 来理解用户请求，并基于任务函数进行任务规划。
• 通过调用独立的基于 AI 的遥感模型迭代执行子任务。
• 通过设计的视觉线索向 ChatGPT 注入可视信息，以弥补其缺乏直接图像感知的不足。
• 返回基于子任务输出的最终解读结果和语言反馈。
• 提供公开可用的代码和演示以说明工作流程。

实验结果

研究问题

• RQ1 由大语言模型驱动的代理是否能够正确解析遥感请求并规划一系列子任务？
• RQ2 在与可视模型配对时，视觉线索如何有效地使 ChatGPT 解释遥感图像？
• RQ3 该系统在多种遥感解译任务中的端到端性能如何？
• RQ4 该框架在引入更多遥感模型和任务方面的可扩展性如何？

主要发现

• 该框架能够在结合视觉模型的情况下，使用 ChatGPT 处理广泛的遥感任务。
• 视觉线索使 ChatGPT 能够在不直接感知图像视觉内容的情况下整合感知信息。
• 系统支持基于子任务输出产生的语言反馈和解读结果。
• 该方法具有可扩展性，可以通过更先进的遥感基础模型进行扩展。