[論文レビュー] Can Large Language Models Generate Geospatial Code?

Shuyang Hou, Shen Zhangxiao+5 more|arXiv (Cornell University)|Oct 13, 2024

Geographic Information Systems Studies被引用数 5

ひとこと要約

本論文は GeoCode-Eval (GCE) を導入し、地理空間コードタスクにおける LLM を評価し、GeoCode-Bench を数千問規模で構築・評価し、事前学習/指示データの域特化コード生成の改善を示す。

ABSTRACT

With the growing demand for spatiotemporal data processing and geospatial modeling, automating geospatial code generation has become essential for productivity. Large language models (LLMs) show promise in code generation but face challenges like domain-specific knowledge gaps and "coding hallucinations." This paper introduces GeoCode-Eval (GCE), a framework for assessing LLMs' ability to generate geospatial code across three dimensions: "Cognition and Memory," "Comprehension and Interpretation," and "Innovation and Creation," distributed across eight capability levels. We developed a benchmark dataset, GeoCode-Bench, consisting of 5,000 multiple-choice, 1,500 fill-in-the-blank, 1,500 true/false questions, and 1,000 subjective tasks covering code summarization, generation, completion, and correction. Using GeoCode-Bench, we evaluated three commercial closed-source LLMs, four open-source general-purpose LLMs, and 14 specialized code generation models. We also conducted experiments on few-shot and zero-shot learning, Chain of Thought reasoning, and multi-round majority voting to measure their impact on geospatial code generation. Additionally, we fine-tuned the Code LLaMA-7B model using Google Earth Engine-related JavaScript, creating GEECode-GPT, and evaluated it on subjective tasks. Results show that constructing pre-training and instruction datasets significantly improves code generation, offering insights for optimizing LLMs in specific domains.

研究の動機と目的

• Automate automated geospatial code generation for spatiotemporal data processing and geospatial modeling.
• Define a structured evaluation framework (GeoCode-Eval) across cognition, comprehension, and creation with multiple capability levels.
• Create a large, diverse benchmark (GeoCode-Bench) for geospatial code tasks including code summarization, generation, completion, and correction.
• Explore the impact of few-shot/zero-shot learning, Chain of Thought, and multi-round voting on geospatial code generation.
• Assess the potential of fine-tuning domain-specific models (e.g., GEECode-GPT) for improved subjective task performance.

提案手法

• Develop GeoCode-Eval (GCE) as a three-dimension, eight-level framework for geospatial code capabilities.
• Construct GeoCode-Bench with 5,000 multiple-choice, 1,500 fill-in-the-blank, 1,500 true/false questions, and 1,000 subjective tasks across code-related tasks.
• Evaluate 3 commercial closed-source LLMs, 4 open-source general-purpose LLMs, and 14 specialized code-generation models on GeoCode-Bench.
• Experiment with few-shot and zero-shot prompts, Chain of Thought reasoning, and multi-round majority voting to assess improvements in geospatial code generation.
• Fine-tune Code LLaMA-7B on Google Earth Engine (GEE)–related JavaScript to create GEECode-GPT and test on subjective tasks.

実験結果

リサーチクエスチョン

• RQ1 How well do LLMs generate geospatial code across diverse task types (summarization, generation, completion, correction)?
• RQ2 What is the impact of prompt design, few-shot/zero-shot learning, and Chain of Thought reasoning on geospatial code performance?
• RQ3 Can domain-specific fine-tuning (e.g., GEECode-GPT) improve subjective geospatial coding tasks?
• RQ4 What is the value of a large, specialized benchmark (GeoCode-Bench) for guiding LLM improvements in geospatial domains.

主な発見

• Pre-training data and instruction tuning significantly influence geospatial code generation capability.
• Various LLM families show improvements with appropriate prompting and reasoning strategies.
• Few-shot/zero-shot setups and Chain of Thought can affect performance on geospatial coding tasks.
• A domain-tuned model (GEECode-GPT) was developed and evaluated on subjective tasks, indicating potential gains from targeted fine-tuning.
• The GeoCode-Bench benchmark provides a structured framework to measure geospatial code abilities across multiple task types.