[Paper Review] Evaluation of Code LLMs on Geospatial Code Generation

Piotr Gramacki, Bruno Martins+1 more|arXiv (Cornell University)|Oct 6, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper introduces a novel, manually curated benchmark for evaluating code generation Large Language Models (LLMs) in geospatial data science, categorizing tasks by complexity, spatial reasoning, and tool usage. The authors evaluate multiple 7B/8B LLMs on this dataset, revealing significant performance gaps—especially in using specialized geospatial libraries like OSMNX and MovingPandas—highlighting the need for domain-specific fine-tuning.

ABSTRACT

Software development support tools have been studied for a long time, with recent approaches using Large Language Models (LLMs) for code generation. These models can generate Python code for data science and machine learning applications. LLMs are helpful for software engineers because they increase productivity in daily work. An LLM can also serve as a "mentor" for inexperienced software developers, and be a viable learning support. High-quality code generation with LLMs can also be beneficial in geospatial data science. However, this domain poses different challenges, and code generation LLMs are typically not evaluated on geospatial tasks. Here, we show how we constructed an evaluation benchmark for code generation models, based on a selection of geospatial tasks. We categorised geospatial tasks based on their complexity and required tools. Then, we created a dataset with tasks that test model capabilities in spatial reasoning, spatial data processing, and geospatial tools usage. The dataset consists of specific coding problems that were manually created for high quality. For every problem, we proposed a set of test scenarios that make it possible to automatically check the generated code for correctness. In addition, we tested a selection of existing code generation LLMs for code generation in the geospatial domain. We share our dataset and reproducible evaluation code on a public GitHub repository, arguing that this can serve as an evaluation benchmark for new LLMs in the future. Our dataset will hopefully contribute to the development new models capable of solving geospatial coding tasks with high accuracy. These models will enable the creation of coding assistants tailored for geospatial applications.

Motivation & Objective

• To address the lack of standardized evaluation for code LLMs in geospatial data science, a domain with unique challenges in spatial reasoning and tool-specific libraries.
• To construct a high-quality, manually curated dataset of geospatial coding problems with automated test scenarios for correctness validation.
• To assess the capabilities and limitations of existing code generation LLMs across dimensions like spatial reasoning, data processing complexity, and library usage.
• To identify which types of geospatial tasks are most challenging for current LLMs, guiding future model development.
• To provide a reproducible, extensible benchmark to support the development of geospatial-aware AI coding assistants.

Proposed method

• The authors categorized geospatial tasks based on complexity (single-step vs. multi-step), required spatial reasoning, and specific geospatial tools (e.g., shapely, H3, OSMNX, MovingPandas).
• They manually created a dataset of coding problems with precise natural language descriptions and corresponding test scenarios to enable automated correctness checking.
• The benchmark includes structured inputs (e.g., latitude/longitude pairs or Shapely Point objects) to evaluate model sensitivity to data format.
• A comparative evaluation was conducted on a selection of 7B and 8B parameter code LLMs, including CodeLlama, Llama-3, Gemma, and Mistral.
• Model outputs were validated against predefined test cases to measure pass@1 accuracy across different task categories.
• The dataset and evaluation code are publicly released on GitHub to enable community adoption and extension.

Figure 1. Example of a prompt from the dataset, with an example solution generated (the highlighted part).

Experimental results

Research questions

• RQ1 RQ1: Are code generation LLMs capable of solving different types of geospatial tasks?
• RQ2 RQ2: Can code generation LLMs use spatial reasoning and world knowledge when solving geospatial tasks?
• RQ3 RQ3: Considering a broad categorisation of geospatial tasks, what types of problems are currently more challenging for code generation LLMs?
• RQ4 RQ4: How does model performance vary with input format (e.g., lat/lon vs. Shapely geometry objects)?
• RQ5 RQ5: To what extent can LLMs utilize specialized geospatial libraries such as OSMNX and MovingPandas?

Key findings

• Models showed significantly better performance when given Shapely geometry objects (e.g., 66.67% pass@1 for bigcode_starcoder2-7b) compared to raw lat/lon coordinates (33.33%).
• The best-performing model, meta-llama_CodeLlama-7b-Python-hf, achieved 23.81% pass@1 on lat/lon tasks and 26.67% on Shapely tasks.
• Models failed to generate correct code for OSMNX and MovingPandas libraries, often returning placeholder functions instead of actual implementations.
• Only the shapely and H3 libraries were reliably used, with pass@1 scores of 4.76% to 33.33% depending on the model.
• A substantial number of model outputs for MovingPandas and OSMNX tasks were incomplete or syntactically incorrect, indicating poor tool knowledge.
• The results suggest that current code LLMs lack sufficient exposure to geospatial libraries and require domain-specific fine-tuning to become effective assistants in geospatial development.