[Paper Review] Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study

Liuchang Xu, Shuo Zhao+7 more|arXiv (Cornell University)|Aug 26, 2024

Geographic Information Systems Studies4 citations

TL;DR

This study introduces a novel multi-task benchmark for evaluating large language models (LLMs) on spatial reasoning tasks, encompassing 12 spatial task types with verified answers. It evaluates models like GPT-4o and Moonshot-v1-8k using zero-shot and prompt-tuning strategies, showing that chain-of-thought prompting boosts GPT-4o’s path planning accuracy from 12.4% to 87.5%, highlighting the critical role of prompt engineering in spatial reasoning performance.

ABSTRACT

The emergence of large language models such as ChatGPT, Gemini, and others highlights the importance of evaluating their diverse capabilities, ranging from natural language understanding to code generation. However, their performance on spatial tasks has not been thoroughly assessed. This study addresses this gap by introducing a new multi-task spatial evaluation dataset designed to systematically explore and compare the performance of several advanced models on spatial tasks. The dataset includes twelve distinct task types, such as spatial understanding and simple route planning, each with verified and accurate answers. We evaluated multiple models, including OpenAI's gpt-3.5-turbo, gpt-4-turbo, gpt-4o, ZhipuAI's glm-4, Anthropic's claude-3-sonnet-20240229, and MoonShot's moonshot-v1-8k, using a two-phase testing approach. First, we conducted zero-shot testing. Then, we categorized the dataset by difficulty and performed prompt-tuning tests. Results show that gpt-4o achieved the highest overall accuracy in the first phase, with an average of 71.3%. Although moonshot-v1-8k slightly underperformed overall, it outperformed gpt-4o in place name recognition tasks. The study also highlights the impact of prompt strategies on model performance in specific tasks. For instance, the Chain-of-Thought (CoT) strategy increased gpt-4o's accuracy in simple route planning from 12.4% to 87.5%, while a one-shot strategy improved moonshot-v1-8k's accuracy in mapping tasks from 10.1% to 76.3%.

Motivation & Objective

• To address the lack of systematic evaluation of LLMs on spatial reasoning tasks.
• To develop a comprehensive, multi-task benchmark with verified ground-truth answers for spatial capabilities.
• To compare the performance of leading LLMs—such as GPT-3.5-turbo, GPT-4o, and Moonshot-v1-8k—across diverse spatial tasks.
• To investigate the impact of prompt engineering strategies, including chain-of-thought and one-shot prompting, on model performance.
• To provide insights into the strengths and limitations of current LLMs in geospatial understanding and reasoning.

Proposed method

• The study constructs a multi-task spatial benchmark with 12 distinct spatial reasoning task types, each with human-verified ground-truth answers.
• Models are evaluated in a two-phase testing protocol: first zero-shot, then with categorized difficulty and prompt tuning.
• Prompt engineering techniques such as Chain-of-Thought (CoT) and one-shot prompting are systematically applied to assess their impact on performance.
• Performance is measured using task-specific accuracy metrics across all 12 task types.
• Models are categorized by difficulty level to analyze performance trends across varying complexity levels.
• The evaluation includes state-of-the-art LLMs such as GPT-4o, GPT-3.5-turbo, Moonshot-v1-8k, and GLM-4, enabling cross-model comparison.

Experimental results

Research questions

• RQ1 How do leading large language models perform on a diverse set of spatial reasoning tasks without fine-tuning?
• RQ2 To what extent does prompt engineering, such as Chain-of-Thought, improve LLM performance on complex spatial tasks like path planning?
• RQ3 Which LLM demonstrates the highest overall accuracy across the multi-task benchmark?
• RQ4 How does model performance vary across different levels of task difficulty?
• RQ5 Are there specific spatial tasks—such as place name recognition or mapping—where certain models outperform others?

Key findings

• GPT-4o achieved the highest overall accuracy in the zero-shot evaluation phase, with an average of 71.3% across all tasks.
• Moonshot-v1-8k underperformed overall but surpassed GPT-4o in place name recognition tasks, indicating task-specific strengths.
• Chain-of-Thought prompting increased GPT-4o’s path planning accuracy from 12.4% to 87.5%, demonstrating a dramatic performance gain through prompting.
• One-shot prompting improved Moonshot-v1-8k’s accuracy in mapping tasks from 10.1% to 76.3%, highlighting the effectiveness of few-shot strategies in specific domains.
• The study confirms that prompt engineering strategies significantly influence model performance, especially on complex reasoning tasks.
• Performance varied substantially across task types, with path planning and spatial reasoning tasks showing the largest gains from advanced prompting techniques.