[Paper Review] DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarios

Vinicius G. Goecks, Nicholas R. Waytowich|arXiv (Cornell University)|Jun 29, 2023

Disaster Management and Resilience14 citations

TL;DR

DisasterResponseGPT uses in-context learning with LLMs to rapidly generate and iteratively refine plans of action for disaster response, delivering multiple actionable options comparable to human plans.

ABSTRACT

The development of plans of action in disaster response scenarios is a time-consuming process. Large Language Models (LLMs) offer a powerful solution to expedite this process through in-context learning. This study presents DisasterResponseGPT, an algorithm that leverages LLMs to generate valid plans of action quickly by incorporating disaster response and planning guidelines in the initial prompt. In DisasterResponseGPT, users input the scenario description and receive a plan of action as output. The proposed method generates multiple plans within seconds, which can be further refined following the user's feedback. Preliminary results indicate that the plans of action developed by DisasterResponseGPT are comparable to human-generated ones while offering greater ease of modification in real-time. This approach has the potential to revolutionize disaster response operations by enabling rapid updates and adjustments during the plan's execution.

Motivation & Objective

• Motivate the need for faster plan-of-action development in time-critical disaster scenarios.
• Propose a framework that uses LLMs to generate valid plans of action quickly by incorporating disaster response guidelines in prompts.
• Demonstrate that LLM-generated plans are comparable in quality to human-created plans and are easily modifiable in real-time.

Proposed method

• Leverage in-context learning by prompting LLMs with a knowledge base (e.g., FEMA guidelines) and planning examples.
• Prompt the LLM to produce three plans of action containing objective, critical tasks, main/auxiliary operations, end states, and feasibility/appropriateness/suitability checks.
• Enable user interaction to select and iteratively refine the chosen plan via natural language feedback.
• Compare plans generated by multiple backend LLMs (GPT-3.5, GPT-4, Bard) against a human-generated plan.
• Generate optional visual sketches or instructions for sketches when backend supports multimodality (not achieved in this study).

Experimental results

Research questions

• RQ1 Can LLMs rapidly generate valid, actionable plans of action for disaster response scenarios?
• RQ2 How do LLM-generated plans compare to human plans in quality and adaptability under real-time refinement?
• RQ3 Do different LLM backends (GPT-3.5, GPT-4, Bard) produce comparable plans and what are the qualitative differences?
• RQ4 What are the limitations in multimodal output (e.g., sketches) and context size for LLM-based planning?

Key findings

• LLM-backed DisasterResponseGPT can generate multiple plans within seconds, with final plans ready after brief user interaction.
• Plans produced by GPT-3.5, GPT-4, and Bard are largely comparable in quality to the human-generated plan within the scenario.
• DisasterResponseGPT-3.5 most closely matched the human plan and uniquely allocated survivors to rescue tasks, unlike some baselines.
• Some backends tended to reuse the same asset across tasks, creating an impression of sequential task execution rather than parallel allocation.
• Sketch generation for plans was not achieved automatically by the LLMs in this study; sketches were either unavailable or required user creation.
• The framework demonstrates real-time plan modification potential, aiding rapid adjustments during execution.