[Paper Review] FloodBrain: Flood Disaster Reporting by Web-based Retrieval Augmented Generation with an LLM

Grace Colverd, Paul Darm+2 more|arXiv (Cornell University)|Nov 5, 2023

Disaster Management and Resilience12 citations

TL;DR

FloodBrain uses web-based retrieval augmented generation with LLMs to produce flood disaster reports, evaluated against ReliefWeb ground truth via G-EVAL, ROUGE, and human judgments, plus an ablation study.

ABSTRACT

Fast disaster impact reporting is crucial in planning humanitarian assistance. Large Language Models (LLMs) are well known for their ability to write coherent text and fulfill a variety of tasks relevant to impact reporting, such as question answering or text summarization. However, LLMs are constrained by the knowledge within their training data and are prone to generating inaccurate, or "hallucinated", information. To address this, we introduce a sophisticated pipeline embodied in our tool FloodBrain (floodbrain.com), specialized in generating flood disaster impact reports by extracting and curating information from the web. Our pipeline assimilates information from web search results to produce detailed and accurate reports on flood events. We test different LLMs as backbones in our tool and compare their generated reports to human-written reports on different metrics. Similar to other studies, we find a notable correlation between the scores assigned by GPT-4 and the scores given by human evaluators when comparing our generated reports to human-authored ones. Additionally, we conduct an ablation study to test our single pipeline components and their relevancy for the final reports. With our tool, we aim to advance the use of LLMs for disaster impact reporting and reduce the time for coordination of humanitarian efforts in the wake of flood disasters.

Motivation & Objective

• Motivate rapid, accurate flood impact reporting to aid humanitarian coordination amid increasing flood events.
• Develop a specialized pipeline that extracts, curates, and summarizes web-sourced information for flood reports.
• Address LLM hallucination risks by integrating verified passages, source citation, and human-in-the-loop checks.

Proposed method

• Define a key phrase (date and location) to trigger web search for flood sources.
• Expand queries with an LLM to retrieve additional relevant sources and filter by relevancy.
• Extract information from sources and feed into an LLM to answer a set of questions for the report.
• Summarize the extracted content into a coherent, final flood report with citations for traceability.
• Provide a web UI and a mapping component to display text, sources, and flood extent.

Experimental results

Research questions

• RQ1 How well do FloodBrain-generated flood reports align with human-written ReliefWeb reports across metrics like content coverage and factual accuracy?
• RQ2 What is the impact of different LLM backbones (GPT-4, GPT-3.5, PaLM-Text-Bison) on report quality?
• RQ3 Do pipeline components (enhanced search, source relevancy checks) improve report quality and/or reduce computational cost?
• RQ4 Can G-EVAL serve as a reliable surrogate for human evaluation in this domain?
• RQ5 What is the trade-off between report quality and computational efficiency in the FloodBrain pipeline?

Key findings

• GPT-4-based reports showed the highest overlap with human-authored ReliefWeb reports on average among the tested backbones.
• G-EVAL scores demonstrated the highest correlation with human annotators, suggesting its usefulness for evaluating flood reports.
• Ablation showed that removing LLM-assisted search degrades performance across ROUGE metrics, while removing source relevancy checks reduces computational cost with mixed ROUGE effects.
• Augmenting search with LLM-generated queries improves agreement with human reports.
• The full pipeline (including search, source relevance, and QA) accelerates report generation and reduces unnecessary LLM calls by filtering sources.