[Paper Review] AviationGPT: A Large Language Model for the Aviation Domain

Liya Wang, Jason Chou+3 more|arXiv (Cornell University)|Nov 29, 2023

Topic Modeling4 citations

TL;DR

AviationGPT is a domain-specific large language model fine-tuned on curated aviation datasets using open-source LLaMA-2 and Mistral architectures, achieving over 40% performance gain in NLP tasks like question-answering and information extraction while delivering contextually accurate responses in aviation-specific contexts.

ABSTRACT

The advent of ChatGPT and GPT-4 has captivated the world with large language models (LLMs), demonstrating exceptional performance in question-answering, summarization, and content generation. The aviation industry is characterized by an abundance of complex, unstructured text data, replete with technical jargon and specialized terminology. Moreover, labeled data for model building are scarce in this domain, resulting in low usage of aviation text data. The emergence of LLMs presents an opportunity to transform this situation, but there is a lack of LLMs specifically designed for the aviation domain. To address this gap, we propose AviationGPT, which is built on open-source LLaMA-2 and Mistral architectures and continuously trained on a wealth of carefully curated aviation datasets. Experimental results reveal that AviationGPT offers users multiple advantages, including the versatility to tackle diverse natural language processing (NLP) problems (e.g., question-answering, summarization, document writing, information extraction, report querying, data cleaning, and interactive data exploration). It also provides accurate and contextually relevant responses within the aviation domain and significantly improves performance (e.g., over a 40% performance gain in tested cases). With AviationGPT, the aviation industry is better equipped to address more complex research problems and enhance the efficiency and safety of National Airspace System (NAS) operations.

Motivation & Objective

• To address the scarcity of aviation-specific large language models despite the abundance of unstructured aviation text data.
• To improve the utilization of technical aviation documentation, reports, and operational data through domain-adapted LLMs.
• To enhance performance in aviation NLP tasks such as question-answering, summarization, and data cleaning.
• To enable more efficient and safer operations within the National Airspace System (NAS) through advanced language understanding.
• To bridge the gap between general-purpose LLMs and the specialized linguistic demands of the aviation domain.

Proposed method

• Fine-tuning the open-source LLaMA-2 and Mistral LLM architectures on a curated collection of aviation-specific text datasets.
• Using a diverse set of aviation corpora, including technical manuals, NOTAMs, flight reports, and air traffic control communications.
• Applying continuous pre-training and domain adaptation techniques to align model behavior with aviation terminology and context.
• Employing prompt engineering and retrieval-augmented generation (RAG) techniques to improve factual consistency and response relevance.
• Validating model performance through benchmarking on multiple aviation NLP tasks, including information extraction and summarization.
• Optimizing inference efficiency and safety through structured prompting and domain-aware decoding strategies.

Experimental results

Research questions

• RQ1 Can a large language model fine-tuned on aviation-specific data significantly outperform general-purpose LLMs in domain-specific NLP tasks?
• RQ2 To what extent does domain adaptation improve factual accuracy and contextual relevance in aviation text generation and understanding?
• RQ3 How does AviationGPT perform across diverse aviation NLP workloads such as report summarization, data cleaning, and interactive querying?
• RQ4 What performance gains does AviationGPT achieve over baseline models in critical aviation applications?
• RQ5 Can AviationGPT effectively handle complex aviation terminology and unstructured documentation without fine-tuning per task?

Key findings

• AviationGPT achieved a performance gain of over 40% on tested aviation NLP tasks compared to baseline models.
• The model demonstrated high accuracy and contextual relevance in answering complex aviation-related questions using technical documentation.
• It effectively handled diverse NLP tasks, including summarization, information extraction, and report generation, with minimal prompt engineering.
• The model significantly improved data cleaning and interactive data exploration capabilities in aviation datasets.
• Fine-tuning on curated aviation corpora enhanced factual consistency and reduced hallucination in domain-specific responses.
• AviationGPT outperformed general-purpose LLMs in domain-specific benchmarks, confirming the value of domain-specific adaptation.