[Paper Review] Using Large Language Models for Natural Language Processing Tasks in Requirements Engineering: A Systematic Guideline

Andreas Vogelsang, Jannik Fischbach|arXiv (Cornell University)|Feb 21, 2024

Software Engineering Techniques and Practices4 citations

TL;DR

This paper presents a systematic guideline for leveraging pre-trained Large Language Models (LLMs) in Requirements Engineering (RE) tasks, covering fundamentals of LLMs, three distinct LLM types (encoder-only, decoder-only, encoder-decoder), and practical fine-tuning strategies. It enables researchers and practitioners to select and adapt LLMs effectively for NLP tasks in RE with minimal labeled data, overcoming traditional data scarcity challenges.

ABSTRACT

Large Language Models (LLMs) are the cornerstone in automating Requirements Engineering (RE) tasks, underpinning recent advancements in the field. Their pre-trained comprehension of natural language is pivotal for effectively tailoring them to specific RE tasks. However, selecting an appropriate LLM from a myriad of existing architectures and fine-tuning it to address the intricacies of a given task poses a significant challenge for researchers and practitioners in the RE domain. Utilizing LLMs effectively for NLP problems in RE necessitates a dual understanding: firstly, of the inner workings of LLMs, and secondly, of a systematic approach to selecting and adapting LLMs for NLP4RE tasks. This chapter aims to furnish readers with essential knowledge about LLMs in its initial segment. Subsequently, it provides a comprehensive guideline tailored for students, researchers, and practitioners on harnessing LLMs to address their specific objectives. By offering insights into the workings of LLMs and furnishing a practical guide, this chapter contributes towards improving future research and applications leveraging LLMs for solving RE challenges.

Motivation & Objective

• To address the long-standing challenge of data scarcity in Requirements Engineering (RE) by enabling effective use of Large Language Models (LLMs) with minimal labeled data.
• To provide researchers and practitioners with a foundational understanding of LLMs, including word embeddings, attention mechanisms, and model architectures.
• To present a structured, decision-driven approach for selecting and applying the most suitable LLM type (encoder-only, decoder-only, encoder-decoder) for specific NLP tasks in RE.
• To guide users through systematic fine-tuning strategies, including prompt engineering, parameter-efficient adaptation, and hyperparameter tuning, while addressing key risks like overfitting and bias amplification.
• To support experimental and adaptive use of LLMs in RE by highlighting the dynamic nature of the LLM landscape and the availability of specialized pre-trained models.

Proposed method

• Introduces foundational NLP concepts including sparse, static, and contextual word embeddings, explaining their role in enabling LLMs to capture semantic meaning.
• Explains the self-attention mechanism and transformer architecture as core components enabling contextualized embeddings in modern LLMs.
• Classifies LLMs into three types: encoder-only (e.g., BERT), decoder-only (e.g., GPT), and encoder-decoder (e.g., T5), each suited to different NLP tasks in RE.
• Proposes a decision tree to guide selection of the optimal LLM type based on task requirements, such as text understanding, generation, or sequence-to-sequence tasks.
• Outlines fine-tuning techniques including prompt tuning, prefix tuning, and full fine-tuning, with emphasis on regularization and hyperparameter optimization to prevent overfitting and bias amplification.
• Highlights the use of open-source model repositories like Hugging Face, which host over 40,000 pre-trained models, enabling access to domain-specific or task-specific LLM variants.

Experimental results

Research questions

• RQ1 Which LLM architecture (encoder-only, decoder-only, or encoder-decoder) is most suitable for a given NLP task in Requirements Engineering?
• RQ2 How can researchers and practitioners effectively fine-tune pre-trained LLMs for RE tasks with limited labeled data?
• RQ3 What are the key risks—such as overfitting, bias amplification, and hyperparameter sensitivity—when fine-tuning LLMs for RE, and how can they be mitigated?
• RQ4 In what scenarios is fine-tuning LLMs impractical or inadvisable, and what alternative approaches should be considered?
• RQ5 How can the rapidly evolving ecosystem of pre-trained LLMs be leveraged effectively in RE research and practice?

Key findings

• LLMs significantly reduce the need for large labeled datasets in RE tasks, enabling high-quality automation even with fewer than 10,000 labeled examples, unlike traditional models that require >10,000 entries for effective performance.
• Contextualized embeddings, enabled by self-attention mechanisms in transformers, allow the same word to have different vector representations based on context, overcoming a key limitation of static embeddings.
• Fine-tuning LLMs is effective for RE tasks, but requires careful management of overfitting through techniques like dropout, weight decay, early stopping, and cross-validation.
• Bias amplification is a critical risk during fine-tuning, as model weights may reinforce existing biases in the fine-tuning data, necessitating careful data curation and auditing.
• Not all LLMs support fine-tuning; API-only models or applications with rapidly changing or user-specific data may render fine-tuning impractical or ineffective.
• The availability of over 40,000 pre-trained models on Hugging Face enables researchers to access specialized models for specific RE tasks, languages, or domains, enhancing performance and reducing reliance on base models.