[Paper Review] Model Generation with LLMs: From Requirements to UML Sequence Diagrams

Alessio Ferrari, Sallam Abualhaija+1 more|arXiv (Cornell University)|Apr 9, 2024

Business Process Modeling and AnalysisBusiness, Management and Accounting3 citations

TL;DR

This study investigates ChatGPT's ability to generate UML sequence diagrams from natural language requirements, finding strong performance in understandability and standard compliance but significant issues in completeness and correctness—especially with ambiguous or inconsistent requirements. The key contribution is a structured framework of 23 issue categories and recommendations for iterative, domain-aware prompting to improve reliability in requirements engineering workflows.

ABSTRACT

Complementing natural language (NL) requirements with graphical models can improve stakeholders' communication and provide directions for system design. However, creating models from requirements involves manual effort. The advent of generative large language models (LLMs), ChatGPT being a notable example, offers promising avenues for automated assistance in model generation. This paper investigates the capability of ChatGPT to generate a specific type of model, i.e., UML sequence diagrams, from NL requirements. We conduct a qualitative study in which we examine the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Observations from the analysis of the generated diagrams have systematically been captured through evaluation logs, and categorized through thematic analysis. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their completeness and correctness with respect to the specified requirements often present challenges. This issue is particularly pronounced in the presence of requirements smells, such as ambiguity and inconsistency. The insights derived from this study can influence the practical utilization of LLMs in the RE process, and open the door to novel RE-specific prompting strategies targeting effective model generation.

Motivation & Objective

• To assess the reliability of ChatGPT in generating UML sequence diagrams from natural language requirements.
• To identify common quality issues in LLM-generated models, especially concerning completeness and correctness.
• To explore the impact of requirement quality—such as ambiguity and inconsistency—on model generation outcomes.
• To develop a structured framework of issues to guide future improvements in LLM-based model generation for requirements engineering.
• To propose practical prompting strategies that incorporate domain knowledge and human-in-the-loop refinement to enhance model fidelity.

Proposed method

• Conducted an exploratory qualitative study using 28 diverse requirements documents across different formats (e.g., 'shall' statements, user stories, use cases).
• Employed three experienced researchers to prompt ChatGPT with requirements and generate corresponding UML sequence diagrams.
• Collected evaluation logs documenting quality issues for each generated diagram, focusing on completeness, correctness, and structural fidelity.
• Performed thematic analysis on evaluation logs to categorize 23 recurring issues in model generation, such as missing elements or incorrect message sequences.
• Used a triangulated approach with independent assessment and Cohen’s kappa to ensure reliability of evaluation judgments.
• Explored the impact of contextual and domain-specific knowledge by introducing requirement variants and assessing model responses under evolving or ambiguous inputs.

Experimental results

Research questions

• RQ1 To what extent can ChatGPT generate UML sequence diagrams that are semantically correct and complete when prompted with natural language requirements?
• RQ2 How do requirements smells—such as ambiguity and inconsistency—affect the quality of LLM-generated sequence diagrams?
• RQ3 What are the most frequent types of errors or omissions in the generated diagrams, and how do they relate to input quality or domain complexity?
• RQ4 How can iterative, domain-aware prompting strategies improve the correctness and completeness of LLM-generated models in requirements engineering?
• RQ5 What role does contextual and tacit domain knowledge play in the reliability of LLM-based model generation, and how can it be effectively incorporated?

Key findings

• ChatGPT-generated sequence diagrams demonstrated strong understandability and adherence to UML standards, with consistent terminology alignment to the input requirements.
• Despite good standard compliance, the models frequently suffered from incompleteness, including missing participants, messages, or lifelines, especially in complex or ambiguous requirements.
• Correctness issues were prevalent, including incorrect message ordering, misrepresentation of interaction flows, and structural deviations from the specified behavior.
• Hallucinations and incorrect interpretations were observed, particularly when requirements lacked clarity or required technical domain knowledge for proper disambiguation.
• The presence of requirements smells—such as ambiguity and inconsistency—significantly increased the likelihood of model errors, highlighting the sensitivity of LLMs to input quality.
• Thematic analysis revealed 23 distinct categories of issues, with the most frequent involving missing elements, incorrect message sequences, and failure to model conditional or concurrent behaviors.