[Paper Review] How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts

Beian Wang, Chong Wang+3 more|arXiv (Cornell University)|Apr 27, 2024

Business Process Modeling and Analysis4 citations

TL;DR

This study investigates how large language models (LLMs) assist novice software engineering students in creating UML use case diagrams, class diagrams, and sequence diagrams. Through an exploratory experiment with 45 undergraduates, the authors found that LLMs provide useful guidance and improve modeling efficiency, but exhibit limitations in accurately capturing relationships and handling format-specific constraints, highlighting the need for improved prompting and human oversight in educational and professional settings.

ABSTRACT

Since the emergence of GPT-3, Large Language Models (LLMs) have caught the eyes of researchers, practitioners, and educators in the field of software engineering. However, there has been relatively little investigation regarding the performance of LLMs in assisting with requirements analysis and UML modeling. This paper explores how LLMs can assist novice analysts in creating three types of typical UML models: use case models, class diagrams, and sequence diagrams. For this purpose, we designed the modeling tasks of these three UML models for 45 undergraduate students who participated in a requirements modeling course, with the help of LLMs. By analyzing their project reports, we found that LLMs can assist undergraduate students as novice analysts in UML modeling tasks, but LLMs also have shortcomings and limitations that should be considered when using them.

Motivation & Objective

• To examine the role of LLMs in supporting novice analysts during UML modeling tasks.
• To evaluate the effectiveness of LLMs in generating accurate and complete UML diagrams—specifically use case, class, and sequence diagrams.
• To identify the strengths and limitations of LLMs in assisting with requirements modeling for beginners.
• To inform software engineering education by assessing how LLMs can be used as auxiliary tools without replacing core modeling competencies.
• To guide future development of AI-assisted modeling tools by identifying key failure points, such as relationship misrecognition and format-related information loss.

Proposed method

• Conducted a controlled experiment with 45 undergraduate software engineering students in a requirements modeling course.
• Students were tasked with creating three types of UML models (use case, class, sequence diagrams) with LLM assistance using natural language prompts.
• Collected and manually analyzed student project reports, including both the final UML models and the human-LLM interaction logs.
• Evaluated model quality using a binary scoring system per criterion (correct/incorrect) to assess accuracy and completeness.
• Compared different LLM output formats (e.g., text vs. PlantUML) to assess their impact on modeling element recognition and diagram quality.
• Identified recurring errors and limitations in LLM-generated models, particularly in modeling relationships and maintaining structural fidelity.

Figure 1 : The process of the experiment

Experimental results

Research questions

• RQ1 To what extent can LLMs assist novice analysts in creating accurate and complete UML use case diagrams?
• RQ2 How effective are LLMs in supporting the creation of class diagrams, particularly in modeling class attributes, methods, and associations?
• RQ3 In what ways do LLMs support or hinder the generation of sequence diagrams, especially in modeling message flows and lifelines?
• RQ4 How do different LLM output formats (e.g., natural language vs. PlantUML) affect the quality and correctness of the resulting UML models?
• RQ5 What are the primary limitations of LLMs in UML modeling, and how do these impact the reliability of AI-generated models for novice users?

Key findings

• LLMs significantly assist novice analysts in generating UML models, providing useful suggestions and accelerating the modeling process.
• LLMs demonstrated strong performance in generating basic structural elements such as actors, classes, and messages, but struggled with complex relationships like associations and dependencies.
• The use of structured output formats like PlantUML improved the accuracy of model elements compared to free-text LLM outputs, reducing formatting and syntactic errors.
• A notable limitation was the LLM’s tendency to omit or misrepresent relationship-related elements, leading to incomplete or semantically incorrect diagrams.
• Information loss occurred when converting LLM-generated text descriptions into diagram formats, particularly in sequence and class diagrams.
• Students frequently needed to manually correct or rework LLM-generated models, indicating that LLMs are best used as assistants rather than autonomous modeling tools.

Figure 2 : Distribution of the participants with/without experience of using LLMs