Centrala begrepp
Large Language Models (LLMs) can assist novice analysts in creating UML use case models, class diagrams, and sequence diagrams, but they also have limitations in accurately identifying relationships between modeling elements.
Sammanfattning
The study explored how LLMs, such as ChatGPT, can aid novice analysts in creating three types of UML models: use case diagrams, class diagrams, and sequence diagrams. 45 undergraduate students majoring in Software Engineering participated in the experiment, where they were asked to create these UML models for a given case study with the help of LLMs.
The key findings are:
LLMs perform well in identifying specific modeling elements like actors, use cases, classes, and objects, but struggle more with accurately recognizing relationships between these elements.
The correctness rate was highest for sequence diagrams, followed by class diagrams and use case diagrams. This suggests LLMs are better at identifying object-centric elements compared to relationship-centric ones.
The format of LLM output affects the quality of the resulting UML models. Hybrid-created diagrams, where students combine LLM suggestions with their own modeling, achieved the highest average scores compared to fully auto-generated diagrams.
While LLMs can provide useful assistance, they do not guarantee that novice analysts can create fully compliant and correct UML models. Proper training and understanding of requirements analysis and modeling is still essential.
The findings provide insights for software engineering educators, students, and professionals on the current capabilities and limitations of using LLMs for requirements analysis and UML modeling tasks.
Statistik
The use case model has a 59.44% average correctness rate across the four evaluation criteria.
The class diagram has a 64.44% average correctness rate across the four evaluation criteria.
The sequence diagram has a 74.81% average correctness rate across the three evaluation criteria.
Citat
"LLMs generally excel in recognizing specific objects (such as classes or use cases) from natural language text. However, extracting and analyzing relationships is not LLM's strong suit."
"Conveying information to humans first ensures that this information is reviewed and corrected before transformation, further amplifying the advantage of hybrid-created diagrams."