University of Oulu

R. Ren, J. W. Castro, A. Santos, O. Dieste and S. T. Acuña, "Using the SOCIO Chatbot for UML Modelling: A Family of Experiments," in IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 364-383, 1 Jan. 2023, doi: 10.1109/TSE.2022.3150720

Using the SOCIO chatbot for UML modelling : a family of experiments

Saved in:
Author: Ren, Ranci1; Castro, John W.2; Santos, Adrián3;
Organizations: 1Universidad Autónoma de Madrid, Madrid, Spain
2Universidad de Atacama, Copiapó, Chile
3University of Oulu, Oulu, Finland
4Universidad Politécnica de Madrid, Boadilla del Monte, Spain
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 2.3 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2022
Publish Date: 2023-03-02


Context: Recent developments in natural language processing have facilitated the adoption of chatbots in typically collaborative software engineering tasks (such as diagram modelling). Families of experiments can assess the performance of tools and processes and, at the same time, alleviate some of the typical shortcomings of individual experiments (e.g., inaccurate and potentially biased results due to a small number of participants).

Objective: Compare the usability of a chatbot for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately).

Method: We conducted a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool in academic settings.

Results: The student participants were faster at building class diagrams using the chatbot than with the online collaborative tool and more satisfied with SOCIO. Besides, the class diagrams built using the chatbot tended to be more concise —albeit slightly less complete.

Conclusion: Chatbots appear to be helpful for building class diagrams. In fact, our study has helped us to shed light on the future direction for experimentation in this field and lays the groundwork for researching the applicability of chatbots in diagramming.

see all

Series: IEEE transactions on software engineering
ISSN: 0098-5589
ISSN-E: 1939-3520
ISSN-L: 0098-5589
Volume: 49
Issue: 1
Pages: 364 - 383
DOI: 10.1109/TSE.2022.3150720
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
Funding: This work was supported in part by the Spanish Ministry of Science, Innovation and Universities research under Grant PGC2018-097265-B-I00, in part by MASSIVE under Grant RTI2018-095255-B-I00, and in part by the Madrid Region R&D programme under Grant FORTE, P2018/TCS-4314.
Copyright information: © The Author(s) 2022. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see