Evaluating e-learning satisfaction with a quali-quantitative analysis of the open-ended responses: a case study of a continuing medical education course

Camilla Lugli; Alfonso Mazzaccara; Debora Guerrera; Serenella Arecchi; Fortunato D'Ancona; Giulia Fadda; Stefania Giannitelli; Tommaso Filippini; Lucia Palandri; Elena Righi; Ughetta Maria Favazzi

doi:10.23750/abm.v96i5.17305

Evaluating e-learning satisfaction with a quali-quantitative analysis of the open-ended responses: a case study of a continuing medical education course

Authors

Camilla Lugli Clinical and Experimental Medicine PhD Program, Department of Biomedical, Metabolic and Neural Sciences, University of Modena and Reggio Emilia, Modena, Italy; Training Office, National Institute of Health (ISS), Rome, Italy https://orcid.org/0009-0002-9280-0509
Alfonso Mazzaccara Training Office, National Institute of Health (ISS), Rome, Italy
Debora Guerrera Training Office, National Institute of Health (ISS), Rome, Italy
Serenella Arecchi Training Office, National Institute of Health (ISS), Rome, Italy
Fortunato D'Ancona Department of Infectious Diseases, National Institute of Health (ISS), Rome, Italy https://orcid.org/0000-0002-9855-2924
Giulia Fadda Department of Infectious Diseases, National Institute of Health (ISS), Rome, Italy
Stefania Giannitelli Department of Infectious Diseases, National Institute of Health (ISS), Rome, Italy
Tommaso Filippini Section of Public Health, Department of Biomedical, Metabolic and Neural Sciences, University of Modena and Reggio Emilia, Modena, Italy; CREAGEN – Environmental, Genetic and Nutritional Epidemiology Research Center, Department of Biomedical, Metabolic and Neural Sciences, University of Modena and Reggio Emilia, Modena, Italy; School of Public Health, University of California Berkeley, Berkeley, CA, USA. https://orcid.org/0000-0003-2100-0344
Lucia Palandri Clinical and Experimental Medicine PhD Program, Department of Biomedical, Metabolic and Neural Sciences, University of Modena and Reggio Emilia, Modena, Italy; Section of Public Health, Department of Biomedical, Metabolic and Neural Sciences, University of Modena and Reggio Emilia, Modena, Italy https://orcid.org/0000-0001-9787-7965
Elena Righi Section of Public Health, Department of Biomedical, Metabolic and Neural Sciences, University of Modena and Reggio Emilia, Modena, Italy https://orcid.org/0000-0002-3469-7446
Ughetta Maria Favazzi Training Office, National Institute of Health (ISS), Rome, Italy https://orcid.org/0000-0002-1256-1572

Keywords:

Continuing medical education (CME), Satisfaction, Qualitative text analysis, Lexicometric analysis, e-learning

Abstract

Background and aim: Continuing education is essential in the health sector, ensuring professionals stay updated with advancements and best practices. The growing reliance on e-learning, accelerated by the COVID-19 pandemic, has improved accessibility and engagement along with the use of interactive methodologies. These factors showed to increase participant satisfaction. The Italian National Institute of Health (ISS) has implemented e-learning courses using problem-based learning (PBL) to foster active learning. This study evaluates the informative potential of the feedback survey (FS) used in ISS courses, focusing on a quali-quantitative analysis of participant feedback from a case study course.

Methods: The case study course was designed using PBL methodology and targeted healthcare professionals across all disciplines. Data on sociodemographic characteristics and FS responses to closed and open-ended questions were extracted from EDUISS, ISS e-learning platform. A combination of quantitative and quali-quantitative text analysis was conducted to examine responses and develop a thematic vocabulary.

Results: A Satisfaction Training Index, derived from 18 Likert-scale items, showed high overall satisfaction (mean: 4.4, SD: 0.6) among 47,017 respondents out of 68,805 total participants (70% females, 57% nurses). Text analysis of open-ended responses revealed predominant themes of positive evaluations and clarity of content and delivery. Suggestions focused on enhancing content depth and improving evaluation methods. Thematic analysis produced a thematic vocabulary of 4 macro-themes and 13 specific themes covering positive comments (90%) and suggestions (>70%).

Conclusions: Although overall satisfaction levels were high, the qualitative analysis of open-ended responses provided actionable insights for improving course delivery and tools.

References

1. Institute of Medicine (US) Committee on Planning a Continuing Health Professional Education Institute. Redesigning Continuing Education in the Health Professions [Internet]. Washington (DC): National Academies Press (US); 2010 [cited 2025 Mar 21]. Available from: http://www.ncbi.nlm.nih.gov/books/NBK219811/

2. Palandri L, Urbano T, Pezzuoli C, et al. The key role of public health in renovating Italian biomedical doctoral programs. Ann Ig. 2024;36(3):353–62. doi: 10.7416/ai.2024.2592

3. Lugli C, Ferrari E, Filippini T, et al. It’s (not) rocket science: public health communication experience as expressed by participants to an international workshop. Popul Med. 2024 Jul 26;6:1–7. doi: 10.18332/popmed/191254

4. Pressato L. The Italian way to Continuing Education in Medicine. Clin Chim Acta. 2002 May 21;319(2):155–9. doi: 10.1016/s0009-8981(02)00037-2

5. Ferrari E, Scannavini P, Palandri L, et al. Training in infection prevention and control: survey on the volume and on the learning demands of healthcare-associated infections control figures in the Emilia-Romagna Region (Northern Italy). Ann Ig. 2025;37(1):14–24. doi: 10.7416/ai.2024.2643

6. Righi E, Lauriola P, Ghinoi A, Giovannetti E, Soldati M. Disaster risk reduction and interdisciplinary education and training. Prog Disaster Sci. 2021;10. doi: 10.1016/j.pdisas.2021.100165

7. Di Biagio A, Taramasso L, Gustinetti G, et al. Missed opportunities to prevent mother-to-child transmission of HIV in Italy. HIV Med. 2019;20(5):330–6. doi: 10.1111/hiv.12728

8. Palandri L, Morgado M, Colucci ME, et al. Reorganization of active surveillance of acute flaccid paralysis (AFP) in Emilia-Romagna, Italy: A two-step public health intervention. Acta Biomed. 2020;91:85–91. doi: 10.23750/abm.v91i3-S.9436

9. Veronesi L, Colucci ME, Capobianco E, et al. Immunity status against poliomyelitis in young migrants: A seroprevalence study. Acta Biomed. 2019;90(9-S):28–34. doi: 10.23750/abm.v90i9-S.8700

10. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of e-learning in medical education. Acad Med. 2006;81(3):207–12. doi: 10.1097/00001888-200603000-00002

11. Ruggeri K, Farrington C, Brayne C. A global model for effective use and evaluation of e-learning in health. Telemed J E Health. 2013;19(4):312–21. doi: 10.1089/tmj.2012.0175

12. Ng V, Gupta A, Erlich D. Brought about by necessity: how the pandemic accelerated a transformation of continuing professional development. Educ Prim Care. 2022 Jan 2;33(1):2–5. doi: 10.1080/14739879.2021.1920474

13. Jeffries PR, Bushardt RL, DuBose-Morris R, et al. The Role of Technology in Health Professions Education During the COVID-19 Pandemic. Acad Med. 2022 Mar 1;97(3S):S104–9. doi: 10.1097/ACM.0000000000004523

14. Al-Azri H, Ratnapalan S. Problem-based learning in continuing medical education: review of randomized controlled trials. Can Fam Physician. 2014 Feb;60(2):157–65. PMID: 24522680

15. Barrows HS, Tamblyn RM. Problem-Based Learning: An Approach to Medical Education. Springer Publishing Company; 1980. 225 p. ISBN: 0826128424

16. Ridda I, Rafiq S, Rashid H, Koul P. Innovative Approaches in Continuing Medical Education: Revolutionizing the Path to Excellence. J Pub Health Comm Med. 2024;(0):1. doi: 10.5455/JPHCM.20240124073537

17. Bates R. A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Eval Program Plann. 2004;27(3):341–7. doi: 10.1016/j.evalprogplan.2004.04.011

18. Thepwongsa I, Muthukumar R, Sripa P, et al. The perspectives of learners at a public medical school on the evaluation of an online learning management system for degree and non-degree courses. Med Educ Online. 2024;29(1). doi: 10.1080/10872981.2023.2299535

19. Heydari MR, Taghva F, Amini M, Delavari S. Using Kirkpatrick’s model to measure the effect of a new teaching and learning methods workshop for health care staff. BMC Res Notes. 2019;12(1). doi: 10.1186/s13104-019-4421-y

20. Asoodar M, Vaezi S, Izanloo B. Framework to improve e-learner satisfaction and further strengthen e-learning implementation. Comput Human Behav. 2016;63:704–16. doi: 10.1016/j.chb.2016.05.060

21. Bynum AB, Irwin CA, Cohen B. Satisfaction with a distance continuing education program for health professionals. Telemed J E Health. 2010;16(7):776–86. doi: 10.1089/tmj.2010.0005

22. Baburajan V, de Abreu e Silva J, Pereira FC. Open vs closed-ended questions in attitudinal surveys – Comparing, combining, and interpreting using natural language processing. Transp Res Part C Emerg Technol. 2022 Apr 1;137:103589. doi: 10.1016/j.trc.2022.103589

23. Bringula R, Ulfa S, Miranda JPP, Atienza FAL. Text mining analysis on students’ expectations and anxieties towards data analytics course. Cogent Eng. 2022 Dec 31;9(1):2127469. doi: 10.1080/23311916.2022.2127469

24. Menaha R, Ananthi K. Reviewing the effectiveness of lexicon-based techniques for sentiment analysis in massive open online courses. Int J Data Sci Anal. 2024 Jun 19. doi: 10.1007/s41060-024-00585-y

25. Barbina D, Riccardo F, Di Pucchio A, et al. Public Health Workforce training Laboratorium: Pilot e-Learning course on CBE and PBL models. Eur J Public Health. 2022 Oct 1;32(Supplement_3):ckac131.091. doi: 10.1093/eurpub/ckac131.091

26. Alturkistani A, Lam C, Foley K, et al. Massive Open Online Course Evaluation Methods: Systematic Review. J Med Internet Res. 2020 Apr 27;22(4):e13851. doi: 10.2196/13851

27. Favazzi UM, Marconi M, Carbone P, et al. Evaluating the impact of distance learning on gender-affirming healthcare competence: knowledge acquisition and satisfaction among healthcare professionals in Italy. Front Public Health. 2024;12. doi: 10.3389/fpubh.2024.1393188

28. Bolasco S. Analisi multidimensionale dei dati. Metodi, strategie e criteri d’interpretazione. Carocci; 1999. 360 p. ISBN: 8843014013

29. Krippendorff K. Content Analysis: An Introduction to Its Methodology. SAGE Publications; 2018. 473 p. ISBN: 1412983150

30. Baggieri M, Barbina D, Marchi A, et al. Measles and rubella in Italy, e-learning course for health care workers. Ann Ist Super Sanita. 2019;55(4):386–91. doi: 10.4415/ANN_19_04_13

31. Stellacci E, Martinelli S, Carbone P, et al. Bridging the educational gaps of health professionals in oncogenomics: results from a pilot e-learning course. Front Med (Lausanne). 2024;11. doi: 10.3389/fmed.2024.1422163

32. Longhini J, Rossettini G, Palese A. Massive open online courses for nurses’ and healthcare professionals’ continuous education: a scoping review. Int Nurs Rev. 2021;68(1):108–21. doi: 10.1111/inr.12649

33. Longhini J, De Colle B, Rossettini G, Palese A. What knowledge is available on massive open online courses in nursing and academic healthcare sciences education? A rapid review. Nurse Educ Today. 2021;99. doi: 10.1016/j.nedt.2021.104812

34. Zhou R, Wang X, Zhang L, Guo H. Who tends to answer open-ended questions in an e-service survey? The contribution of closed-ended answers. Behav Inf Technol. 2017;36(12):1274–84. doi: 10.1080/0144929X.2017.1381165