Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach
- PDF / 596,999 Bytes
- 6 Pages / 595.276 x 790.866 pts Page_size
- 72 Downloads / 211 Views
SOFTWARE
Open Access
Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach Inga Hege1,2*, Isabel Kiesewetter3 and Martin Adler4
Abstract Background: The ability to compose a concise summary statement about a patient is a good indicator for the clinical reasoning abilities of healthcare students. To assess such summary statements manually a rubric based on five categories - use of semantic qualifiers, narrowing, transformation, accuracy, and global rating has been published. Our aim was to explore whether computer-based methods can be applied to automatically assess summary statements composed by learners in virtual patient scenarios based on the available rubric in real-time to serve as a basis for immediate feedback to learners. Methods: We randomly selected 125 summary statements in German and English composed by learners in five different virtual patient scenarios. Then we manually rated these statements based on the rubric plus an additional category for the use of the virtual patients’ name. We implemented a natural language processing approach in combination with our own algorithm to automatically assess 125 randomly selected summary statements and compared the results of the manual and automatic rating in each category. Results: We found a moderate agreement of the manual and automatic rating in most of the categories. However, some further analysis and development is needed, especially for a more reliable assessment of the factual accuracy and the identification of patient names in the German statements. Conclusions: Despite some areas of improvement we believe that our results justify a careful display of the computer-calculated assessment scores as feedback to the learners. It will be important to emphasize that the rating is an approximation and give learners the possibility to complain about supposedly incorrect assessments, which will also help us to further improve the rating algorithms. Keywords: Clinical reasoning, Summary statement, Virtual patients, Natural language processing, Machine learning
Background Clinical reasoning is a complex set of skills healthcare students have to acquire during their education. Apart from face-to-face teaching scenarios, such as bedside teaching, clinical reasoning can be trained with webbased virtual patients (VPs) [1]. VPs are scalable, allow * Correspondence: [email protected] 1 Medical Education Sciences, University of Augsburg, Augsburg, Germany 2 Institute for Medical Education, Klinikum der Ludwig-Maximilians-Universität München, Pettenkoferstr. 8a, 80336 Munich, Germany Full list of author information is available at the end of the article
for deliberate practice, and provide a safe environment in which neither students nor patients are harmed. CASUS [2] is a virtual patient software, that supports clinical reasoning training in multiple ways - with a variety of question types, a specific clinical reasoning tool [3], and the composition of a summary statement by the learners. Summ
Data Loading...
