Situational judgment test validity: an exploratory model of the participant response process using cognitive and think-a
- PDF / 806,772 Bytes
- 12 Pages / 595.276 x 790.866 pts Page_size
- 6 Downloads / 121 Views
RESEARCH ARTICLE
Open Access
Situational judgment test validity: an exploratory model of the participant response process using cognitive and think-aloud interviews Michael D. Wolcott1,2,3* , Nikki G. Lobczowski3,4, Jacqueline M. Zeeman1 and Jacqueline E. McLaughlin1,3
Abstract Background: Situational judgment tests (SJTs) are used in health sciences education to measure examinee knowledge using case-based scenarios. Despite their popularity, there is a significant gap in the validity research on the response process that demonstrates how SJTs measure their intended constructs. A model of SJT response processes has been proposed in the literature by Robert Ployhart; however, few studies have explored and expanded the factors. The purpose of this study was to describe the factors involved in cognitive processes that examinees use as they respond to SJT items in a health professions education context. Methods: Thirty participants—15 student pharmacists and 15 practicing pharmacists—completed a 12-item SJT designed to measure empathy. Each participant engaged in a think-aloud interview while completing the SJT, followed by a cognitive interview probing their decision-making processes. Interviews were transcribed and independently coded by three researchers to identify salient factors that contributed to response processes. Results: The findings suggest SJT response processes include all four stages (comprehension, retrieval, judgment, and response selection) as initially proposed by Ployhart. The study showed factors from other published research were present, including job-specific knowledge and experiences, emotional intelligence, and test-taking. The study also identified new factors not yet described, including identifying a task objective in the scenario, assumptions about the scenario, perceptions about the scenario, and the setting of the item. Conclusions: This study provides additional SJT validity evidence by exploring participants’ response processes through cognitive and think-aloud interviews. It also confirmed the four-stage model previously described by Ployhart and identified new factors that may influence SJT response processes. This study contributes to the literature with an expanded SJT response process model in a health professions education context and offers an approach to evaluate SJT response processes in the future. Keywords: Cognitive interview, Empathy, Qualitative methodology, Response process, Situational judgment test, Think-aloud protocol, Validity
* Correspondence: [email protected] 1 The University of North Carolina Eshelman School of Pharmacy, 321 Beard Hall, Chapel Hill, NC 27599, USA 2 The University of North Carolina Adams School of Dentistry, Chapel Hill, NC, USA Full list of author information is available at the end of the article © The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appr
Data Loading...
