Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measure
- PDF / 1,185,307 Bytes
- 16 Pages / 595.276 x 790.866 pts Page_size
- 112 Downloads / 165 Views
Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measurement bias, and prediction bias Sabine Zinn 1 & Uta Landrock 2 & Timo Gnambs 2,3 Accepted: 4 September 2020 # The Author(s) 2020
Abstract Educational large-scale studies typically adopt highly standardized settings to collect cognitive data on large samples of respondents. Increasing costs alongside dwindling response rates in these studies necessitate exploring alternative assessment strategies such as unsupervised web-based testing. Before respective assessment modes can be implemented on a broad scale, their impact on cognitive measurements needs to be quantified. Therefore, an experimental study on N = 17,473 university students from the German National Educational Panel Study has been conducted. Respondents were randomly assigned to a supervised paperbased, a supervised computerized, and an unsupervised web-based mode to work on a test of scientific literacy. Mode-specific effects on selection bias, measurement bias, and predictive bias were examined. The results showed a higher response rate in web-based testing as compared to the supervised modes, without introducing a pronounced mode-specific selection bias. Analyses of differential test functioning showed systematically larger test scores in paper-based testing, particularly among low to medium ability respondents. Prediction bias for web-based testing was observed for one out of four criteria on studyrelated success factors. Overall, the results indicate that unsupervised web-based testing is not strictly equivalent to other assessment modes. However, the respective bias introduced by web-based testing was generally small. Thus, unsupervised web-based assessments seem to be a feasible option in cognitive large-scale studies in higher education. Keywords Modeeffect . Web-based testing . Computerizedtesting . Measurement invariance . Selection effect . Highereducation
Large-scale educational studies collect information on individuals’ domain-specific competencies and general cognitive abilities to study their relevance for educational choices and peoples’ successful participation in society (see Blossfeld, Maurice and Schneider, 2019; Reiss, Obersteiner, Heinze, Itzlinger-Bruneforth and Lin, 2019; Strietholt and Scherer, 2018). For example, the Programme for International Student Assessment (PISA; http://www.oecd.org/pisa/) and the Programme for the International Assessment of Adult Competencies (PIAAC; https://www.oecd.org/skills/piaac/) Electronic supplementary material The online version of this article (https://doi.org/10.3758/s13428-020-01480-7) contains supplementary material, which is available to authorized users. * Timo Gnambs [email protected] 1
German Institute for Economic Research, Berlin, Germany
2
Leibniz Institute for Educational Trajectories, Bamberg, Germany
3
Johannes Kepler University Linz, Linz, Austria
assess the competence levels of adolescents and adults from over 40 countries around the world in, among ot
Data Loading...
