Item position effects in listening but not in reading in the European Survey of Language Competences
- PDF / 669,898 Bytes
- 21 Pages / 439.642 x 666.49 pts Page_size
- 106 Downloads / 180 Views
Item position effects in listening but not in reading in the European Survey of Language Competences ´ Christiansen1 Andres
· Rianne Janssen1
Received: 27 December 2019 / Accepted: 5 October 2020 / © Springer Nature B.V. 2020
Abstract In contrast with the assumptions made in standard measurement models used in largescale assessments, students’ performance may change during the test administration. This change can be modeled as a function of item position in case of a test booklet design with item-order manipulations. The present study used an explanatory item response theory (IRT) framework to analyze item position effects in the 2012 European Survey on Language Competences. Consistent item position effects were found for listening but not for reading. More specifically, for a large subset of items, item difficulty decreased along with item position, which is known as a practice effect. The effect was found across all tested languages, although the effect sizes varied across items, test levels, and countries. Keywords Item position effects · Multilevel IRT · Large-scale assessments
1 Introduction International large-scale assessments, such as the Programme for International Student Assessment (PISA) or the Trends in International Mathematics and Science Study (TIMSS), make use of models from item response theory (IRT) to transform each student’s responses on a subset of the test items into a single-scale score. The same measurement scale is assumed to hold across test booklets within and across
Andr´es Christiansen
[email protected] Rianne Janssen [email protected] 1
Centre of Educational Effectiveness and Evaluation, Katholieke Universiteit Leuven, Leuven, Belgium
Educational Assessment, Evaluation and Accountability
years of administration. Hence, the correct modeling of the response data is a prerequisite for the validity of the comparisons of countries and students. It is known that in standardized assessments, the performance of test takers can be influenced by the position of the items in the test. In such cases, one may say that “measuring changes the measure” (p. 312; Knowles 1988). If such item position effects are not included in the measurement model, this may lead to distorted item parameter estimates (see, e.g., Oshima 1994, Wise 1986) and may be a possible threat to IRT’s local independence assumption (Hartig and Buchholz 2012). In the present study, it will be investigated whether item position effects are present in a large-scale European assessment on foreign language comprehension of 15-year-old students, and if they vary across educational systems. In the following, previous studies on the effect of item position on psychometric item properties are summarized first. Afterwards, the research design and modeling approach of the present study are discussed. This project has received funding from the European Union’s Framework Programme for Research and Innovation Horizon 2020 (2014-2020) under the Marie Skłodowska-Curie Grant Agreement No. 765400. 1.1 Item position
Data Loading...
