Iterative Design and Classroom Evaluation of Automated Formative Feedback for Improving Peer Feedback Localization
- PDF / 1,652,111 Bytes
- 41 Pages / 439.642 x 666.49 pts Page_size
- 84 Downloads / 256 Views
Iterative Design and Classroom Evaluation of Automated Formative Feedback for Improving Peer Feedback Localization Huy Nguyen1
· Wenting Xiong2 · Diane Litman3
© International Artificial Intelligence in Education Society 2017
Abstract A peer-review system that automatically evaluates and provides formative feedback on free-text feedback comments of students was iteratively designed and evaluated in college and high-school classrooms. Classroom assignments required students to write paper drafts and submit them to a peer-review system. When student peers later submitted feedback comments on the papers to the system, Natural Language Processing was used to automatically evaluate peer feedback quality with respect to localization (i.e., pinpointing the source of the comment in the paper being reviewed). These evaluations in turn triggered immediate formative feedback by the system, which was designed to increase peer feedback localization whenever a feedback submission was predicted to have a ratio of localized comments less than a threshold. System feedback was dynamically generated based on the results of localization prediction. Reviewers could choose to either revise their feedback comments to address the system’s feedback or could ignore the feedback. Our analysis of data from system logs demonstrates that our peer feedback localization prediction
Huy Nguyen
[email protected] Wenting Xiong [email protected] Diane Litman [email protected] 1
Department of Computer Science, University of Pittsburgh, Pittsburgh, PA 15260, USA
2
IBM Watson, Yorktown Heights, NY 10598, USA
3
Department of Computer Science and Learning Research and Development Center, University of Pittsburgh, Pittsburgh, PA 15260, USA
Int J Artif Intell Educ
model triggered the formative feedback with high precision, particularly when peer feedback comments were written by college students. Our findings also show that although students often incorrectly disagree with the system’s feedback, when they do revise their peer feedback comments, the system feedback was successful in increasing peer feedback localization (although the sample size was low). Finally, while most peer comments were revised immediately after the system feedback, the desired revision behavior also occurred further after such system feedback. Keywords Peer feedback · Feedback localization · Automated formative feedback
Introduction A typical peer assessment practice when learning to write is asking students to reciprocally review other students’ work and generate peer feedback. Peer feedback is an important alternative to teacher feedback and is used frequently because it enhances students’ learning by giving them learning opportunities in their roles as both author and reviewer while not increasing teacher workload (Kern et al. 2003; Cho et al. 2007; Lundstrom and Baker 2009; Cho and MacArthur 2011; Nicol et al. 2014). In the domain of writing, peer feedback is usually referred to as “peer review” or “peer assistance when writing” (Gielen et al. 2010). Peer feedback
Data Loading...
