Automated Assessment of the Quality of Peer Reviews using Natural Language Processing Techniques
- PDF / 4,152,003 Bytes
- 48 Pages / 439.642 x 666.49 pts Page_size
- 85 Downloads / 176 Views
Automated Assessment of the Quality of Peer Reviews using Natural Language Processing Techniques Lakshmi Ramachandran1 · Edward F. Gehringer2 · Ravi K. Yadav2
Published online: 11 January 2017 © International Artificial Intelligence in Education Society 2017
Abstract A review is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through metareview or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors’ submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review’s coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications. Lakshmi Ramachandran is currently working for A9.com, Palo Alto, California. Lakshmi Ramachandran
[email protected] Edward F. Gehringer [email protected] Ravi K. Yadav [email protected] 1
Pearson, Boulder, CO, USA
2
North Carolina State University, Raleigh, NC, USA
Int J Artif Intell Educ (2017) 27:534–581
535
Keywords Intelligent tutoring systems · Collaborative learning · Peer reviews
Introduction In recent years a considerable amount of research has been directed towards developing educational systems that foster collaborative learning. Collaborative learning systems provide an environment for students to interact with other students, exchange ideas, provide feedback and use the feedback to improve their own work. Systems such as Scaffolded Writing and Rewriting in the Discipline (SWoRD—now called Peerceptiv) (Cho and Schunn 2007) and Expertiza (Gehringer 2010) are web-based, peer-review systems, that allows students to exchange ideas and to build shared knowledge. The past few years have witnessed a growth in Massive Open Online Courses (MOOCs) such as Coursera and Udacity, which serve as platforms for webbased collaborative learning. MOOCs require a scalable means of assessment, and for material that cannot be assessed by multiple-choice tests, peer-review fills the bill. Text-based feedback helps authors identify mistakes in their work, and learn how to improve it. Students learn from giving feedback as well as from receiving it. Ra
Data Loading...
