Designing Crowdcritique Systems for Formative Feedback
- PDF / 1,755,305 Bytes
- 41 Pages / 439.37 x 666.142 pts Page_size
- 19 Downloads / 249 Views
Designing Crowdcritique Systems for Formative Feedback Matthew W. Easterday 1 & Daniel Rees Lewis 1 & Elizabeth M. Gerber 1
Published online: 18 November 2016 # International Artificial Intelligence in Education Society 2016
Abstract Intelligent tutors based on expert systems often struggle to provide formative feedback on complex, ill-defined problems where answers are unknown. Hybrid crowdsourcing systems that combine the intelligence of multiple novices in face-toface settings might provide an alternate approach for providing intelligent formative feedback. The purpose of this study was to develop empirically grounded design principles for crowdcritique systems that provide intelligent formative feedback on complex, ill-defined problems. In this design research project, we iteratively developed and tested a crowdcritique system through 3 studies of 43 novice problem solvers in 3 formal and informal learning environments. We collected observations, interviews, and surveys and used a grounded theory approach to develop and test socio-technical design principles for crowdcritique systems. The project found that to provide formative feedback on ill-defined problems, crowdcritique systems should provide a combination of technical features including: quick invite tools; formative framing; a public, near-synchronous social media interface; critique scaffolds; Blike^ system; community hashtags; analysis tools and Bto do^ lists; along with social practices including: prep/write-first/write-last script and critique training. Such a system creates a dualchannel conversation that increases the volume of quality critique by grounding comments, scaffolding and recording critique, and reducing production blocking. Such a design provides the benefits of both face-to-face critique and computer-
* Matthew W. Easterday [email protected] Daniel Rees Lewis [email protected] Elizabeth M. Gerber [email protected]
1
Northwestern University, Evanston, IL, USA
624
Int J Artif Intell Educ (2017) 27:623–663
support in both formal and informal learning environments while reducing the orchestration burden on instructors. Keywords Formative feedback . Complex problems . Critique . Crowdwork . Crowdsourcing . Project-based learning . Design principles . Social-technical systems Formative feedback from human experts is one of the most effective interventions for promoting learning (Hattie and Timperley 2007). Unfortunately, providing this feedback to learners is difficult to organize because experts are in high demand. As the number of learners increases, the orchestration challenge for experts becomes nearly impossible (Dillenbourg and Jermann 2010). The consequence is a failure to learn critical skills, knowledge, and attitudes. To overcome this problem, Artificial Intelligence in Education (AIED) researchers have embraced a 1-1 human tutoring paradigm in which intelligent tutoring systems, rather than human experts, provide feedback to learners on well defined problems such as solving Algebra equatio
Data Loading...
