Global and Individual Treatment Effects Using Machine Learning Methods
- PDF / 1,887,892 Bytes
- 28 Pages / 439.642 x 666.49 pts Page_size
- 52 Downloads / 196 Views
Global and Individual Treatment Effects Using Machine Learning Methods Bevan I. Smith1
3 ¨ · Charles Chimedza2 · Jacoba H. Buhrmann
© International Artificial Intelligence in Education Society 2020
Abstract Identifying students at risk of failing a course has potential benefits, such as recommending the At-Risk students to various interventions that could improve pass rates. The challenges however, are firstly in measuring how effective these interventions are, i.e. measuring treatment effects, and secondly, to not only predict overall (average) treatment effects, but also individual treatment effects. Previous work has used machine learning (ML) to estimate treatment effects by training models on treatment and control groups and using these models to predict counterfactuals. This is called the machine learning for predicting treatment effects, or MLTE method, in this study. This paper extends this previous work by (1) being the first to compare, or validate, the MLTE method, with the traditional propensity score matching (PSM) method; (2) using the MLTE method to compute not only global, but also subgroup and individual (heterogeneous) treatment effects and (3) by developing a method that combines the MLTE method with methods such as SHAP (SHapley Additive exPlanations), to predict final grades, predict individual treatment effects and characterize individual students. This is a comparative study between control and experimental groups, using observational data from a first year engineering mechanics course. Students could self-select to attend extra weekly tutorials (called the intervention), resulting in treatment and control groups. The aim was to measure the treatment effects of the intervention by comparing the difference in outcomes between treated and control groups. One of the main results was to develop a method for not only predicting final grades for subgroups and individuals, but also predicting treatment effects. This is valuable for predicting how a student/subgroup would perform and whether the intervention would benefit them. It allows for determining which types of students (or subgroups) would benefit from the intervention. This method shows the most promise for catering to individual students by not only predicting individual students’ final grades (mean absolute errors of between 6% and 8% were computed), but also
Bevan I. Smith
[email protected]
Extended author information available on the last page of the article.
International Journal of Artificial Intelligence in Education
predicting by how much the intervention would benefit the student (individual treatment effect, ITE) and by characterizing these students using SHAP. In terms of the ML models used, tree-based models performed the best in predictions. The method developed in this study therefore promises much for using ML to help individual students pass a course by identifying individuals that will receive the most benefit from the intervention. Keywords Machine learning · Treatment effects · Counterfactuals · Propensity score
Data Loading...
