Model Performance Assessment
In previous chapters, we used prediction accuracy to evaluate classification models. However, having accurate predictions in one dataset does not necessarily imply that the model is perfect or that it will reproduce when tested on external data. We need a
- PDF / 1,900,721 Bytes
- 22 Pages / 439.37 x 666.142 pts Page_size
- 71 Downloads / 195 Views
Model Performance Assessment
In previous chapters, we used prediction accuracy to evaluate classification models. However, having accurate predictions in one dataset does not necessarily imply that the model is perfect or that it will reproduce when tested on external data. We need additional metrics to evaluate the model performance and to make sure it is robust, reproducible, reliable, and unbiased. In this chapter, we will discuss (1) various evaluation strategies for prediction, clustering, classification, regression, and decision trees; (2) visualization of ROC curves and performance tradeoffs; and (3) estimation of future performance, internal statistical cross-validation and bootstrap sampling.
14.1
Measuring the Performance of Classification Methods
As mentioned previously, classification model performances could not be evaluated by prediction accuracy alone. We make different classification models for different purposes. For example, in newborns screening for genetic defects we want the model to have as few true negatives as possible. We don’t want to classify anyone as “no defect” when they actually have a defect gene, since early treatment might alter the destiny of this newborn. We can use the following three types of data to evaluate the performance of a classifier model. • Actual class values (for supervised classification). • Predicted class values. • Estimated probability of the prediction. We are familiar with the first two cases. The last type of validation relies on the predict(model, test_data) function that we have talked about in previous classification and prediction chapters (Chaps. 7, 8, and 9). Let’s revisit the model and test data we discussed in Chap. 8; the Inpatient Head and Neck Cancer Medication data. © Ivo D. Dinov 2018 I. D. Dinov, Data Science and Predictive Analytics, https://doi.org/10.1007/978-3-319-72347-1_14
475
476
14
Model Performance Assessment
We will demonstrate prediction probability estimation using this case-study CaseStudy14_HeadNeck_Cancer_Medication.csv pred_raw
Data Loading...
