Evaluating time series forecasting models: an empirical study on performance estimation methods
- PDF / 2,283,952 Bytes
- 32 Pages / 439.37 x 666.142 pts Page_size
- 59 Downloads / 185 Views
Evaluating time series forecasting models: an empirical study on performance estimation methods Vitor Cerqueira1 · Luis Torgo1,2,3 · Igor Mozetič4 Received: 30 May 2019 / Revised: 1 June 2020 / Accepted: 25 August 2020 © The Author(s), under exclusive licence to Springer Science+Business Media LLC, part of Springer Nature 2020
Abstract Performance estimation aims at estimating the loss that a predictive model will incur on unseen data. This process is a fundamental stage in any machine learning project. In this paper we study the application of these methods to time series forecasting tasks. For independent and identically distributed data the most common approach is cross-validation. However, the dependency among observations in time series raises some caveats about the most appropriate way to estimate performance in this type of data. Currently, there is no consensual approach. We contribute to the literature by presenting an extensive empirical study which compares different performance estimation methods for time series forecasting tasks. These methods include variants of cross-validation, out-of-sample (holdout), and prequential approaches. Two case studies are analysed: One with 174 real-world time series and another with three synthetic time series. Results show noticeable differences in the performance estimation methods in the two scenarios. In particular, empirical experiments suggest that blocked cross-validation can be applied to stationary time series. However, when the time series are non-stationary, the most accurate estimates are produced by outof-sample methods, particularly the holdout approach repeated in multiple testing periods. Keywords Performance estimation · Model selection · Cross validation · Time series · Forecasting
Editors: Larisa Soldatova, Joaquin Vanschoren. * Vitor Cerqueira [email protected] Luis Torgo [email protected] Igor Mozetič [email protected] 1
LIAAD-INESC TEC, Porto, Portugal
2
University of Porto, Porto, Portugal
3
Dalhousie University, Halifax, Canada
4
Jozef Stefan Institute, Ljubljana, Slovenia
13
Vol.:(0123456789)
Machine Learning
1 Introduction Performance estimation denotes the process of using the available data to estimate the loss that a predictive model will incur in new, yet unseen, observations. Estimating the performance of a predictive model is a fundamental stage in any machine learning project. Practitioners carry out performance estimation to select the most appropriate model and its parameters. Crucially, the process of performance estimation is one of the most reliable approaches to analyse the generalisation ability of predictive models. Such analysis is important not only to select the best model, but also to verify that the respective model solves the underlying predictive task. Choosing an appropriate performance estimation method usually depends on the characteristics of the data set. When observations are independent and identically distributed (i.i.d.), cross validation is one of the most widely used approaches
Data Loading...
