Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student- t models
- PDF / 581,534 Bytes
- 19 Pages / 439.37 x 666.142 pts Page_size
- 1 Downloads / 155 Views
Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models Paul-Christian Bürkner1
· Jonah Gabry2 · Aki Vehtari1
Received: 27 March 2020 / Accepted: 3 November 2020 © The Author(s) 2020
Abstract Cross-validation can be used to measure a model’s predictive accuracy for the purpose of model comparison, averaging, or selection. Standard leave-one-out cross-validation (LOO-CV) requires that the observation model can be factorized into simple terms, but a lot of important models in temporal and spatial statistics do not have this property or are inefficient or unstable when forced into a factorized form. We derive how to efficiently compute and validate both exact and approximate LOO-CV for any Bayesian non-factorized model with a multivariate normal or Student-t distribution on the outcome values. We demonstrate the method using lagged simultaneously autoregressive (SAR) models as a case study. Keywords Cross-validation · Pareto-smoothed importance-sampling · Non-factorized models · Bayesian inference · SAR models
1 Introduction In the absence of new data, cross-validation is a general approach for evaluating a statistical model’s predictive accuracy for the purpose of model comparison, averaging, or selection (Geisser and Eddy 1979; Hoeting et al. 1999; Ando and Tsay 2010; Vehtari and Ojanen 2012). One widely used variant of cross-validation is leave-one-out crossvalidation (LOO-CV), where observations are left out one at a time and then predicted based on the model fit to the remaining data. Predictive accuracy is evaluated by first computing a pointwise predictive measure and then taking the sum of these values over all observations to obtain a single measure of predictive accuracy (e.g., Vehtari et al. 2017). In this paper, we focus on the expected log predictive density (ELPD) as the measure of predictive accuracy. The ELPD takes the whole predictive distribution into
B
Paul-Christian Bürkner [email protected]
1
Department of Computer Science, Aalto University, Espoo, Finland
2
Applied Statistics Center and ISERP, Columbia University, New York, USA
123
P.-C. Bürkner et al.
account and is less focused on the bulk of the distribution compared to other common measures such as the root mean squared error (RMSE) or mean absolute error (MAE; see for details Vehtari and Ojanen 2012). Exact LOO-CV is costly, as it requires fitting the model as many times as there are observations in the data. Depending on the size of the data, complexity of the model, and estimation method, this can be practically infeasible as it simply requires too much computation time. For this reason, fast approximate versions of LOO-CV have been developed (Gelfand et al. 1992; Vehtari et al. 2017), most recently using Pareto-smoothed importance-sampling (PSIS; Vehtari et al. 2017, 2019). A standard assumption of any such fast LOO-CV approach using the ELPD is that the model over all observations has to have a factorized form. That is, the overall observation model should be represented as the
Data Loading...
