Simple measures of uncertainty for model selection
- PDF / 455,552 Bytes
- 20 Pages / 439.37 x 666.142 pts Page_size
- 69 Downloads / 214 Views
Simple measures of uncertainty for model selection Xiaohui Liu1 · Yuanyuan Li2 · Jiming Jiang2 Received: 11 April 2020 / Accepted: 23 October 2020 © Sociedad de Estadística e Investigación Operativa 2020
Abstract We develop two simple measures of uncertainty for a model selection procedure. The first measure is similar in spirit to confidence set in parameter estimation; the second measure is focusing on error in model selection. The proposed methods are simpler, both conceptually and computationally, than the existing measures of uncertainty in model selection. We recognize major differences between model selection and traditional estimation or prediction problems, and propose reasonable frameworks, under which these measures are developed, and their theoretical properties are established. Empirical studies demonstrate performance of the proposed measures, their superiority over the existing methods, and their relevance to real-life applications. Keywords Average probability of coverage · Bootstrapping · Consistency · LogP measure · Model confidence set · Model selection · Uncertainty Mathematics Subject Classification 62A99
1 Introduction In a way, statistics is characterized by measures of uncertainty. In the modern era of data science, basic statistical concepts and tools, such as estimation, prediction, regression analysis, and associated computational software are becoming more and more familiar to researchers and practitioners outside statistics. But one important statistical concept is able to hold its place as a defining feature of what many consider distinguishes statistics from the rest of the data science. This is called measure of uncertainty.
Electronic supplementary material The online version of this article (https://doi.org/10.1007/s11749020-00737-9) contains supplementary material, which is available to authorized users.
B
Jiming Jiang [email protected]
1
Jiangxi University of Finance and Economics, Nanchang, China
2
University of California, Davis, Davis, USA
123
X. Liu et al.
For the most part, there are two types of measure of uncertainty (MOU). One has to do with the variation, or spread, of an estimator or predictor. This includes the variance (standard deviation), interquartile range, and confidence/prediction intervals/sets. The latter may also be viewed as an estimator, whose value is in the form of a range, or set, rather than a single number, or vector of numbers. The other type of MOU focuses on error of estimation, or prediction. This includes the mean squared error (MSE) in case of an estimator, or mean squared prediction error (MSPE) in case of a predictor. Along with parameter estimation, another important element of statistical model building is model selection (e.g., Lahiri 2001; Jiang and Nguyen 2015). However, unlike in estimation or prediction problems, MOU in model selection has not been extensively studied in the literature. Hansen et al. (2011) introduced model confidence set (MCS), which is defined as a set of candidate models that is constructed to have a g
Data Loading...
