Learning with mitigating random consistency from the accuracy measure
- PDF / 2,369,226 Bytes
- 35 Pages / 439.37 x 666.142 pts Page_size
- 43 Downloads / 196 Views
Learning with mitigating random consistency from the accuracy measure Jieting Wang1 · Yuhua Qian1 · Feijiang Li1 Received: 13 April 2020 / Revised: 27 July 2020 / Accepted: 19 September 2020 © The Author(s) 2020
Abstract Human beings may make random guesses in decision-making. Occasionally, their guesses may generate consistency with the real situation. This kind of consistency is termed random consistency. In the area of machine leaning, the randomness is unavoidable and ubiquitous in learning algorithms. However, the accuracy (A), which is a fundamental performance measure for machine learning, does not recognize the random consistency. This causes that the classifiers learnt by A contain the random consistency. The random consistency may cause an unreliable evaluation and harm the generalization performance. To solve this problem, the pure accuracy (PA) is defined to eliminate the random consistency from the A. In this paper, we mainly study the necessity, learning consistency and leaning method of the PA. We show that the PA is insensitive to the class distribution of classifier and is more fair to the majority and the minority than A. Subsequently, some novel generalization bounds on the PA and A are given. Furthermore, we show that the PA is Bayesrisk consistent in finite and infinite hypothesis space. We design a plug-in rule that maximizes the PA, and the experiments on twenty benchmark data sets demonstrate that the proposed method performs statistically better than the kernel logistic regression in terms of PA and comparable performance in terms of A. Compared with the other plug-in rules, the proposed method obtains much better performance. Keywords Random consistency · Accuracy · Pure accuracy · Bayes-risk consistent
Editors: Kee-Eung Kim, Vineeth N. Balasubramanian. * Yuhua Qian [email protected] Jieting Wang [email protected] Feijiang Li [email protected] 1
Institute of Big Data Science and Industry, Shanxi University, Taiyuan 030006, Shanxi Province, China
13
Vol.:(0123456789)
Machine Learning
1 Introduction In the process of decision-making, human beings may make random guesses without logical reasoning when they lack sufficient evidence or detailed knowledge. For instance, intern doctors are likely to diagnose patients with colds during flu season, and students are likely to choose a lucky option when faced with a difficult multiple-choices question. Sometimes, these random guesses may generate consistency with the real situation. We term this consistency the random consistency. In the area of machine learning, randomness is unavoidable and ubiquitous in constructing classifiers, such as collecting and labeling data, selecting the structures or parameters of models and even in setting random operations (Ghahramani 2015). The prediction results of the learning models may also contain the random consistency. The random consistency produces dishonest feedback, misleads the decision direction and harms the improvement of the generalization ability, especially when
Data Loading...
