A Unified Treatment of Agreement Coefficients and their Asymptotic Results: the Formula of the Weighted Mean of Weighted
- PDF / 697,368 Bytes
- 33 Pages / 439.37 x 666.142 pts Page_size
- 17 Downloads / 177 Views
A Unified Treatment of Agreement Coefficients and their Asymptotic Results: the Formula of the Weighted Mean of Weighted Ratios Haruhiko Ogasawara 1
# The Classification Society 2020
Abstract
A unified treatment of agreement coefficients for multiple raters is shown, where the chance-expected proportions of the Bennett et al.-type, Scott-type, its new variation, and Cohen-type are dealt with using full or lower-order agreement among raters. When only pairwise agreement is used for multiple raters, chance corrections of the Gwet-type and its new variation are also considered. For the unified treatment, Conger’s two formulas of the ratio of means and the mean of ratios are combined into a parent formula of the weighted mean of weighted ratios. The corresponding unified expressions of their new asymptotic results are presented. Keywords Cohen’s kappa . Asymptotic variance . Weighted agreement coefficients . Disagreement . The Cornish-Fisher expansion . Asymptotic cumulants
1 Introduction Agreement coefficients for ratings of subjects/objects by two raters using nominal or ordinal categories have a long history dating back to the 1940s (see Guttman 1946). One of the typical coefficients is Cohen’s (1960) kappa using chance-corrected proportions for agreement, which has an extension with weighted profiles (Cohen 1968), where a profile indicates a combination of ratings by two raters, e.g. Category B given by the first rater and Category A by the second rater. In this profile denoted by BA, when ordered categories A, B, C, D, and E are used, the ratings do not agree but show similar ones suggesting a weighted profile with a partial credit. Following Cohen’s seminal papers, the vast amount of literatures for agreement coefficients has appeared (see, e.g. Fleiss et al. 2003, Chapter 18; Zhao et al. 2013; Gwet 2014; Berry et al. 2018, Sections 4.5, 6.5, and 6.6). When there are more than two raters, agreement coefficients have been extended or investigated by Light (1971), Fleiss (1971), Hubert (1977), Landis and
* Haruhiko Ogasawara [email protected]
1
Otaru University of Commerce 3-5-21, Midori, Otaru 047-8501, Japan
Journal of Classification
Koch (1977a, 1977b), Conger (1980), and Gwet (2002, 2008a) among others (see also Berry et al. 2018, Section 6.6). In contrast to the large amount of literatures mentioned above, we have a less amount of literatures for statistical treatment of kappa statistics and their multiple-rater extensions though Cohen (1960), Equation (7)) gave a naive standard error of his kappa neglecting the randomness of the chance-expected proportions. The corresponding correct asymptotic standard errors (ASEs) of the sample kappa and weighted kappa were derived by Fleiss et al. (1969). The ASE of Fleiss’ (1971) coefficient for multiple raters was also given in his paper and its correction by Fleiss et al. (1979). The ASE of the weighted coefficient for m-wise agreement for m raters was given by Landis and Koch (1977a) as an application of the method by Grizzle et al. (1969) and Koc
Data Loading...
