An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys ad
- PDF / 1,263,749 Bytes
- 10 Pages / 595.276 x 790.866 pts Page_size
- 67 Downloads / 156 Views
(2020) 20:251
RESEARCH ARTICLE
Open Access
An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours Andrew Copas1* , Sarah Burkill1,2, Fred Conrad3, Mick P. Couper3 and Bob Erens1,4
Abstract Background: In health research, population estimates are generally obtained from probability-based surveys. In market research surveys are frequently conducted from volunteer web panels. Propensity score adjustment (PSA) is often used at analysis to try to remove bias in the web survey, but empirical evidence of its effectiveness is mixed. We assess the ability of PSA to remove bias in the context of sensitive sexual health research and the potential of web panel surveys to replace or supplement probability surveys. Methods: Four web panel surveys asked a subset of questions from the third British National Survey of Sexual Attitudes and Lifestyles (Natsal-3). Five propensity scores were generated for each web survey. The scores were developed from progressively larger sets of variables, beginning with demographic variables only and ending with demographic, sexual identity, lifestyle, attitudinal and sexual behaviour variables together. The surveys were weighted to match Natsal-3 based on propensity score quintiles. The performance of each survey and weighting was assessed by calculating the average ‘absolute’ odds ratio (inverse of the odds ratio if less than 1) across 22 prespecified sexual behaviour outcomes of interest comparing the weighted web survey with Natsal-3. The average standard error across odds ratios was examined to assess the impact of weighting upon variance. Results: Propensity weighting reduced bias relative to Natsal-3 as more variables were added for males, but had little effect for females, and variance increased for some surveys. Surveys with more biased estimates before propensity weighting showed greater reduction in bias from adjustment. Inconsistencies in performance were evident across surveys and outcomes. For most surveys and outcomes any reduction in bias was only partial and for some outcomes the bias increased. Conclusions: Even after propensity weighting using a rich range of information, including some sexual behaviour variables, some bias remained and variance increased for some web surveys. Whilst our findings support the use of PSA for web panel surveys, the reduction in bias is likely to be partial and unpredictable, consistent with the findings from market research. Our results do not support the use of volunteer web panels to generate unbiased population health estimates. Keywords: Internet survey, Web survey, Survey methods, Sampling bias, Selection bias, Sexual behaviour, Propensity score adjustment
* Correspondence: [email protected] 1 Institute for Global Health, University College London, London, UK Full list of author information is available at the end of the article © The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 Internationa
Data Loading...
