Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to updat
- PDF / 2,578,067 Bytes
- 13 Pages / 595.276 x 790.866 pts Page_size
- 70 Downloads / 182 Views
RESEARCH
Open Access
Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence Shivani M. Reddy1* , Sheila Patel2, Meghan Weyrich3, Joshua Fenton3 and Meera Viswanathan2
Abstract Background: The exponential growth of the biomedical literature necessitates investigating strategies to reduce systematic reviewer burden while maintaining the high standards of systematic review validity and comprehensiveness. Methods: We compared the traditional systematic review screening process with (1) a review-of-reviews (ROR) screening approach and (2) a semi-automation screening approach using two publicly available tools (RobotAnalyst and AbstrackR) and different types of training sets (randomly selected citations subjected to dual-review at the title-abstract stage, highly curated citations dually reviewed at the full-text stage, and a combination of the two). We evaluated performance measures of sensitivity, specificity, missed citations, and workload burden Results: The ROR approach for treatments of early-stage prostate cancer had a poor sensitivity (0.54) and studies missed by the ROR approach tended to be of head-to-head comparisons of active treatments, observational studies, and outcomes of physical harms and quality of life. Title and abstract screening incorporating semi-automation only resulted in a sensitivity of 100% at high levels of reviewer burden (review of 99% of citations). A highly curated, smaller-sized, training set (n = 125) performed similarly to a larger training set of random citations (n = 938). Conclusion: Two approaches to rapidly update SRs—review-of-reviews and semi-automation—failed to demonstrate reduced workload burden while maintaining an acceptable level of sensitivity. We suggest careful evaluation of the ROR approach through comparison of inclusion criteria and targeted searches to fill evidence gaps as well as further research of semi-automation use, including more study of highly curated training sets.
Background Many clinical guideline and policy statements rely on a systematic evidence review (SR) that synthesizes the evidence base. SRs are labor-intensive projects due to high standards of validity and comprehensiveness. Guidance from the Institute of Medicine and the Effective Health Care (EHC) Program favor high sensitivity of literature searches and screening over specificity. All citations * Correspondence: [email protected] 1 RTI International, 307 Waverly Oaks Road, Suite 101, Waltham, MA 02452, USA Full list of author information is available at the end of the article
require dual review of abstracts and articles at each step of screening to identify and include all relevant research in the SR [1–3]. With the exponential growth of biomedical literature, a sensitive search can yield thousands of citations for dual abstract review and hundreds of articles for dual full-text review, whereas the number of articles ultimately included in the evidence review is typically much less. In this study, we examine s
Data Loading...
