Using black-box performance models to detect performance regressions under varying workloads: an empirical study
- PDF / 1,112,640 Bytes
- 31 Pages / 439.642 x 666.49 pts Page_size
- 50 Downloads / 198 Views
Using black-box performance models to detect performance regressions under varying workloads: an empirical study Lizhi Liao1 · Jinfu Chen1 · Heng Li2 · Yi Zeng1 · Weiyi Shang1 · Jianmei Guo3 · Catalin Sporea4 · Andrei Toma4 · Sarah Sajedi4 Published online: 28 August 2020 © Springer Science+Business Media, LLC, part of Springer Nature 2020
Abstract Performance regressions of large-scale software systems often lead to both financial and reputational losses. In order to detect performance regressions, performance tests are typically conducted in an in-house (non-production) environment using test suites with predefined workloads. Then, performance analysis is performed to check whether a software version has a performance regression against an earlier version. However, the real workloads in the field are constantly changing, making it unrealistic to resemble the field workloads in predefined test suites. More importantly, performance testing is usually very expensive as it requires extensive resources and lasts for an extended period. In this work, we leverage black-box machine learning models to automatically detect performance regressions in the field operations of large-scale software systems. Practitioners can leverage our approaches to complement or replace resource-demanding performance tests that may not even be realistic in a fast-paced environment. Our approaches use black-box models to capture the relationship between the performance of a software system (e.g., CPU usage) under varying workloads and the runtime activities that are recorded in the readily-available logs. Then, our approaches compare the black-box models derived from the current software version with an earlier version to detect performance regressions between these two versions. We performed empirical experiments on two open-source systems and applied our approaches on a large-scale industrial system. Our results show that such black-box models can effectively and timely detect real performance regressions and injected ones under varying workloads that are unseen when training these models. Our approaches have been adopted in practice to detect performance regressions of a large-scale industry system on a daily basis.
Communicated by: Ali Ouni, David Lo, Xin Xia, Alexander Serebrenik and Christoph Treude This article belongs to the Topical Collection: Recommendation Systems for Software Engineering Lizhi Liao
l [email protected]
Extended author information available on the last page of the article.
Empirical Software Engineering (2020) 25:4130–4160
4131
Keywords Performance regression · Black-box performance models · Field workloads · Performance engineering
1 Introduction Many large-scale software systems (e.g., Amazon, Google, Facebook) provide services to millions or even billions of users every day. Performance regressions in such systems usually lead to both reputational and monetary losses. For example, a recent incident of performance regressions at Salesforce1 affected eight data centers, resulting in a negative impact
Data Loading...
