Parameter tuning or default values? An empirical investigation in search-based software engineering
- PDF / 547,472 Bytes
- 30 Pages / 439.37 x 666.142 pts Page_size
- 81 Downloads / 206 Views
Parameter tuning or default values? An empirical investigation in search-based software engineering Andrea Arcuri · Gordon Fraser
Published online: 24 February 2013 © Springer Science+Business Media New York 2013
Abstract Many software engineering problems have been addressed with search algorithms. Search algorithms usually depend on several parameters (e.g., population size and crossover rate in genetic algorithms), and the choice of these parameters can have an impact on the performance of the algorithm. It has been formally proven in the No Free Lunch theorem that it is impossible to tune a search algorithm such that it will have optimal settings for all possible problems. So, how to properly set the parameters of a search algorithm for a given software engineering problem? In this paper, we carry out the largest empirical analysis so far on parameter tuning in search-based software engineering. More than one million experiments were carried out and statistically analyzed in the context of test data generation for object-oriented software using the EvoSuite tool. Results show that tuning does indeed have impact on the performance of a search algorithm. But, at least in the context of test data generation, it does not seem easy to find good settings that significantly outperform the “default” values suggested in the literature. This has very practical value for both researchers (e.g., when different techniques are compared) and practitioners. Using “default” values is a reasonable and justified choice, whereas parameter tuning is a long and expensive process that might or might not pay off in the end.
Communicated by: Mel O. Cinneide This paper is an extension of a conference paper published in the International Symposium on Search Based Software Engineering (SSBSE) (Arcuri and Fraser 2011), 2011. A. Arcuri Certus Software V&V Center at Simula Research Laboratory, P.O. Box 134, Lysaker, Norway e-mail: [email protected] G. Fraser (B) Department of Computer Science, University of Sheffield, Regent Court, 211 Portobello, S1 4DP, Sheffield, UK e-mail: [email protected]
Empir Software Eng (2013) 18:594–623
595
Keywords Search-based software engineering · Test data generation · Object-oriented · Unit testing · Tuning · EvoSuite · Java · Response surface · Design of experiments
1 Introduction Recent years have brought a large growth of interest in search-based software engineering (SBSE) (Harman et al. 2009), especially in software testing (Ali et al. 2010). The field has even matured to a stage where industrial applications have started to appear (Vos et al. 2010; Arcuri et al. 2010). One of the key strengths of SBSE leading to this success is its ability of automatically solving very complex problems where exact solutions cannot be deterministically found in reasonable time. However, to make SBSE really usable in practice, no knowledge of search algorithms should be required from practitioners who want to use it, as such knowledge is highly specialized and might not be widespread. In other words, SBS
Data Loading...
