Testing machine learning based systems: a systematic mapping
- PDF / 3,588,258 Bytes
- 62 Pages / 439.642 x 666.49 pts Page_size
- 4 Downloads / 233 Views
Testing machine learning based systems: a systematic mapping Vincenzo Riccio1 · Gunel Jahangirova1 · Andrea Stocco1 · Nargiz Humbatova1 · Michael Weiss1 · Paolo Tonella1
© The Author(s) 2020
Abstract Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks. Keywords Systematic mapping · Systematic review · Software testing · Machine learning
Communicated by: David Lo Vincenzo Riccio
[email protected]
Extended author information available on the last page of the article.
Empirical Software Engineering
1 Introduction Humanity long dreamed about reproducing intelligence within artificial machines. Back in 1872, the novelist S. Butler was the first to describe machines developing consciousness in his work entitled “Erewhon”. Scientists did not wait long to investigate in this direction: in 1950 Alan Turing proposed his famous operational test to verify a machine’s ability to exhibit intelligent behaviour indistinguishable from that of a human (Turing 2009). The advent of Machine Learning (ML) along with recent technological advancements allowed giant steps towards the realisation of this dream. Unlike traditional software systems, in which developers explicitly program the systems’ behaviour, ML entails techniques that mimic the human ability to automatically learn how to perform tasks through training examples (Mannin
Data Loading...
