Artificial neural networks training acceleration through network science strategies
- PDF / 616,635 Bytes
- 9 Pages / 595.276 x 790.866 pts Page_size
- 34 Downloads / 191 Views
FOCUS
Artificial neural networks training acceleration through network science strategies Lucia Cavallaro1
· Ovidiu Bagdasar1
· Pasquale De Meo2
· Giacomo Fiumara3
· Antonio Liotta4
© The Author(s) 2020
Abstract The development of deep learning has led to a dramatic increase in the number of applications of artificial intelligence. However, the training of deeper neural networks for stable and accurate models translates into artificial neural networks (ANNs) that become unmanageable as the number of features increases. This work extends our earlier study where we explored the acceleration effects obtained by enforcing, in turn, scale freeness, small worldness, and sparsity during the ANN training process. The efficiency of that approach was confirmed by recent studies (conducted independently) where a million-node ANN was trained on non-specialized laptops. Encouraged by those results, our study is now focused on some tunable parameters, to pursue a further acceleration effect. We show that, although optimal parameter tuning is unfeasible, due to the high non-linearity of ANN problems, we can actually come up with a set of useful guidelines that lead to speed-ups in practical cases. We find that significant reductions in execution time can generally be achieved by setting the revised fraction parameter (ζ ) to relatively low values. Keywords Network science · Artificial neural networks · Multilayer perceptron · Revise phase
1 Introduction The effort to simulate the human brain behaviour is one of the top scientific trends today. In particular, deep learning strategies pave the way to many new applications, thanks to their ability to manage complex architectures. Notable examples are: speech recognition (Hinton et al. 2012), cyber-security (Berman et al. 2019), image (Krizhevsky et al. 2017), and signal processing (Dong and Li 2011). Other applications gaining popularity are related to bio-medicine (Cao et al. Communicated by Yaroslav D. Sergeyev.
B
Lucia Cavallaro [email protected] Antonio Liotta [email protected]
1
University of Derby, Kedleston Road, Derby DE22 1GB, UK
2
University of Messina, Polo Universitario Annunziata, 98122 Messina, Italy
3
MIFT Department, University of Messina, 98166 Messina, Italy
4
Faculty of Computer Science, Free University of Bozen-Bolzano, Bolzano, Italy
2018) and drug discovery (Chen et al. 2018; Ruano-Ordás et al. 2019). However, despite their success, deep learning architectures suffer from important scalability issues, i.e., the actual artificial neural networks (ANN) become unmanageable as the number of features increases. While most current strategies focus on using more powerful hardware, the approach herein described employs network science strategies to tackle the complexity of ANNs iteratively, that is, at each epoch of the training process. This work originates in our earlier publication (Mocanu et al. 2018), a promising research avenue to speed up neural network training. There, a new approach called sparse evolutionary training (SET) was d
Data Loading...
