A Comparative Study of Neural Network Optimization Techniques
In the last years we developed ENZO, an evolutionary neural network optimizer which we compare in this study to standard techniques for topology optimization: optimal brain surgeon (OBS), magnitude based pruning (MbP), and unit-OBS, an improved algorithm
- PDF / 981,032 Bytes
- 5 Pages / 595.278 x 793.684 pts Page_size
- 83 Downloads / 220 Views
Abstract In the last years we developed ENZO, an evolutionary neural network optimizer which we compare in this study to standard techniques for topology optimization: optimal brain surgeon (OBS), magnitude based pruning (MbP), and unit-OBS, an improved algorithm deduced from OBS. The algorithms are evaluated on several benchmark problems. We conclude that using an evolutionary algorithm as meta-heuristic like ENZO does is currently the best available optimization technique with regard to network size and performance. We show that the time complexity of ENZO is similar to magnitude based pruning and unit-OBS, while achieving significantly smaller topologies. Standard OBS is outperformed in both size reduction and time complexity.
1
Optimization Techniques
Optimizing the topology of neural networks is an important task when one aims to get smaller and faster networks, as well as a better generalization performance. Moreover, automatical optimization avoids the time consuming search for a suitable topology. Techniques for optimizing neural network topologies can be divided into destructive algorithms like pruning or optimal brain surgeon and constructive algorithms, e.g. cascade correlation. Our evolutionary algorithm ENZO [2, 7] is a mixture of both methods, since it allows for reduction as well as for growing structures. We will show that ENZO surpasses magnitude pruning as well as unit-OBS by evolving topologies with significantly less size while using nearly the same amount of computing time. Moreover, the size of the evolved topologies is even smaller as constructed by the very time consuming incremental optimal brain surgeon algorithm, i.e., we get a better network size reduction using several orders of magnitudes less computing power. The main criteria for optimizing the network
G. D. Smith et al., Artificial Neural Nets and Genetic Algorithms © Springer-Verlag Wien 1998
topology is the size of the network, furthermore the time needed for the optimization and the classification error. A problem of pure network reduction algorithms lies in the fact that the smallest network achieving a learning error below a given error limit has not the best generalization performance in general. The tradeoff between network size and generalization capability can be balanced by ENZO using both criteria in the fitness function while MbP and OBS do not consider the generalization and achieve therefore worse generalizing networks. In order to keep the comparison clear, we do not evaluate this essential advantage of ENZO. Therefore, the only criteria is the size of the achieved neural network under the constraint that the learning error (and thereby the classification error) remains under a given error bound. The size of the network is measured by three parameters: number of input units, number of hidden units and number of weights. In typical applications of the multilayer perceptron model there are some redundant or even irrelevant input units which may decrease the generalization capability. Therefore, we are interested in
Data Loading...
