Feed-forward versus recurrent architecture and local versus cellular automata distributed representation in reservoir co
- PDF / 3,334,165 Bytes
- 30 Pages / 439.37 x 666.142 pts Page_size
- 76 Downloads / 191 Views
Feed‑forward versus recurrent architecture and local versus cellular automata distributed representation in reservoir computing for sequence memory learning Mrwan Margem1 · Osman S. Gedik2
© Springer Nature B.V. 2020
Abstract Reservoir computing based on cellular automata (ReCA) constructs a novel bridge between automata computational theory and recurrent neural networks. ReCA has been trained to solve 5-bit memory tasks. Several methods are proposed to implement the reservoir where the distributed representation of cellular automata (CA) in recurrent architecture could solve the 5-bit tasks with minimum complexity and minimum number of training examples. CA distributed representation in recurrent architecture outperforms the local representation in recurrent architecture (stack reservoir), then echo state networks and feedforward architecture using local or distributed representation. Extracted features from the reservoir, using the natural diffusion of CA states in the reservoir offers the state-of-the-art results in terms of feature vector length and the required training examples. Another extension is obtained by combining the reservoir CA states using XOR, Binary or Gray operator to produce a single feature vector to reduce the feature space. This method gives promising results, however using the natural diffusion of CA states still outperform. ReCA can be considered to operate around the lower bound of complexity; due to using the elementary CA in the reservoir. Keywords ReCA · Reservoir computing · Cellular automata · Recurrent architecture · Feed-forward architecture · Distributed representation · Local representation · 5-Bit memory task
* Mrwan Margem [email protected] Osman S. Gedik [email protected] 1
Department of Control Engineering, College of Electronic Technology-Tripoli, Al‑Jaraba Street, 21821 Tripoli, Libya
2
Department of Computer Engineering, Ankara Yildirim Beyazit University, 06220 Keçiören, Ankara, Turkey
13
Vol.:(0123456789)
M. Margem, O. S. Gedik
1 Introduction Many complex real-life systems require processing time-dependent data; therefore, these systems should remember the previous inputs, i.e., they must have a memory. The feed-forward Artificial Neural Networks (ANNs) cannot be used to simulate these types of systems; except if the feed-forward connections change to recurrent connections (there are feed-back connections), in this case, the new network is called recurrent neural network (RNN). RNNs are powerful tools for machine learning (ML) with memory Goodfellow et al. (2016), but it is very difficult to train them using traditional methods as gradient descent Bengio et al. (1994). To simplify the training of RNNs; they will be split into two layers: the untrained layer which is randomly and sparsely connected recurrent neurons called (a reservoir) and the trainable feed-forward ANNs (a read-out layer). This simplification became known as reservoir computing (RC) (Jaeger 2001; Maass et al. 2002). The high dimensional projection provided by the RNN reservoir can be s
Data Loading...
