Recurrent Neural Networks
The brain is a strongly recurrent structure. This massive recurrence suggests a major role of self-feeding dynamics in the processes of perceiving, acting and learning, and in maintaining the organism alive
- PDF / 427,684 Bytes
- 17 Pages / 439.37 x 666.142 pts Page_size
- 49 Downloads / 264 Views
Recurrent Neural Networks
11.1 Introduction The brain is a strongly recurrent structure. This massive recurrence suggests a major role of self-feeding dynamics in the processes of perceiving, acting, and learning, and in maintaining the organism alive. Recurrent networks harness the power of brain-like computing. There is at least one feedback connection in recurrent networks. When recurrent networks are used, the network size is significantly compact compared with feedforward networks for the same approximation accuracy of dynamic systems. The MLP is fundamentally limited in its ability to solve topological relation problems. Recurrent networks can also be used as associative memories to build attractors y p from input-output association x p , y p . The MLP is purely static and is incapable of processing time information. One can add a time window over the data to act as a memory for the past. In the applications of dynamical systems, we need to forecast an input at time t + 1 from the network state at time t. The resulting network model for modeling a dynamical process is referred to as a temporal association network. Temporal association networks must have a recurrent architecture so as to handle the time-dependent nature of the association. To generate a dynamic neural network, memory must be introduced. The simplest memory element is the unit time delay, which has the transfer function H (z) = z −1 . The simplest memory architecture is the tapped delay line consisting of a series of unit time delays. Tapped delay lines are the basis of traditional linear dynamical models such as finite impulse response (FIR) or infinite impulse response (IIR) models. An MLP may be made dynamic by introducing time delay loops to the input, hidden, and/or output layers. The memory elements can be either fully or sparsely interconnected. A network architecture incorporating time delays is the time-delay neural network [46]. Recurrent networks are dynamical systems with temporal state representations. They are computationally powerful, and can be used in many temporal processing models and applications. Moreover, since the recurrent networks are modeled by systems of ordinary differential equations, they are also suitable for digital K.-L. Du and M. N. S. Swamy, Neural Networks and Statistical Learning, DOI: 10.1007/978-1-4471-5571-3_11, © Springer-Verlag London 2014
337
338
11 Recurrent Neural Networks
implementation using standard software for integration of ordinary differential equations. The Hopfield model and the Cohen–Grossberg model are the two common recurrent network models. The Hopfield model can store information in a dynamically stable structure. The Boltzmann machine is a generalization of the Hopfield model. Recurrent networks can generally be classified into globally recurrent networks, in which feedback connections between every neurons are allowed, and locally recurrent, globally feedforward networks [44] with the dynamics realized inside neuron models. Both classes of models can be universal approximat
Data Loading...
