Stochastic Weight Update in Neural Networks: Theoretical Study of Stochastic Neural Networks Learning

This book is focused on the modification of the Backpropagation Through Time algorithm and its implementation on the Recurrent Neural Networks. Our work is inspired and motivated by the results of the Salvetti and Wilamowski experiment focused on the introduction of stochasticity into Backpropagation algorithm on experiments with the XOR problem. The stochasticity can be embedded into different parts of the BP algorithm. We introduced and implemented different types of BP algorithm modifications, which gradually add more stochasticity to the BP algorithm. The goal of this book is to prove, that this stochastic modification is able to learn efficiently and the results are comparable to classical implementation. This stochasticity also brings a simpler implementation of the algorithm, than the classical one, which is especially useful on the Recurrent Neural Networks.