phone +7 (3412) 91 60 92
mail imi@udsu.ru
ru-RU Russian
Home » Issues » Archive of Issues » 2018 - 2 issue » pp. 260-274

Archive of Issues

Russia Zelenograd
Year
2018
Volume
28
Issue
2
Pages
260-274
<<
Section Computer science
Title Neural networks with dynamical coefficients and adjustable connections on the basis of integrated backpropagation
Author(-s) Nazarov M.N.a
Affiliations National Research University of Electronic Technologya
Abstract We consider artificial neurons which will update their weight coefficients with an internal rule based on backpropagation, rather than using it as an external training procedure. To achieve this we include the backpropagation error estimate as a separate entity in all the neuron models and perform its exchange along the synaptic connections. In addition to this we add some special type of neurons with reference inputs, which will serve as a base source of error estimates for the whole network. Finally, we introduce a training control signal for all the neurons, which can enable the correction of weights and the exchange of error estimates. For recurrent neural networks we also demonstrate how to integrate backpropagation through time into their formalism with the help of some stack memory for reference inputs and external data inputs of neurons. Also, for widely used neural networks, such as long short-term memory, radial basis function networks, multilayer perceptrons and convolutional neural networks, we demonstrate their alternative description within the framework of our new formalism. As a useful consequence, our approach enables us to introduce neural networks with the adjustment of synaptic connections, tied to the integrated backpropagation.
Keywords artificial neurons, backpropagation, adaptive connection adjustment, recurrent neural networks
UDC 519.68, 007.5
MSC 68T05, 62M86
DOI 10.20537/vm180212
Received 22 May 2018
Language English
Citation Nazarov M.N. Neural networks with dynamical coefficients and adjustable connections on the basis of integrated backpropagation, Vestnik Udmurtskogo Universiteta. Matematika. Mekhanika. Komp'yuternye Nauki, 2018, vol. 28, issue 2, pp. 260-274.
References
  1. Dreyfus S.E. Artificial neural networks, back propagation, and the Kelley-Bryson gradient procedure, Journal of Guidance, Control and Dynamics, 1990, vol. 13, no. 5, pp. 926-928. DOI: 10.2514/3.25422
  2. Broomhead D.S., Lowe D. Multivariable functional interpolation and adaptive networks, Complex Systems, 1988, vol. 2, pp. 321-355. http://sci2s.ugr.es/keel/pdf/algorithm/articulo/1988-Broomhead-CS.pdf
  3. Lecun Y., Bottou L., Bengio Y., Haffner P. Gradient-based learning applied to document recognition, Proceedings of the IEEE, 1998, vol. 86, issue 11, pp. 2278-2324. DOI: 10.1109/5.726791
  4. Greff K., Srivastava R.K., Koutnik J., Steunebrink B.R., Schmidhuber J. LSTM: A search space odyssey, IEEE Transactions on Neural Networks and Learning Systems, 2017, vol. 28, issue 10, pp. 2222-2232. DOI: 10.1109/TNNLS.2016.2582924
  5. Chen G. A gentle tutorial of recurrent neural network with error backpropagation, 2016, arXiv: 1610.02583v3 [cs]. https://arxiv.org/pdf/1610.02583.pdf
  6. Krizhevsky A., Sutskever I., Hinton G.E. ImageNet classification with deep convolutional neural networks, Communications of the ACM, 2017, vol. 60, issue 6, pp. 84-90. DOI: 10.1145/3065386
  7. Girshick R., Donahue J., Darrell T., Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation, 2014, arXiv: 1311.2524v5 [cs]. https://arxiv.org/pdf/1311.2524.pdf
  8. Park J., Sandberg I.W. Universal approximation using radial-basis-function networks, Neural Computation, 1991, vol. 3, issue 2, pp. 246-257. DOI: 10.1162/neco.1991.3.2.246
  9. Pham V., Bluche T., Kermorvant C., Louradour J. Dropout improves recurrent neural networks for handwriting recognition, 2013, arXiv: 1312.4569v2 [cs]. https://arxiv.org/pdf/1312.4569.pdf
  10. Graves A. Generating sequences with recurrent neural networks, 2014, arXiv: 1308.0850v5 [cs]. https://arxiv.org/pdf/1308.0850.pdf
  11. Sutskever I., Vinyals O., Le Q.V. Sequence to sequence learning with neural networks, 2014, arXiv: 1409.3215v3 [cs]. https://arxiv.org/pdf/1409.3215.pdf
  12. Sak H., Senior A., Beaufays F. Long short-term memory recurrent neural network architectures for large scale acoustic modeling, Proceedings of the Annual Conference of the International Speech Communication Association, Singapore, 2014, pp. 338-342. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf
  13. Fan Y., Qian Y., Xie F., Soong F.K. TTS synthesis with bidirectional LSTM based recurrent neural networks, Proceedings of the Annual Conference of the International Speech Communication Association, Singapore, 2014, pp. 1964-1968. https://pdfs.semanticscholar.org/564f/ed868f652f361bb3e345f6f94073d8f6f261.pdf
  14. Donahue J., Hendricks L.A., Guadarrama S., Rohrbach M., Venugopalan S., Saenko K., Darrell T. Long-term recurrent convolutional networks for visual recognition and description, 2016, arXiv: 1411.4389v4 [cs]. https://arxiv.org/pdf/1411.4389.pdf
  15. Nazarov M.N. Artificial neural network with modulation of synaptic coefficients, Vestn. Samar. Gos. Tekhn. Univ., Ser. Fiz.-Mat. Nauki, 2013, vol. 2, no. 31, pp. 58-71. DOI: 10.14498/vsgtu1052
  16. Maslennikov O.V., Nekorkin V.I. Adaptive dynamical networks, Physics-Uspekhi, 2017, vol. 60, no. 7, pp. 694-704. DOI: 10.3367/UFNe.2016.10.037902
  17. Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R. Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 2014, vol. 15, pp. 1929-1958. http://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf
Full text
/doc/issues-2018/18-02-12.pdf
<< Previous article