TY  - JOUR
A1  - Dubinin, Igor
A1  - Effenberger, Felix
T1  - Fading memory as inductive bias in residual recurrent networks
T2  - Neural networks
N2  - Residual connections have been proposed as an architecture-based inductive bias to mitigate the problem of exploding and vanishing gradients and increased task performance in both feed-forward and recurrent networks (RNNs) when trained with the backpropagation algorithm. Yet, little is known about how residual connections in RNNs influence their dynamics and fading memory properties. Here, we introduce weakly coupled residual recurrent networks (WCRNNs) in which residual connections result in well-defined Lyapunov exponents and allow for studying properties of fading memory. We investigate how the residual connections of WCRNNs influence their performance, network dynamics, and memory properties on a set of benchmark tasks. We show that several distinct forms of residual connections yield effective inductive biases that result in increased network expressivity. In particular, those are residual connections that (i) result in network dynamics at the proximity of the edge of chaos, (ii) allow networks to capitalize on characteristic spectral properties of the data, and (iii) result in heterogeneous memory properties. In addition, we demonstrate how our results can be extended to non-linear residuals and introduce a weakly coupled residual initialization scheme that can be used for Elman RNNs.
KW  - Recurrent neural network
KW  - Inductive bias
KW  - Residual connection
KW  - Memory
Y1  - 2024
UR  - http://publikationen.ub.uni-frankfurt.de/frontdoor/index/index/docId/82945
UR  - https://nbn-resolving.org/urn:nbn:de:hebis:30:3-829451
SN  - 0893-6080
VL  - 173
IS  - 106179
PB  - Elsevier
CY  - Amsterdam
ER  -