On the role of network dynamics for information processing in artificial and biological neural networks

Understanding how interactions in complex systems give rise to various collective behaviours has been of interest for researchers across a wide range of fields. However, despite many years of studying the complexity of interactions in such systems, there are still many open research questions. Recurrent neural networks (RNNs) are one example of such complex systems, where our knowledge is not complete, and thus interpretability is limited. In particular, the understanding of dynamic information processing in RNNs is of great interest, as it has direct application in many fields, from biology to computer science. In this work, we study the information processing capabilities of such neural networks using a dynamical systems approach. First, we review the recent progress that has been achieved in the field of deep learning, particularly in design of artificial neural networks (ANNs) such as feedforward neural networks (FNNs) and recurrent neural networks. Next, we use dynamical system analysis and show how Lyapunov exponents can be utilized to characterize fading memory in recurrent neural networks and analyze the impact which memory properties have on learning dynamics. Our results align well with recent literature, demonstrating that a dynamical systems approach can be successfully used to study properties of RNNs’ dynamics and their learning dynamics such as stability and convergence. Subsequently, we focus on the inductive biases of RNNs, in particular residual connections, a type of inductive bias often used in deep learning. We introduce a new class of RNNs, the so-called weakly coupled residual networks (WCRNN), and demonstrate how these networks can be equipped with different fading memory characteristics through the configuration of their residual connections. We experimentally investigate the dynamics and performance of these networks on benchmark datasets and theoretically via dynamical system analysis. We confirm theoretically predicted trade-off between performance and learning speed at criticality and explore which of heterogeneous, rotational and non-linear residuals result in best practical expressivity. Our results uncover the essential role that inductive biases play for the networks’ practical expressivity, in agreement with recent developments in design of recurrent neural networks. Next, we discuss the role of complex neural dynamics for information processing in the brain and discuss existing biophysical and mechanistic models that are commonly used to model such neural dynamics on different scales. We also discuss how phenomenological models can be utilized to study the feasibility of coding strategies, such as strategies based on multistable attractors and metastable states. We note that despite their complexity, many detailed mechanistic models of neural dynamics have not been shown to perform meaningful information processing, and we present arguments in favor of studying computationally powerful phenomenological models of neural dynamics, thereby motivating our study in next chapter. In the last chapter, we follow the proposed approach and investigate the dynamics of cortical circuits by incorporating general principles observed in experimental studies into computationally powerful recurrent networks. Motivated by recent advances in the field of deep learning, we introduce an RNN model composed of coupled harmonic oscillators (HORN) that is able to capture characteristic properties of neural dynamics. Our results support the idea that characteristic properties of neural dynamics as oscillations, synchronization, resonance, heterogeneity play a functional role in forming complex patterns crucial for information processing in the brain. Our experiments with the HORN model also suggest that the brain can potentially utilize metastable dynamics for efficient neural coding and associative learning. The results obtained with the biologically-inspired HORN model are in good agreement with our experiments on WCRNNs, showing the universal advantage of informed inductive biases. We believe that our findings present a valuable contribution to the long-standing debate on the functional role of the considered properties of neural dynamics. Overall, we think that our approach based on a dynamical systems analysis offers valuable insights into information processing in both artificial and biological recurrent neural networks.