Diversity in the impact of heterogeneities on recurrent networks performing a cognitive task

Background and motivation : Artificial recurrent networks are widely used as models to study the complex dynamics underlying biological neural networks during execution of cognitive tasks. However, most studies assume individual units in the recurrent network to be homogeneous repeating units, whereas real neurons exhibit several forms of heterogeneities. In this study, we designed and employed a systematic framework for quantitative assessment of the impact of neural heterogeneities on recurrent networks that were trained to perform a cognitive task. Methodology : Our framework involved training of a population of recurrent networks, differing in terms of their hyperparameters, to perform a cognitive task in the presence of six graded levels of intrinsic heterogeneities. We tested the impact of heterogeneities on several performance metrics that encompassed training performance, task-execution dynamics, and resilience to different forms of post-training heterogeneities (also introduced at different levels. Results : Our population-of-networks approach demonstrate that intrinsic heterogeneities impacted network performance and dynamics in diverse ways even if they were trained with the same training algorithm, convergence criteria, and task specifications. First, our analyses unveiled pronounced network-to-network variability in the dependence of training performance on the level of heterogeneity, in terms of the number of training trials required for learning and the error values associated with task performance. Second, the impact of training heterogeneities on network dynamics during task execution also manifested substantial variability across networks. Finally, our analyses revealed a prominent impact of different forms of post-training heterogeneities on performance errors and network dynamics. We observed progressive increases in errors as well as in trajectory deviations with graded increases in post-training heterogeneities. Importantly, we observed pronounced variability in how robustness to post-training heterogeneities depended on the level of training heterogeneities. Specifically, certain networks showed enhanced robustness to post-training heterogeneities when training heterogeneities were low, whereas others showed better robustness when training heterogeneities were high. Implications : The striking nature of network-to-network variability observed in our analyses strongly advocates a complex systems viewpoint to study the impact of neural heterogeneities on circuit function. Within such a complex system framework, where several functionally specialized subsystems interact with each other in non-random ways to yield collective performance of the task, we argue that the emphasis should not be on heterogeneities in individual components of neural circuits. Instead, we emphasize the need to focus on the global structure of different forms and degrees of heterogeneities across different components and a systematic assessment of how they interact with each other towards adapting and achieving collective function.