Human-like Modulation Sensitivity Emerging through Optimization to Natural Sound Recognition

Abstract Natural sounds contain rich patterns of amplitude modulation (AM), which is one of the essential sound dimensions for hearing perception. The sensitivity of human hearing to AM measured by psychophysics takes diverse forms, from low-pass to high-pass, depending on the experimental conditions. Here, we address with a single framework the questions of why such patterns of AM sensitivity have emerged in the human auditory system and how they are realized by our neural mechanisms. Assuming that optimization for natural sound recognition has taken place during human evolution and development, we examined its effect on the formation of AM sensitivity by optimizing a computational model, specifically, a multi-layer (or deep) neural network, for natural sound recognition and simulating psychophysical experiments in which the model’s AM sensitivity was measured. Relatively higher layers in the optimized model exhibited qualitatively and quantitatively similar AM sensitivity to that of humans, even though the model was not designed to reproduce human-like AM sensitivity. The similarity of the model’s AM sensitivity to humans’ correlated with its sound recognition accuracy. Optimization of the model to degraded sounds revealed the necessity of natural AM patterns for the emergence of human-like AM sensitivity. Consistent results were observed from optimizations to two different types of natural sound. Moreover, simulated neurophysiological experiments on the same model revealed a correspondence between the model layers and the auditory brain regions that is based on the similarity of their neural AM tunings. The layers in which human-like psychophysical AM sensitivity emerged exhibited substantial neurophysiological similarity with the auditory midbrain and higher regions. These results suggest that the behavioral AM sensitivity of human hearing has emerged as a result of optimization for natural-sound recognition in the course of our evolution and/or development and that it is based on a stimulus representation encoded in the neural firing rates in the auditory midbrain and higher regions.