Neural Correlates of High-Level Visual Saliency Models

AbstractVisual saliency highlights regions in a scene that are most relevant to an observer. The process by which a saliency map is formed has been a crucial subject of investigation in both machine vision and neuroscience. Deep learning-based approaches incorporate high-level information and have achieved accurate predictions of eye movement patterns, the overt behavioral analogue of a saliency map. As such, they may constitute a suitable surrogate of cortical saliency computations. In this study, we leveraged recent advances in computational saliency modeling and the Natural Scenes Dataset (NSD) to examine the relationship between model-based representations and the brain. Our aim was to uncover the neural correlates of high-level saliency and compare them with low-level saliency as well as emergent features from neural networks trained on different tasks. The results identified hV4 as a key region for saliency computations, informed by semantic processing in ventral visual areas. During natural scene viewing, hV4 appears to serve a transformative role linking low- and high-level features to attentional selection. Moreover, we observed spatial biases in ventral and parietal areas for saliency-based receptive fields, shedding light on the interplay between attention and oculomotor behavior.