A bias-free measure of crossmodal audiovisual action adaptation

Actions can be perceived through multiple sensory modalities including vision and audition. It is currently uncertain if audiovisual actions are coded multimodally or in discrete unimodal areas. In order to test if actions are coded multimodally, we used a crossmodal adaptation paradigm to test whether the perception of hand action sounds was influenced by prior adaptation to visual and auditory actions. We generated an action sound continuum between the sound of a hand knocking and slapping by blending the sounds between the two action extremes. Following adaptation to auditory knocks, subsequent ambiguous test sounds appeared more like slaps; following adaptation to auditory slaps, subsequent ambiguous test sounds appeared more like knocks. Smaller but significant auditory aftereffects were generated by adaptation to purely visual information, whilst visual adaptation to visually presented orthographic action information did not generate auditory aftereffects. We also tested whether crossmodal adaptation improved action sound sensitivity around the adapting stimulus to rule out the possibility of observer response bias (Morgan 2013). Just-noticeable differences (JNDs) were calculated for knock sounds after participants adapted to auditory, visual, and audiovisual knocks, or did not adapt. Participants completed a two-interval forced-choice (2-IFC) adaptive staircase procedure, in which they judged which sound was more like a knock. JNDs were derived from a psychometric function fitted to the data. Auditory, audiovisual, and importantly visual only adaptation improved sensitivity to knock sounds when compared to no adaptation. A further experiment confirmed that auditory sensitivity to knock sounds was most enhanced following adaptation to visual knock actions compared to visual slap actions. These results suggest that adaptation to actions, irrespective of the adaptor modality can selectively enhance mechanisms underlying the coding of actions to improve perceptual sensitivity.  Such crossmodal action aftereffects may result from adaptation of multimodal neurons tuned to specific actions.