DAP-Whisper: A Robust Audio-Visual Speech Recognition System via Distribution-Aware Prompting and Consistency-Gated Modulation

Large-scale pre-trained models like Whisper have established a new paradigm for Automatic Speech Recognition (ASR). However, their unimodal nature renders them vulnerable to non-stationary environmental interference and unpredictable distribution shifts in unconstrained scenarios. Existing multimodal adaptation approaches typically rely on static fusion mechanisms, which apply a uniform injection strategy regardless of the input's degradation profile. This rigidity creates a fundamental mismatch between fixed parameters and dynamic noise manifolds. To address this limitation, we propose DAP-Whisper, a parameter-efficient framework that reformulates multimodal adaptation as a continuous interpolation problem within a learned manifold simplex. Specifically, we design a Quality-Aware Distribution-Adaptive Prompting (Q-DAP) mechanism, which dynamically synthesizes instance-specific parameters by interpolating orthogonal basis vectors, ensuring precise alignment with real-time acoustic shifts. Furthermore, to mitigate the epistemic uncertainty arising from visual corruption, we introduce a Consistency-Gated Modulation (CGM) mechanism that acts as a reliability valve to prevent negative transfer. Extensive experiments on the LRS3 benchmark demonstrate that DAP-Whisper outperforms state-of-the-art static baselines.  Specifically, with the Whisper-Medium backbone, it drives the WER down to 3.36% under 0dB noise, while exhibiting superior resilience against joint audio-visual degradation.