Study on a Bimodal Emotion Recognition Algorithm Based on Deep Fusion of Speech and Images

Emotion recognition aims to identify affective categories by analyzing physiological signals and behavioral characteristics and is one of the key research directions in Artificial Intelligence (AI). Addressing the issues of limited unimodal representation capability and insufficiently established cross-modal deep association in existing emotion recognition algorithms, which lead to inadequate accuracy and robustness in complex scenarios, this paper proposes a bimodal emotion recognition algorithm based on deep fusion of speech and image to enhance the accuracy and robustness of emotion recognition by establishing an effective cross-modal interaction and adaptive fusion mechanism. For image modality, Bilinear Interpolation (BI) is used for image scale normalization, followed by an improved Convolutional Neural Network (CNN) to extract deep spatial features. An improved Sparse Autoencoder (SAE) is then employed to compress features, reduce redundancy, and enhance fine-grained details. Finally, an improved Multilayer Perceptron (MLP) performs emotion classification. For speech modality, prosodic features, Mel-Frequency Cepstral Coefficients (MFCCs), geometric features, and attribute features are fused to form a multidimensional acoustic representation, which is subsequently classified using an improved MLP. After unimodal recognition, the outputs of the two modalities are fused at the decision level using a dynamic adaptive weighting strategy to generate the final emotion category and its corresponding probability. The experimental results indicate that, under a unified test set, the accuracy of sentiment recognition in this paper is improved by 14% and 18% respectively compared to single-modality models. Compared to other fusion strategy models, the recognition accuracy is concentrated in the range of 65% to 70%. The deep fusion method proposed in this paper achieves an overall recognition accuracy of 81.18%, which is superior to other methods. This result verifies the effectiveness of the algorithm proposed in this paper in improving the accuracy of sentiment recognition and system robustness.