Exploring the Effect of Demographics Inclusion on Subject-independent Emotion Recognition

Electroencephalography (EEG) can capture electrical activity associated with human emotion processing from the scalp. The electrical activity can be processed using deep learning models to predict emotional states. Two approaches can be employed to develop these deep learning models: subject-dependent and subject-independent. The subject-independent approach is more practical as it trains the model on data from some individuals and tests it on entirely different individuals, ensuring it generalizes well to new users. However, because of the high variability of EEG across individuals, the subject-independent approach tends to yield low performance. Recent studies suggest incorporating demographic information along with EEG signals is one way to overcome this issue. By using the subject-independent approach, this study investigates different demographics factors such as age, biological sex and cultural factors impact emotion prediction. Moreover, this thesis delineates the development of a deep learning models dedicated to emotion recognition on five different datasets. To find the impact of age and biological sex a logistic regression model was used to correlate the output of a deep learning model with subjects’ age and sex, thereby evaluating whether these factors impact emotion prediction. Our analysis indicates that the ‘sex’ variable significantly influenced the predictions of the deep learning model in three out of five emotions, whereas ‘age’ does not have any effect. These findings suggest that sex is a factor that needs to be considered when designing EEG-based emotion recognition models. Furthermore, attention network layers were used to identify brain areas more involved in predicting emotions. Additionally, an odds ratio analysis was conducted using logistic regression to evaluate the impact of sex on emotion prediction. Our findings reveal that cortical activation patterns elicited by emotional audio-visual stimuli differ between females and males, with females showing more neural activation in the left hemisphere and males showing more in the right hemisphere. Moreover, when the output probabilities of the deep learning models are further postprocessing with the subject’s sex, the odds of correctly predicting emotions increase. These findings suggest that sex differences can lead to more robust subject-independent emotion recognition models. Additionally, this study also investigates how cultural factors impact emotion prediction. Specifically, we used a stacking model that combines deep learning with multinomial logistic regression to predict positive, neutral, and negative emotions among 15 Chinese, 8 French, and 8 German subjects. Our approach achieved accuracies of 77.3% for Chinese subjects, 73% for French subjects, and 65% for German subjects, which are comparable to or exceed accuracies reported by previous studies. Our approach highlighted that incorporating cultural information increases the likelihood of predicting positive emotions for Chinese participants and negative emotions for Europeans. Moreover, French and German subjects exhibited similar neural patterns across all emotions, suggesting a more common cultural sharing between those subjects. Overall, our findings emphasize the importance of integrating demographics information considerations into emotion recognition models. This inclusion not only improves emotion prediction accuracy for subject-independent approaches but also promotes inclusivity and ethical practices in emotion recognition systems. Which could lead to more robust subject- independent models with potential applications in areas such as healthcare, education, and marketing.