Studies on visual emotion understanding

As information explodes nowadays, visual data has become a crucial information carrier in various fields: social networks, e-commerce, online entertainment, etc. Visual emotion analysis has become an important research topic and has the potential to be applied to different areas, such as building users' profiles of visual content or finding consumers' interests in the merchandise. This thesis studies visual emotion understanding, fills in the affective gap between the visual features and semantics, and investigates major challenges in visual emotion understanding such as complexity, subjectiveness, generalizability, explainability, and bias. The reasoning behind the justification of visual emotion has been an important but largely unexplored problem. We propose to utilize language, and more specifically, an explanation of the judgment, to increase the explainability of visual emotion recognition. We collect a dataset with images, emotion tags, and explanations and analyze the dataset to gain insights into the ambiguity of human emotion perception. We examine baseline methods to predict emotions from images, explanations, and general image descriptions. We also study the multimodal models of both image and text. Our experiments shed light on effects from different modalities, and we also identify opportunities for future visual emotion categorization research based on the analysis. Visual emotion recognition is a challenging problem due to the ambiguity of emotion perception, diverse concepts related to visual emotion, and the lack of large-scale annotated datasets. We introduce a multimodal pre-training method to learn the visual emotion representation by aligning emotion, object, and attribute with a contrastive loss. Our method achieves state-of-the-art performance for visual emotion classification. We identify bias as an important problem in visual emotion research and establish a new dataset with gender and skin color annotations. Through quantitative study, we identify bias from dataset leakage and model leakage and evaluate adversarial bias mitigation methods.