Leveraging Diversity for Privileged Multi-Teacher Knowledge Distillation for Facial Expression Recognition

Learning privileged information allows a model to exploit data from additional modalities only available during training. State-of-the-art methods for privileged knowledge distillation (PKD) have been proposed to distill information from a teacher model (that combines different prevalent and privileged modalities) to a student model (without access to privileged modalities). Recently, methods have been proposed to capture and distill the structural information and outperform point-to-point PKD methods. However such PKD methods are primarily confined to learning from a single joint teacher representation, which limits their robustness, accuracy, and ability to learn from diverse sources. Diversity in the feature space leads to higher performance in such a distillation scheme. This paper proposes multi-teacher privileged knowledge distillation, a novel method for diversifying the teacher space by recycling the existing backbone feature representations and aligning them with the multimodal space using lightweight adaptation. A simple yet effective teacher selection mechanism implicitly mitigates the negative transfer and allows distillation from the most accurate teacher at each distillation step. The MT-PKDOT employs a structural similarity KD mechanism based on entropy regularized optimal transport for distillation. An additional constraint is added to the loss function to explicitly align the centroids in the student space. The proposed MT-PKDOT method was validated on Affwild2 and Biovid databases. Results indicate that our proposed method is able to improve over the existing state-of-the-art privileged knowledge distillation methods, and in cases where the alignment is ineffective, MTPKDOT is able to maintain the performance similar to single-teacher methods. It improves the visual-only baseline on the Biovid dataset by 5.8%. On the Affwild2 dataset, the proposed method improves 4% and 5% over the visual-only lower bound for valence and arousal, respectively.The code is made publicly available at: https://github.com/haseebaslam95/MT-PKDOT.