Contrastive Distillation Learning with Sparse Spatial Aggregation

Abstract Contrastive learning has advanced significantly and demonstrates excellent transfer learning capabilities. Knowledge distillation is one of the most effective methods of model compression for computer vision. When combined with contrastive learning, it can achieve even better results. Current knowledge distillation techniques based on contrastive learning struggle to efficiently utilize the information from both student and teacher models, often missing out on optimizing the contrastive framework. This results in a less effective knowledge transfer process, limiting the potential improvements in model performance and representation quality. To address this limitation, we propose a new contrastive distillation learning method by redesigning the contrastive learning framework and incorporating sparse spatial aggregation. This method introduces a novel integration of feature alignment and spatial aggregation mechanism to enhance the learning process. It ensures that the representations obtained by the model fully capture the semantics of the original input. Compared to traditional unsupervised learning methods, our approach demonstrates superior performance in both pre-training and transfer learning. It achieves 71.6 Acc@1, 57.6 AP, 75.8 mIoU, 39.8/34.8 AP on ImageNet linear classification, Pascal VOC object detection, Cityscapes semantic segmentation, MS-COCO object detection and instance segmentation. Moreover, our method exhibits stable training and does not require large pre-training batch-sizes or numerous epochs.