Aortic KD former: aortic multiclass segmentation using SegFormer via knowledge distillation

Abstract In this paper, we investigate the effectiveness of knowledge distillation for semantic segmentation of aortic structures using deep learning models. We employ SegFormer B5 as the teacher model and SegFormer B0 as the student model. Knowledge distillation is applied through three methods: output-based distillation, inner feature maps-based distillation, and integrated distillation, which combines both logits and feature maps. Our results show that while the student model (SegFormer B0) has significantly fewer parameters (3,715,686) compared to the teacher model (SegFormer B5) with 84,597,958 parameters, it achieves performance closely approaching that of the teacher. Among the evaluated techniques, inner feature map distillation yields the best performance, with a mean IoU of 0.9994, a mean Dice score of 0.9549, and a loss of 0.0020, demonstrating its superior capability in transferring knowledge effectively. The study highlights the potential of inner feature map distillation as the most effective method for achieving high performance with reduced model complexity.