Fine-Tuning SAM2 for Generalizable Polyp Segmentation with a Channel Attention-Enhanced Decoder

Polyp segmentation is a critical task in medical image analysis, particularly in colonoscopy, where it plays a vital role in the early detection and treatment of colorectal cancer. In recent years, advancements in deep learning, especially the application of Convolutional Neural Networks (CNNs) and Transformer models, have significantly improved segmentation performance. Despite these advancements, the generalizability of these models across different datasets is often limited. Recently, Meta released the Segment Anything Model 2 (SAM2), which has demonstrated exceptional performance in both video and image segmentation tasks. This paper aims to develop a universal polyp segmentation model by fine-tuning the pre-trained encoder of SAM2. We introduce a learnable prompt layer within the Transformer blocks and employ a full-scale skip connection structure as a decoder to integrate multi-scale semantic features. Our model outperforms state-of-the-art methods on datasets such as Kvasir-Seg and CVC-ClinicDB. Additionally, our experiments show that the model exhibits excellent transfer learning capabilities on unseen datasets, making it a robust and generalizable model in the field of polyp segmentation.