PC-CLIP: Probabilistic Calibration CLIP for Zero-Shot Semantic Segmentation

Zero-shot semantic segmentation aims to extend the ability of aligning pixels with pre-defined classes to novel unseen classes. Most previous CLIP-based methods use static prompts as inputs, which are unable to search for the most valuable information in images adaptively. Meanwhile, since CLIP framework is trained on image-text pairs, transferring image-level knowledge into pixel-level understanding tasks, such as semantic segmentation, becomes another challenge. In addition, previous methods are easy to overfit to seen categories. In this paper, we propose a Probabilistic Calibration CLIP network (PC-CLIP) for zero-shot semantic segmentation. Concretely, we first present the Context-Aware Prompt Generation (CAPG) method to produce dynamic contextual prompts as inputs. Then, we develop a novel Local Preserved Patch Enhancement (LPPE) mechanism to enhance the local information in the CLIP framework. Besides, an Adaptive Probability Calibration (APC) approach is designed to mitigate overfitting by dynamically adjusting the class probability distribution. Extensive experiments are conducted on three widely used datasets to demonstrate the effectiveness of the proposed method. We achieve competitive performance against 16 state-of-the-art approaches under four evaluation metrics. Codes will be available on https://github.com/XiaoJinNK/PC-CLIP.