Retinal vessel segmentation driven by structure prior tokens

Abstract Background Accurate retinal vessel segmentation from Optical Coherence Tomography Angiography (OCTA) images is vital in ophthalmic medicine, particularly for the early diagnosis and monitoring of diseases, such as diabetic retinopathy and hypertensive retinopathy. The retinal vascular system exhibits complex characteristics, including branching, crossing, and continuity, which are crucial for precise segmentation and subsequent medical analysis. However, traditional pixel‐wise vessel segmentation methods focus on learning how to effectively divide each pixel into different categories, relying mainly on local features, such as intensity and texture, and often neglecting the intrinsic structural properties of vessels. This can cause suboptimal segmentation accuracy and robustness, particularly when handling low‐contrast, noisy, or pathological images. Purpose This study aims to integrate structural priors into a segmentation framework. Prior embeddings are used to guide the segmentation process, which encode the typical morphology and topological structure of blood vessels. Incorporating these embeddings can improve the accuracy of retinal vessel segmentation, particularly in challenging areas such as small vessels and regions with ambiguous boundaries. This approach could help to preserve the integrity and continuity of the vascular structure, resulting in more reliable and precise segmentation. Methods This study adopts a generative image segmentation framework. A structured representation in a latent embedding space is presented to explore retinal vessel priors. On this basis, a prior‐driven retinal vessel segmentation network is introduced. First, the retinal vessel priors from ground truth data are learned, which are encoded as embedding tokens through a residual quantization reconstruction network. The learned priors are stored in a codebook. In our network, a raw OCTA image is transformed into semantic features using an encoder. Each semantic feature is subsequently represented by a set of embedding tokens from the codebook. Finally, the retinal vessels are reconstructed, preserving the integrity and continuity of the vascular structures using the learned structural priors. Results The performance of our proposed network was assessed across three OCTA datasets: publically available ROSE‐1 and ROSE‐2, and the private dataset OCTA‐Z. Both quantitative and qualitative evaluations revealed that our network outperformed current state‐of‐the‐art methods. In particular, our approach achieved the average Dice scores of 77.63, 71.01, and 81.11% for ROSE‐1, ROSE‐2, and OCTA‐Z, respectively. Conclusions The experimental results demonstrated that the proposed network effectively learned and utilized implicit vessel priors for OCTA vessel segmentation. Extensive evaluations indicated that our network surpassed the current state‐of‐the‐art methods. Specifically, our reconstruction approach of applying latent prior tokens offered a promising solution for the pattern representation of retinal vessels. In the future, we will extend this method to support more comprehensive retinal structure segmentation and eye‐related disease classification tasks. Additionally, we plan to integrate diverse and detailed prior knowledge, such as anatomical and pathological information, to enhance the performance and versatility of our approach.