Scalable Nuclei Detection in HER2-SISH Whole Slide Images via Fine-Tuned Stardist with Expert-Annotated Regions of Interest

Background: Breast cancer remains a critical health concern worldwide, with histopathological analysis of tissue biopsies serving as the clinical gold standard for diagnosis. Manual evaluation of histopathology images is time-intensive and requires specialized expertise, often resulting in variability in diagnostic outcomes. In silver in situ hybridization (SISH) images, accurate nuclei detection is essential for precise histo-scoring of HER2 gene expression, directly impacting treatment decisions. Methods: This study presents a scalable and automated deep learning framework for nuclei detection in HER2-SISH whole slide images (WSIs), utilizing a novel dataset of 100 expert-marked regions extracted from 20 WSIs collected at the University of Malaya Medical Center (UMMC). The proposed two-stage approach combines a pretrained Stardist model with image processing-based annotations, followed by fine tuning on our domain-specific dataset to improve generalization. Results: The fine-tuned model achieved substantial improvements over both the pretrained Stardist model and a conventional watershed segmentation baseline. Quantitatively, the proposed method attained an average F1-score of 98.1% for visual assessments and 97.4% for expert-marked nuclei, outperforming baseline methods across all metrics. Additionally, training and validation performance curves demonstrate stable model convergence over 100 epochs. Conclusions: These results highlight the robustness of our approach in handling the complex morphological characteristics of SISH-stained nuclei. Our framework supports pathologists by offering reliable, automated nuclei detection in HER2 scoring workflows, contributing to diagnostic consistency and efficiency in clinical pathology.