A SAM2-Driven RGB-T Annotation Pipeline with Thermal-Guided Refinement for Semantic Segmentation in Search-and-Rescue Scenes

High-quality RGB–thermal infrared (RGB-T) semantic segmentation datasets are crucial for search-and-rescue (SAR) applications, yet their development is hindered by the scarcity of annotated ground-truth and by the challenges of thermal-camera calibration, which typically depends on heated targets with limited geometric definition. Recent approaches focus on using semantic segmentation annotation tools and transferring RGB masks to multi-spectral data, but they do not fully address the need for robust cross-modal geometric validation, quality control, or human-in-the-loop reliability assessment in RGB-T segmentation. To fill this gap, we propose a validated cross-modal annotation pipeline that combines deep correspondence matching, geometric transformation (affine or homography) of RGB-T pairs, and quantitative alignment validation. Our RGB-T pipeline integrates a semi-automatic annotation pipeline based on the Segment Anything Model 2 (SAM2) in Label Studio, with guided human refinement, and incorporates quantitative cost and quality control via inter-annotator agreement before being used in downstream model training. Results across three annotators show that the proposed approach reduces annotation time by 36% while achieving high annotation quality (mean IoU = 74.9%) and strong inter-annotator agreement (mean pixel accuracy = 74.3%, Cohen’s κ = 65%). The proposed RGB-T pipeline was annotated on a SAR-oriented RGB-T dataset comprising 306 image pairs and trained on two SOTA RGB-T. These findings demonstrate the practical value of the proposed methodology and establish a reproducible framework for generating reliable RGB-T semantic segmentation datasets, complementing and extending recent multispectral auto-labeling approaches.