RGB-D egocentric segmentation of human bodies for XR applications

Introduction Video-based self-avatars represent a promising approach for displaying users’ bodies in XR environments. While previous methods have relied on color cues, depth data, or RGB-based deep learning models, these approaches often su”er from false positives and limited user distance perception. Methods In this work, we introduce a video-based self-avatar system pipeline that leverages RGB-D information to overcome these limitations. We explore two state-of-the-art real time segmentation networks, AsymFormer and ThunderNet, and fine-tune them with the RGB-D Segment Egocentric Bodies dataset, a purpose-built collection of over 9,000 RGB–depth–label triplets. We describe our complete pipeline, from dataset creation and network training to its integration into a Unity- based XR engine. Results To evaluate our approach, we conducted both quantitative and usercentered studies. Quantitatively, the AsymFormer networked trained with our custom RGB-D dataset achieves up to a 14.3% improvement in segmentation accuracy compared with former egocentric real time body segmentation, while significantly reducing false positives. The qualitative evaluation involved two steps: a video-based assessment and a mixed-reality (MR) user study. Participants in the video-based evaluation perceived substantial quality improvements when using RGB-D segmentation, reporting a 39.6% gain over the RGB-only baseline. The MR evaluation further revealed that RGB-D–based self-avatars enhance users’ distance perception. Nevertheless, participants also reported noticeable latency introduced by the RGB-D processing pipeline, especially when compared with a simpler depth-only approach. Discussion Overall, our findings demonstrate the advantages of RGB-D information for video-based self-avatar generation while highlighting current challenges such as latency and hardware limitations. Dataset is available at https://huggingface.co/datasets/ExtendedRealityLab/RGB-D-SegmentEgocentricBodies .