Dimensionality Reduction and Denoising of Spatial Transcriptomics Data Using Dual-Channel Masked Graph Autoencoder

Abstract Recent advances in spatial transcriptomics (ST) technology allow researchers to comprehensively measure gene expression patterns at the level of individual cells or even subcellular compartments while preserving the spatial context of their tissue. Spatial domain identification is a critical task in analyzing the ST data. However, effectively capturing distinctive gene expression features and relationships between genes poses a significant challenge. We develop a graph self-supervised learning method STMask for the analysis and exploration of the ST data. STMask combines the masking mechanism with a graph autoencoder, compelling the gene representation learning channel to acquire more expressive representations. Simultaneously, it combines the masking mechanism with graph self-supervised contrastive learning methods, pulling together the embedding distances between spatially adjacent points and pushing apart the representations of different clusters, allowing the gene relationship learning channel to learn more comprehensive relationships. The applications of STMask to four ST datasets demonstrate that STMask outperforms state-of-the-art methods in various tasks, including spatial clustering and trajectory inference. Source code is available at https://github.com/donghaifang/STMask . Author summary Spatial Transcriptomics (ST) is an emerging transcriptomic sequencing technology aimed at revealing the spatial distribution of gene expression and cell types within tissues. This method enables the acquisition of gene expression profiles at the level of individual cells or spots within the tissue, uncovering the spatial expression patterns of genes. However, accurately identifying spatial domains in ST data remains challenging. In our study, we introduce STMask, a self-supervised learning method that combines a dual-channel masked graph autoencoder with masking and contrastive learning. Our work contributes primarily in two aspects: (1) We propose a novel graph self-supervised learning method (STMask) specifically tailored for the analysis and research of ST data, which enhances the ability to capture the unique features of gene expression and spatial relationships within tissues. (2) Through comprehensive experiments, STMask provides valuable insights into biological processes, particularly in the context of breast cancer. It identifies enrichment of various differentially expressed genes in tumor regions, such as IGHG1 , which can serve as effective targets for cancer therapy.