MOH: a novel multilayer multi-omics heterogeneous graph for single-cell clustering

Abstract Cell clustering is crucial in single-cell multi-omics research for identifying distinct cellular populations. Although there has been progress in integrating multi-omics data for clustering, combining more than two types of omics data remains challenging due to the diversity and heterogeneity of these datasets. Traditional approaches typically use heterogeneous graphs that integrate only two types of omics data, constructing graphs with genes and cells as nodes and a single type of edge representing their relationships. However, this method has limitations as it overlooks cell-cell interactions and struggles to capture complex cellular dynamics. Additionally, the graph structure must be redesigned whenever new omics data are introduced, limiting the scalability of these models. To address these issues, we introduce MOH, a novel single-cell clustering algorithm based on a multilayer multi-omics heterogeneous graph. MOH integrates three key single-cell omics types: scRNA-seq, scATAC-seq, and spatial transcriptomics. It constructs a multilayer heterogeneous graph to simultaneously extract and enhance representations from all three omics layers, incorporating both intra-layer and inter-layer edges to capture association and similarity relationships. This enriched representation leads to an accurate clustering results. Extensive experiments show that MOH outperforms six state-of-the-art methods on unsupervised clustering metrics, offering a precise and comprehensive analysis with consistent improvements across all evaluation criteria. Moreover, downstream analyses validate the results, revealing novel biological insights into immune disorder complications in cancer, cancer drug repurposing, and new signaling pathways, which merit further investigation and validation.