Evaluation of metagenome binning: advances and challenges

Abstract Several recent deep learning methods for metagenome binning claim improvements in the recovery of high-quality metagenome-assembled genomes. These methods differ in their approaches to learn the contig embeddings and to cluster them. Rapid advances in binning require rigorous benchmarking to evaluate the effectiveness of new methods. We have benchmarked newly developed state-of-the-art deep learning binners on CAMI2 and real metagenomic datasets. The results show that SemiBin2 and COMEBin give the best binning performance, although not always the best embedding accuracy. Interestingly, post-binning reassembly consistently improves the quality of low-coverage bins. We find that binning coassembled contigs with multi-sample coverage is effective for low-coverage dataset, while binning sample-wise assembled contigs with multi-sample coverage (multi-sample) is effective for high-coverage samples. In multi-sample binning, splitting the embedding space by sample before clustering showed enhanced performance compared with the standard approach of splitting final clusters by sample. Deep-learning binners using contrastive models emerged as the top-performing tools overall, with MetaBAT2 and GenomeFace demonstrating superior speed. To facilitate future development, we provide workflows for standardized benchmarking of metagenome binners.