vcfdist: Accurately benchmarking phased small variant calls in human genomes

Abstract Accurately benchmarking small variant calling accuracy is critical for the continued improvement of human whole genome sequencing. In this work, we show that current variant calling evaluations are biased towards certain variant representations and may misrepresent the relative performance of different variant calling pipelines. We propose solutions, first exploring the affine gap parameter design space for complex variant representation and suggesting a standard. Next, we present our tool “vcfdist” and demonstrate the importance of enforcing local phasing for evaluation accuracy. We then introduce the notion of partial credit for mostly-correct calls and present an algorithm for clustering dependent variants. Lastly, we motivate using alignment distance metrics to supplement precision-recall curves for understanding variant calling performance. We evaluate the performance of 64 phased “Truth Challenge V2” submissions and show that vcfdist improves measured (SNP, INDEL) performance consistency across variant representations from R 2 = (0.14542, 0.97243) for baseline vcfeval to (0.99999, 0.99996) for vcfdist.