Disq, a library for manipulating bioinformatics sequencing formats in Apache Spark

ADAM and GATK have independently developed parallel and distributed genomic applications on Apache Spark. To access flat file formats such as BAM, CRAM, SAM, and VCF, both depend on the htsjdk library, which provides low­level codecs, and the Hadoop-­BAM library, which extends these for parallel and distributed access. Hadoop-­BAM was found to have correctness (invalid BAM file splits, leading to corrupt read data) and performance (sequential implementation of some parallelizable tasks) issues. The Spark­-BAM project demonstrated these issues could be addressed, and developed a comprehensive benchmark. Thus members of the ADAM, Hadoop-­BAM, htsjdk, GATK, Spark­-BAM, and ViraPipe projects identified an opportunity to collaborate on a replacement library. Discussion between collaborators began virtually, then in­person at OpenBio Winter Codefest 2018 in Boston, and continued at GCCBOSC Collaboration Fest 2018 in Portland. A new project Disq was started in 2018, and has since made at least three releases (most recently version 0.3.0, released 19 March 2019). Benchmarks show that Disq is faster and more accurate than Hadoop­-BAM, and at least as fast as Spark­BAM. Disq also adds significant new features, such as support for writing sharded files for efficiency, for taking advantage of index files while reading (e.g. .sbi index files to find splits between BAM records, .crai index files to find record boundaries in CRAM files), and for writing index files where appropriate. In addition to unit tests, Disq includes integration tests that run against real­world files (multi­GB in size). SAMtools and BCFtools are used to verify files written with Disq can be read successfully. Disq has been incorporated into ADAM and GATK, and will provide a convenient venue for further collaboration between those project teams. We also welcome new collaborators seeking correct and performant access to flat file formats on Apache Spark.