Understanding reproducibility of bioinformatics workflows

Reproducibility is an essential factor in establishing the reliability of results reported in the scientific literature and ultimately in the knowledge that is generated. This knowledge demands that experiments should be verifiable and potentially extended through open and transparent processes. This need for reproducibility is magnified in the field of genomics, where advances in massively parallel DNA sequencing technologies have resulted in an explosion of data and increased dependence on computational analysis of this data. Workflows are one common way through which such data is analyzed. In this context, there is an absolute need for reliable, reproducible, effective and timely interpretation of genomics data sets especially since they can be used for understanding of diseases and support personalized medicine. Faced with the complexity of ge- nomics datasets and the plethora of bioinformatics tools and workflow systems that have emerged over the last few years, a major challenge is supporting re- producibility of bioinformatics workflows. This is complicated by the current incomplete understanding regarding requirements for assuring reproducibility of workflows and assumptions relating to the many different approaches that can be taken when defining and enacting (executing) workflows. This problem is exacerbated by the lack of clarity in terminologies and concepts associated with reproducibility. The objective of this project is to empirically study the reproducibility requirements associated with bioinformatics workflows. This research highlights the need for, and critical dimensions of, reproducibility of bioinformatics workflows through a case study exploring the genomic analysis of a rare genetic disorder. In addition, it focuses on the reproducibility challenges that arose during the data analysis by the different (independent) teams. We subsequently categorize widely used workflow definition and implementation approaches for genomic data analysis and offer detailed implementation discussion of a complex but widely adopted genomic data analysis workflow in a selected exemplar from each of the workflow categories. Through empirical analysis, we identify assumptions that were implicit in the exemplar approaches, resulting in insufficient documentation of workflow requirements that impacted directly on the reproducibility of the results. Building on this, we identify key implicit and explicit requirements that impact directly on the reproducibility of bioinformatics workflows. We characterize these implicit and explicit requirements according to critical dimensions of reproducibility to establish a reproducibility framework that considers different categories of workflow approaches. The framework offers a coherent explanation and exploration of the concepts associated with workflow reproducibility. This research illustrates the benefits of the framework by demonstrating the portability of a fully declarative complex workflow to an independent workflow platform to support reproducible experiments. The evaluation illustrates that the highest degree of reproducibility is supported by workflow definition approaches that make the least assumptions on the software environment.