Persistent evidential discordance

Successful replication is a hallmark of scientific truth. Discordant evidence refers to the situation where findings from different studies of the same phenomenon do not agree. Although evidential discordance can spur scientific discovery, it also gives scientists a reason to rationally disagree and thereby compromises the formation of scientific consensus. Discordance indicates that facts about the phenomenon of interest remain unsettled and that a finding may not be reliably replicable. We single out persistent evidential discordance as a particularly difficult problem for the epistemology of science, and distinguish between different causes of evidential discordance – non-systematic error, noise, and bias. Unlike discordance brought about by non-systematic error or noise, persistent discordance often cannot be rationally resolved by temporarily suspending judgment and collecting more data within existing lines of inquiry. We suggest that the analysis of enriched lines of evidence (Boyd 2018) provides a useful approach to diagnosing and evaluating episodes of persistent evidential discordance. Attention to the line of evidence, which extends from raw data to an evidential claim supporting or disconfirming a hypothesis, can help researchers to locate the source of discordance between inconsistent findings. We argue that reference to metadata, information about how the data were generated and processed, can be a key step in the process of resolving normative questions of correctness, i.e., whether a line of evidence provides a legitimate answer to a particular research question. We illustrate our argument with two cases: the alleged discovery of gravitational waves in the late 1960s, and the social priming controversy in experimental psychology.