Reliability for multilevel data: A correlation approach

Studying the reliability of a measurement instrument is essential. Despite therecognition of the importance of reliability in psychology and medicine and the variousreliability coefficients that have been proposed, research on reliability for nested or multilevel data, ubiquitously in observational studies, remains limited. Two recent papers (Schönbrodt et al., 2022; ten Hove et al., 2022) address how to quantify reliability in multilevel settings based on generalizability theory. Specifically, ten Hove et al. (2022) defined between-cluster and within-cluster interrater intraclass correlation coefficients for multilevel designs where persons or raters are nested within clusters. Schönbrodt et al. (2022) also defined reliability coefficients at between-cluster and within-cluster (i.e., between-person) levels for designs where persons nested in couples are assessed numerous times daily over a number of days. Nevertheless, when applied to a common design, both approaches give inconsistent results regarding their definition of cluster-level reliability. In this paper, we propose an alternative approach to defining reliability coefficients for multilevel data that are based on calculating the expected correlation between repeated measurements. We will compare our approach with that of Schönbrodt et al. (2022) and ten Hove et al. (2022) and explain the differences between the three approaches in a number of common nested data structures: (1) raters crossed with both persons and clusters, but persons are nested within clusters, (2) raters nested within both persons and clusters, and (3) persons nested in clusters and crossed with raters and days.