Testing a mixed method learning evaluation approach for real-time assessments of the implementation process

Abstract Background: Stakeholders involved in implementation efforts progress through multiple cycles of learning and problem solving, to which traditional evaluation approaches for implementation of healthcare innovations often fail to pay attention. Learning evaluation principles provide an alternative evaluation approach to capture and feedback ongoing learning in an iterative and informative way. This paper outlines a novel evaluation approach aligned with learning evaluation principles which will be tested in an implementation study examining an intervention designed to improve mental and physical health in people with severe mental illness (UCLPartners-Primrose). Methods: This evaluation outlines five key steps: (1) Conduct baseline data collection to understand the implementation context; (2) Identify process and outcome measures of interest to each implementation site; (3) Collect and analyse qualitative and quantitative data on contextual factors in real-time via regular contact interviews; (4) Feedback synthesised data and learning to sites on an ongoing basis via lightning reports; (5) Generate learning from process, action, and outcome data, lightning reports, and post intervention interviews. This protocol draws from learning evaluation principles and the steps will be achieved using the Stanford lightning reports and the Consolidated Framework for Implementation Research. The application of this approach will be tested over 18 months through the implementation evaluation of an intervention, UCLPartners-Primrose. Discussion: This protocol describes a flexible, iterative, and contextualised learning evaluation approach, which is expected to capture within-system learning through rapid analysis and feedback loops, amenable to the needs of implementation sites. This approach will enable the triangulation of the implementation experiences of the end users with the knowledge of implementation researchers, to build a more holistic picture of implementation barriers and enablers to bridge the gap between real-time data collection and its transformation into actions. Conclusion: This is an innovative implementation evaluation approach developed to be more rapid, relevant, and responsive than traditional methods and encourages greater researcher-implementer collaboration. It therefore holds benefits for implementers and research teams involved in implementation efforts.