Program Evaluation Standards for Utility Facilitate Stakeholder Internalization of Evaluative Thinking in the West Virginia Clinical Translational Science Institute

Background: The Program Evaluation Standards (PES) can be considered established criteria for high quality evaluations. We emphasize PES Utility Standards and evaluation capacity building as we strive for meaningful application of our work in the real world. Purpose: We focused our methodology on understanding how stakeholders discussed utility and how their perceptions related to our evaluation work aligned with PES Utility Standards. Setting: The West Virginia Clinical Translational Science Institute (WVCTSI) is a statewide multi-institutional entity for which we conduct tracking and evaluation since 2012. Intervention: Sustained collaborative engagement of evaluation stakeholders with the goal of increasing their utilization of evaluation products and evaluative thinking. Research Design: Case study. Data Collection and Analysis: We interviewed five key stakeholders. Themes developed from analysis of PES Utility standard coding of interview data informed document analysis. Interview and document analysis were used to develop themes and illustrative examples, as well as to develop and describe a five-level Evaluation Uptake Scale. Findings: We describe shifts in initiation, use, and internalization of evaluative thinking by non-evaluation personnel that prompted development and application of an Evaluation Uptake Scale to capture increased evaluation capacity among stakeholders over time. We discuss how focus on PES Utility and evaluation capacity building facilitated such shifts and their implications for maximizing utility of evaluation activity in large complex programmatic evaluations. Keywords: Program evaluation standards, evaluation utility, evaluation capacity building.