Evaluating Behavioral Intervention Technologies: Integrating Human-Centered Design and Implementation Science Outcomes (Preprint)

BACKGROUND Background: Behavioral intervention technologies (BITs) have been the tools for advanced health solutions by increasing access to cost-effective care. However, BITs often show reduced impacts in community settings and are rarely sustained, limiting the capacity to fulfill their promising benefit. Human-Centered Design (HCD) is an approach that can address these implementation challenges by incorporating the stakeholders’ needs, making the content and functionality contextually appropriate. Implementation Science (IS) helps researchers understand barriers and facilitators, develop strategies that overcome obstacles, and enhance BIT uptake and sustainability of BITs in real-world settings. HCD and IS shared common goals to improve BITs’ usability and implementation by applying iterative and contextual design processes, engaging stakeholders as the design partner, involving a multidisciplinary design team, and conducting iterative testing and evaluation as early and frequently as possible. In this study, we proposed a heuristic model that can maximize the potential of BITs’ success. OBJECTIVE We aimed to 1) synthesize the HCD and IS outcomes, integrate them into an evaluation model for BITs’ design, development, and testing process; 2) apply the integrated evaluation model to a case study, Sleep Shared-Management Intervention for Children with Juvenile Idiopathic Arthritis and Their Parents. METHODS We conducted a narrative literature review on HCD and implementation outcomes on BITs from the PubMed, CINAHL, and Web of Science databases (2000-2023). Article inclusion criteria are as follows: 1) published through the search date of January 11, 2023, 2) focused on usability constructs or implementation outcomes, and 3) related to evidence-based practices (e.g., intervention, services, policy) in healthcare. RESULTS ¬Informed by evolved frameworks of HCD and IS outcomes, we proposed a heuristic model that can guide the development and evaluation of BITs: USIS model, in which Usability and IS are combined. We categorized the outcomes into five domains 1) User-Centeredness (empathy, engagement, and equity), 2) Efficiency (cost, timeliness, and rapidity), 3) Feasibility (learnability, memorability, error reduction, and low cognitive load), 4) Satisfaction (acceptability, appropriateness), and 5) Fidelity (adoption, penetration, sustainability). USIS conceptualized constructs to reflect key outcomes of BIT’s usability and implementation, address the missing piece of user-focused outcomes like equity, accessibility, empathy, and engagement, and implementation outcomes such as timeliness and rapidity. Guided by the USIS model, we found the strengths and areas for improvement for the SLEEPSMART project. CONCLUSIONS USIS model offers specific tools to design better BITs and evaluate existing BITs to improve health innovations’ translation into practice. With the USIS model, HCD and IS practitioners can use a common language to enhance implementation and health outcomes for broader communities, support researchers in identifying multilevel barriers to implementation, and guide evaluating the BITs. We believe the USIS model offers a pathway for improving innovation advancement by incorporating the processes and tools from HCD and IS.