Consolidated Benchmark for Efficacy and Effectiveness Frameworks in EdTech

Post-COVID19 evaluation reports of Educational Technologies (EdTech) pointed to the imperative to scientifically evaluate EdTech’s impact on learners. The absence of a shared definition for such evaluations, coupled with the availability of diverse frameworks and criteria in the field, poses a challenge. This paper is concerned with two critical dimensions of impact on learning outcomes — efficacy and effectiveness — with a focus on teaching and learning EdTech for the K12 age range. A systematic literature search identified 65 frameworks that target the efficacy or effectiveness of K12 EdTech products. The frameworks were analysed in relation to their rigour, using the science of learning principles embedded in the EdTech Evidence Evaluation Routine (Kucirkova, Brod & Gaab, 2023). The results were synthesised into a consolidated benchmark that categorises the available frameworks at three levels based on the rigour applied to their assessments. The consolidated Effectiveness/Efficacy benchmark serves as a valuable tool for evaluating any EdTech type with available frameworks, facilitating informed decision-making in the dynamic landscape of educational technology.