A Tale of Two Models: Leveraging Self-Explanation to Contextualize Divergent Models of Student Carelessness

There has been greater interest in measuring carelessness in digital learning platforms, based on growing evidence that carelessness is associated with learning and learner outcomes. Multiple operationalizations of carelessness have been developed, leveraging well-established knowledge tracing models and the same theoretical framing of carelessness. However, those models do not always agree with each other as to the implications of carelessness -- in one recent study, two models applied to the same data found strong inverse relations to learning. This raises the question of which of these models (if either) is truly capturing carelessness, or if they are capturing different aspects of a construct that is more complex and multi-dimensional than we had realized. In this paper, we evaluate the convergent validity of two such carelessness models -- the contextual slip model based on Bayesian Knowledge Tracing (BKT) and the beyond-knowledge feature carelessness (BKFC) model -- using data from student self-explanations where they explain the causes of their own errors. While we had hypothesized that one model or the other would correlate well to the student self-explanations, the results show that neither model’s estimates align well with all of the annotations. Examining where the two models correlate with human annotations, we find that the two models were each associated with distinct types of self-explanation responses. Specifically, the BKT-based model’s inferences of carelessness were associated with students’ self-recoverable slips, where they erred in arguably minor ways that they felt they could easily correct themselves. On the contrary, the BKFC model’s inferences were associated with students demonstrating hasty, rushed behaviors in general. These results highlight the possibility of different models capturing different aspects of carelessness, and underline a broader challenge in operationalizing complex constructs of this nature.