Investigating effectiveness of distraction rate: augmented reality-based eye-tracking feature to predict student formative and summative performance

Augmented reality (AR) is gaining attraction as a valuable aid in training and educational settings. However, the cognitive overload due to the new learning environment may hamper effective learning during the AR sessions. For this reason, monitoring students learning status with an effective metric is required. Distraction rate (DR) is a feature extracted from a student’s eye-tracking coordinates data developed to measure the distracted proportion of a student in an AR learning session (Deay, 2023). In this study, we investigate DR with students’ formative and summative assessment outcomes to validate its effectiveness as a predictor for student performance.Methods: To do this, students learned a topic of biomechanics through several AR modules. The results of quizzes taken after each AR module and their class exam outcomes taken at the end of semesters provide formative and summative evaluation performances, respectively. The data were collected in two years in the same setup. To compute DR, the standard eye-tracking coordinates, called baseline, and those of an observed student are compared. In order to reduce false alarms, two sources of noise are accounted for. First, temporal noise caused by quick deviations from the baseline that only lasts for a short period of time is removed by computing the moving average of eye-tracking curves. Second, spatial noise caused by slight deviations in a student’s sight from the virtual instructor is reduced by applying a threshold to determine whether the deviation is large enough. Finally, the proportion of moving average signals exceeding the threshold is computed.Using mixed effects logistic regression models, this study shows how DR and students' performance are related while considering the year and student variations. To extract DR from eye-tracking data, two parameters should be determined, the window size and threshold. In this study, we carried out a comparison study with several parameter values with respect to the model’s prediction performance to find the best parameter tuning.Key Findings:For the formative performance, the results indicate that DR is a significant predictor for the probability of correct answers. For the summative performance, DR does not show a significant relationship with students’ exam scores, yet the negative regression coefficient of DR can be still found, indicating that the high DR value results in low performance in the exam. It can be interpreted that, due to the time interval between AR learning and exams, even if some students may have not paid much attention during the AR learning sessions, they could catch up on the material later by themselves. Overall, it is found that the exam performance is less sensitive, compared to the quiz performance, to students’ attention paid to AR learning sessions. Accordingly, the relationship between DR and summative performance is likely to be weaker than the case of formative assessment.