Automatic Text Analysis of Reflective Essays to Quantify the Impact of the Modification of a Mechanical Engineering Course

Students’ reflective essays in engineering education provide insight and context for instructional modification and assessment. However, the assessment of reflective essays numbering in thousands can be time-consuming. This is notably important when trying to find specific changes in focus from one essay to another and measuring how strong those changes are across multiple corpora of essays.In this paper we describe and demonstrate an automated text analysis method for the at-scale, corpus-normalized analysis of reflective essays. We apply it to quantitatively measure whether the modification of an undergraduate mechanical engineering course had the conjectured impact of a stronger emphasis on teamwork.Our analytical method is a “pipeline” composed of Text Mining (TM), Natural LanguageProcessing (NLP), and Recurrence Quantification Analysis (RQA). We use this method to measure the presence of a specific thematic element in reflective essays to confirm the impact of the modification of a team-driven, model-based engineering design course. The original course and its modification were visualized using Sandoval’s conjecture mapping framework.The novel innovation of this approach is that the input (text from hundreds of reflective essays, sourced one at a time) when passed through this pipeline quickly produces a quantitative indication of the presence of thematic elements and their recurrence normalized across a corpus of hundreds of essays. A comparison of this quantitative indicator across separate corpora (each corpus of essays is for a different year) of reflective essays signaled a change in student focus toward the conjectured outcome.We conclude that the TM-NLP-RQA pipeline can be applied for quick and at-scale extraction ofthe relative magnitude of thematic statements from reflective essays. We observe that ourconjectured redesign had the impact that we desired