Designing and deploying scalable intelligent tutoring systems to enhance adult education

Intelligent tutoring systems have consistently been shown to be effective in enhancing student learning outcomes. However, despite their demonstrated benefits, these systems have not been adopted as broadly as expected. Over the past two decades, significant advances in tutor design and AI-powered authoring tools have enabled non-programmers to create tutors, thus reducing the time and cost aspects of building these technologies. However, there are still challenges that prevent the adoption of educational technologies. Further, research on tutoring systems' usage has largely focused on K-12 learner populations in settings where specific usage is prescribed, neglecting the needs of other user groups, such as adult learners. To promote widespread adoption, it is essential to understand the unique requirements of diverse learners, particularly in contexts where tutor usage is supplementary rather than mandatory. This thesis comprises three key chapters, each representing my contribution to the field of intelligent tutoring systems and educational technology. Chapter 1 examines the sociotechnical factors that influence the adoption and usage of intelligent tutoring systems in self-directed learning contexts, focusing specifically on adult learners. The study is divided into two parts. First, we present Apprentice Tutors, a novel intelligent tutoring system designed to address the unique needs of adult learners. The platform includes adaptive problem selection, real-time feedback, and visual dashboards to support learning in college algebra topics. Second, we investigate the specific needs and experiences of adult users through a deployment study and a series of focus groups. Using thematic analysis, we identify key challenges and opportunities to improve tutor design and adoption. Based on these findings, we offer actionable design recommendations to help developers create intelligent tutoring systems that better align with the motivations and learning preferences of adult learners. This work contributes to a wider understanding of how to improve educational technologies to support lifelong learning and professional development. Chapter 2 examines the adoption, usage patterns, and effectiveness of a novel tutoring system, Apprentice Tutors, among adult learners at a state technical college. We analyze three types of data including, user demographics, grades, and tutor interactions, to assess whether voluntary tutor usage translates into measurable learning gains. Our findings reveal key temporal patterns in tutor engagement and provide evidence of learning within tutors, as determined through skill improvement in knowledge components across tutors. We also found evidence that this learning transferred outside the tutor, as observed through higher course assessment scores following tutor usage. These results suggest that intelligent tutors are a viable tool for adult learners, warranting further research into their long-term impact on this population. Chapter 3 explores the strengths and limitations of large language models (LLMs) for math tutoring contexts. Researchers have made notable progress in applying Large Language Models (LLMs) to solve math problems, as demonstrated through efforts like GSM8k, ProofNet, AlphaGeometry, and MathOdyssey. This progress has sparked interest in their potential use for tutoring students in mathematics. However, the reliability of LLMs in tutoring contexts---where correctness and instructional quality are crucial---remains underexplored. Moreover, LLM problem-solving capabilities may not necessarily translate into effective tutoring support for students. In this work, we present two novel approaches to evaluate the correctness and quality of LLMs in math tutoring contexts. The first approach uses an intelligent tutoring system for college algebra as a testbed to assess LLM problem-solving capabilities. We generate benchmark problems using the tutor, prompt a diverse set of LLMs to solve them, and compare the solutions to those generated by the tutor. The second approach evaluates LLM as tutors rather than problem solvers. We employ human evaluators, who act as students seeking tutoring support from each LLM. We then assess the quality and correctness of the support provided by the LLMs via a qualitative coding process. We applied these methods to evaluate several ChatGPT models, including 3.5 Turbo, 4, 4o, o1-mini, and o1-preview. Our findings show that when used as problem solvers, LLMs generate correct final answers for 85.5% of the college algebra problems tested. When employed interactively as tutors, 90% of LLM dialogues show high-quality instructional support; however, many contain errors---only 56.6% are entirely correct. We conclude that, despite their potential, LLMs are not yet suitable as intelligent tutors for math without human oversight or additional mechanisms to ensure correctness and quality. This thesis contributes to the advancement of intelligent tutoring systems by addressing key gaps in user-centered design, large-scale deployment, and the evaluation of cutting-edge AI technologies, while focusing specifically on the adult learner population.