Exploring the Impact of LLM-Based Scaffolding on Academic Performance and the Mediating Roles of AI Literacy and Prior Knowledge

Abstract

Large Language Models (LLMs) have rapidly gained popularity among university students for tasks such as essay writing, virtual tutoring, and coding exercises. Although the number of studies applying LLM-based solutions in educational contexts has grown significantly, researchers emphasize the need for empirical studies to evaluate the effectiveness of these tools to support teaching and learning. To address this gap, this paper presents a study analyzing the effects of an LLM-based system designed to support students’ self-reflection. The study compares the grades and course outcomes of two groups of students enrolled in the same Thermodynamics course during two academic years (2023 and 2024, 234 students in total), but only the 2024 cohort had access to an LLM-based chatbot specifically designed to support self-reflection. The data showed no significant correlation between students’ AI literacy profiles, their prior experience with generative AI, and the adoption of the tool. However, an analysis of engagement with the tool in the 2024 cohort revealed that students who interacted more extensively with the chatbot—particularly medium and high achievers (based on prior academic performance)—demonstrated a significant improvement in their final grades.