An Exploration of the Impact of Using ChatGPT on Students' On-Task Performance and Task Success

Abstract

Learning programming is inherently challenging and often difficult for beginners. The acquisition of this skill requires the development of logical thinking using effective problem-solving performance. These cognitive processes can be assessed through on-task performance metrics, which provide insight into students' performance and progress. This experimental study investigates whether ChatGPT helps students improve programming performance by exploring self-reported on-task metrics and task success. The participants were first-year college IT and CS students currently taking introductory programming courses. They were divided into a control group (n = 31) and an experimental group (n = 31), and their on-task programming performance metrics were compared using a statistical test. The study revealed that by comparing the group's on-task performance metrics, the use of the help-seeking strategy was shown to be significantly different among the groups. However, when comparing the on-task metrics of students with their task success, there is no significant statistical association found between the on-task performance metrics. Despite this, a statistically significant difference was revealed between the self-reported task success of students during hands-on programming tasks, suggesting that ChatGPT provided useful support in problem-solving. These findings highlight the evolving role of generative AI tools in computing education and suggest potential shifts in pedagogical approaches that promote essential problem-solving behaviors among students.