Efficiency or Understanding? Novice Programmers' Problem-Solving With and Without ChatGPT

Abstract

Large Language Models (LLMs) such as ChatGPT are increasingly present in programming education, but their educational value depends on how they are integrated into student learning. This paper reports on a baseline experiment involving thirty sophomore computing students at a rural state university in the Philippines, classified as novices based on pre-test performance and randomly assigned to AI-assisted and non-AI groups. The AI-assisted group accessed ChatGPT through the free-tier web interface, which in July 2024 defaulted to the GPT-4o model. Students were given two hours to solve three programming problems, followed by a post-test and a focus group discussion. Screen recordings, rubric-based scoring, and observer notes were analyzed to examine coding behaviors and learning outcomes. As expected, the AI-assisted group achieved higher task completion and produced more structurally advanced code, though this often relied on repeated prompting rather than independent debugging. By contrast, the non-AI group generated less polished solutions but engaged more deeply in problem decomposition and iterative debugging, resulting in significantly greater post-test gains. Rather than presenting this efficiency-understanding trade-off as a new discovery, the study contributes contextualized evidence from an underexplored novice setting and provides empirical grounding for ongoing work that designs scaffolded, pedagogically guided integration of LLMs. The findings highlight that unstructured AI use fosters dependency, whereas structured use has the potential to balance efficiency with meaningful conceptual learning.