Developing a LLMs-Driven System Based on Human-AI Progressive Code Generation Framework to Assist Mathematics Learning

Abstract

This paper proposed a system interface based on a novel progressive code generation framework to produce verified programming codes using natural language for mathematics learning. With the latest advancement of large language models such as GPT-4, people can ask GPTs to generate programming code using natural language. However, the codes generated may not be directly executable or aligned with the user's desired goal or purpose. Furthermore, different people may express their goals and purpose in different ways. Hence, there are many possible ways to map natural languages and code. Many of the code evaluation frameworks such as pass-ratio@n, BLEU and CodeBLEU evaluate large language models (LLMs) code generation ability based on their actual execution results on the first prompt without giving feedback of the code execution error. With this in mind, a framework that allows code generated from any large language model to progressively improve was proposed. The idea is to re-prompt the LLMs with the error feedback to generate codes until the code is executable and achieve the user's desire goal or purpose with a predefined number of iterations. If a certain number of iterations has been reached and the code is still not executable or cannot achieve users' intended goals or desire, the prompt would be useful to include in the training dataset for fine-tuning for that large language model. In this study, the application of the proposed framework to enhance the accuracy of mathematical problem-solving problems was tested and reported. This framework may be useful to improve any LLM code generating ability continuously. Discussions were made.