Composite Score for ChatGPT Prompt Efficiency: A Computational Linguistic Analysis of Engineered Chatbot Prompts

Abstract

The use of chatbots has become increasingly popular in the field of education. Hence, the quality of the prompts used by chatbots can greatly influence the quality of the generated responses which in tum contributes to desirable outcomes related to the overall learning experience. However, even as known criteria for efficient prompts are prevalent, there is a dearth of measurable and concrete linguistic factors that guide prompt engineering. This paper presents a composite score for ChatGPT prompts which can be made applicable to other foundational generative Al chatbots. Through a computational linguistic analysis of known efficient prompts used in learning, emergent linguistic factors point to the relationship of linguistic features and the confidence of ChatGPT responses to well-structured prompts that use the said linguistic features. The linguistic features are the average collostructural strength, collostructural ratio diversity, specificity, and academic language use. These features depict the quality of prompts that pertain to the grammatical structure, specificity, and relevance to the task at hand, and academic language use. Further, these features constitute a composite score for prompts introduced in this study that represent linguistic efficiency and subsequently, correlates to perplexity or certainty estimates of the generated responses of ChatGPT.