Enhancing Language Learning Through Multimodal AI-Driven Feedback on Picture Descriptions: An Eye-Tracking Study

Abstract

To enhance English language learning through descriptive tasks based on everyday life scenes, we propose leveraging multimodal Al technologies. This involves utilizing multimodal large language models to automatically assess the quality of student responses and provide personalized, timely feedback in the form of Al-driven comments and suggestions for improvement. Furthermore, we will investigate students' perceptions of the feedback provided and its effectiveness in enhancing their writing skills. To achieve this, 30 participants were recruited to describe a set of daily-life pictures with the Al-driven automated assistance. During the experiment, eye-tracking technology was employed to capture students' eye movement data, enabling analysis of their visual perception and attention allocation. A survey questionnaire was administered to gather students' perceptions of this language learning approach and the effectiveness of the feedback. The experimental results indicate that students exhibit a positive attitude towards this language learning approach, as evidenced by their high levels of learning interest and motivation. Moreover, students demonstrate a willingness to actively engage with automatic feedback, with a particular inclination towards investing more attention and time in the suggestions generated by a large language model.