Real-time Estimation of Learners’ Mental States from Learners’ Physiological Information Using Deep Learning

Abstract

It is important to know the mental states of learners during the learning process to improve the effectiveness of teaching and learning. In this study, we first extracted the relationships between learners’ mental states and teachers’ speech acts, as well as learners’ physiological information, by constructing a deep learning system. The physiological indexes were near infrared spectroscopy (NIRS), electroencephalography (EEG), respiration intensity, skin conductance, and pulse volume. Learners’ mental states were divided into nine categories in accordance with the Achievement Emotions Questionnaire. In our experiment, the system achieved a high accuracy in predicting the learner’s mental states from the teacher’s speech acts and the learner’s physiological information. A mock-up experiment was then conducted, which revealed that the system’s interface was able to support teaching and learning in real time.