Evaluating the Assessment of Comment Quality in Learning Communication Skills using Active Video Watching

Abstract

Supporting student engagement remains one of the key challenges in video-based learning. This challenge is addressed by active video watching (AVW), a learning approach that supports engagement through different interventions, such as note-taking in the form of comments that learners submit while watching videos. One platform to support AVW is AVW-Space. Previous studies on AVW-Space detail improvements in the system, such as the integration of Artificial Intelligence and Machine Learning (ML) models in the comment feature of the system. This study investigates two machine learning models used to automatically assess the quality of comments when learning communication skills via AVW. One model is generated based on a large set of comments created by students when engaging with videos about presentation skills. For this study, a new model is developed from comments that students submitted when engaging with videos about communication skills. Results show that the new model, which was created from data on communication skills, performed better when assessing comments for communication skills compared to the model generated from comments for another skill. This has been demonstrated by the higher value of inter-rater agreement with the comment quality assessment made by human coders.