Vid2Log: A Machine Learning Approach to Structured Screen Activity Logging

Abstract

Collecting and analyzing learner actions on computer systems during ill-structured tasks presents significant methodological challenges, as these activities often span multiple tools and environments beyond the scope of specialized logging systems. Consequently, researchers frequently rely on screen recordings to capture comprehensive user behavior, necessitating subsequent manual conversion into structured activity logs. This manual annotation process is inherently labor-intensive, error-prone, and severely limits the scalability of behavioral analysis studies. To address these limitations, we present Vid2Log, an open-source tool that automates the conversion of raw screen-recording videos into structured action logs through machine learning techniques. Our approach leverages transfer learning, adapting a pre-trained model to the specific domain of computer screen-recording analysis. This methodology proves highly efficient by significantly reducing both the need for extensive training data and computational resources compared to developing models from scratch. Vid2Log specifically targets the generation of macro-level activity logs from screen-recording videos, thereby reducing annotation overhead while maintaining the granularity necessary for meaningful behavioral analysis. This capability enables large-scale studies of complex workflows across diverse domains, including programming, education, design, and ill-structured problem solving in computer-based systems.