A warm-up for adaptive online learning environments – the Elo rating approach for assessing the cold start problem

Abstract

The aim of this study is to present and evaluate the Elo rating algorithm as a tool for assessing the task difficulty in terms of the so-called “cold-start” problem – during the initial phase of the introduction of the adaptive system to the public. This analysis has been performed on the real data originating from the online programming course available on the RunCode platform: the online learning environment with multiple attempts allowed and feedback provided after every attempt. There have been 50055 submissions on 76 tasks uploaded by 299 RunCode users. It has been found that the Elo rating algorithm achieves the correlation of 0.702 with the reference values already for the sample size of n = 5, and the correlation of 0.905 for the sample size of n = 50. The Elo algorithm outperforms the Proportion Correct method for small sample sizes and may be a more reasonable choice as a simple method for task difficulty estimation during the initial phase of introducing the adaptive system to the public.