‘I know that I clicked but not if I read’: An Exploratory Study Comparing Data Traces and Self-Reports on Feedback Engagement

Abstract

The use of educational digital tools – both within and outside the classroom – has opened up for novel ways to investigate and pursue research on learning. For instance, the built- in intelligence that many applications have may also log the learner's actual activities, transforming them into models and statistics. Research reveals that such data-traces sometimes point in other directions than traditional self-reports, where students are asked about their learning activities or their use of a certain application or system. We wished to pursue this line of inquiry and explore if there are some types of estimations that correlate with objective measures, and hence if some types of estimations are better carried out through objective measures than through self-reports. Our study compared data-logged activities (clicks and eye- tracking measures) with self-reports on game-behavior for two types of estimations (requests for more feedback and reading feedback). To our knowledge, this kind of inquiry, comparing different types of data on feedback management, is rare. The results reveal that even if the students were quite good at estimating and reporting their requests for more feedback (in terms of clicks), they were substantially poorer at evaluating the extent of feedback messages they had read. These findings suggest that estimations of tasks that require more complex inner cognitive processes (e.g. reading) are more difficult for students themselves to report appropriately – whereas more procedural processes or concrete behaviors (i.e. clicking) are easier. These results are further discussed together with possible limitations in the measures used.