A Thematic Summarization Dashboard for Navigating Student Reflections at Scale

Abstract

Instructors often ask students to reflect on projects or tasks because it has been shown to be effective for learning. Instructors also use these reflections to improve future offerings of a course. Sifting through reflections manually, however, is both time-consuming and inefficient, especially for large courses. This paper describes a method for organizing student reflections by named entities (i.e., topics of interest) and instructor-defined “themes” to produce summaries that better meet the needs of instructors. Named entities are first extracted from the reflection corpus. Upon choosing one named entity to explore, sentences mentioning that entity are collated from across student reflections. The selected sentences are then classified into instructor-defined themes. Instructors can choose to re-define themes as necessary with support from the system in the form of prevalence statistics and theme-definition suggestions. Finally, a summary of student reflections for each theme is provided. This process and the resulting summaries were evaluated in a semi-structured Wizard of Oz interview study with the teaching assistants of a 160-student graduate-level course on Cloud Computing offered online to the students at Carnegie Mellon University. Results from quantitative Likert-scale analyses and qualitative coding show that teaching assistants preferred our topic and theme-focused summaries over general summaries generated from a random subset of student reflections. Deployment in the form of an instructor-facing dashboard and improvement to the system to allow for uncommonly expressed content to be better discoverable through the dashboard are planned for future work.