Utilizing Crowdsourcing and Topic Modeling to Generate Knowledge Components for Math and Writing Problems

Abstract

Combining assessment items with their hypothesized knowledge components (KCs) is critical in acquiring fine-grained data on student performance as they work in an ed tech system. However, creating this association is an arduous process and requires substantial instructor effort. In this study, we present the results of crowdsourcing KCs for problems in the domain of mathematics and English writing, as a first step in leveraging the crowd to expedite this task. We presented crowdworkers with a problem in each domain and asked them to provide three underlying skills required to solve it. These inputs were then analyzed through two topic modeling techniques, to compare how they might cluster around potential KCs. Results of the models’ output were evaluated against KCs generated by domain experts to determine their usability. Ultimately, we found that half of the crowdsourced KCs matched expert-generated KCs in each problem. This work demonstrates a method to leverage the crowd’s collective knowledge and topic modeling methods to facilitate the process of generating KCs for assessment items, which can be integrated in future learnersourced environments.