Reliable Peer Assessment for Team-project-based Learning using Item Response Theory

Abstract

This study addresses the reliability of peer assessment in team-project-based learning using item response models that incorporate rater characteristic parameters. The following difficulties can arise when applying item response models to peer assessment in team-project- based learning. (1) Earlier item response models incorporate the assumption that peer rating data consist of three-way data, which are learners × performance tasks × raters. However, the data used for this article are four-way data, which are learners × project tasks × raters × dimensions of abilities because multiple dimensions of learner abilities are generally assessed in project-based learning. Previous models are not directly applicable to four-way data. (2) In team-project-based learning, the learners are partitioned into several teams and are peer assessed within each team. In this case, the peer assessment reliability depends on the team assembly because the ability estimation accuracy for each learner depends on the characteristics of peer raters within the same team. If the teams are assembled randomly or fixed for all project tasks, then differences in the accuracy of ability estimation among learners would increase. To resolve this problem, this article presents a proposal of reliable peer assessment in team-project-based learning. Concretely, we extend the previous item response model that incorporates rater characteristic parameters to apply to four-way data. Furthermore, we formulate a team assembly problem for team-project-based learning as an integer programming problem. The team assembly method maximizes the difference between teams that are assembled for a project task and those assembled for previous project tasks.