Reproducibility Evaluation of Real-World Educational Evidence Extracted by Causal Inference

Abstract

For enhancing the reliability of Evidence-Based Education (EBE), the evaluation of reproducibility of causal relationships extracted by causal inference from Real-World educational Evidence (RWeE) is crucial. To address this challenge, this study attempted to quantitatively evaluate the reproducibility of educational evidence derived from causal inference by systematically applying the Correspondence Test (CT). Specifically, features of learning behavior were generated from summer vacation learning log data of junior high school students in grades 1 to 3 (6 datasets, n=109-117), and causal relationships associated with changes in deviation scores were extracted using LiNGAM-MMI, which can account for unobserved confounding factors. The results of the CT reproducibility evaluation revealed that factors related to “maintaining a learning rhythm” exhibited high reproducibility with a practically significant effect size, leading to the successful extraction of robust RWeE. This study establishes a new method that enables the quality assurance of RWeE by integrating causal discovery and CT, and demonstrates that these reproducible findings provide the foundation for designing high-quality educational interventions.