Automated Classification of Bloom's Taxonomy Levels for Japan's Information Technology Engineers Examinations

Abstract

Understanding the cognitive demands of assessment items is essential for providing evidence-based learning support. This study analyzes Japan's Information Technology Engineers Examinations (ITEE), a national qualification exam system whose hierarchical proficiency levels align with the cognitive stages of the revised Bloom's Taxonomy. We first conducted a manual classification of exam questions across all four levels and found that the distribution of cognitive process levels remains largely consistent regardless of proficiency level, suggesting that exam difficulty is differentiated by factors other than cognitive complexity, such as the depth of domainspecific knowledge. To alleviate manual classification burdens, we compared four automated approaches: a lexical baseline (TF-IDF), a fine-tuned transformer (DeBERTa), LLM prompting (Gemini), and a multi-agent deliberation framework (Claude). Results show that while four-class classification remains challenging, binary classification into lower-order and higher-order thinking skills achieves high accuracy across all four baseline methods. Among the four-class approaches, the multi-agent method outperformed LLM prompting single-agent classification, demonstrating the value of structured deliberation for nuanced cognitive level assessment. These findings contribute to the development of metrics for characterizing exam content at scale, which can inform adaptive learning systems that optimize practice based on learners' cognitive skill profiles.