Reliability Investigation of Automatic Evaluation of Kit-Build Concept Map by Comparing with Handmade Evaluation Methods

Abstract

This paper describes an investigation of the reliability of automatic evaluation of concept maps in the framework of Kit-Build concept map. A concept map is a graphical tool for organizing and representing knowledge. In this framework, a learner is provided parts of a concept map and requested to build the concept map by combining the pans- This framework provides a kit-building task ofa concept map which is a promising task to enhance and assess learners' comprehension in a topic that they already learned. The framework is practically used in several kinds of school. However, the assessment reliability has not investigated. In this research, we try to investigate the reliability of assessment learner's comprehension of Kit-build concept map by comparing with handmade evaluation methcxis of the concept map. Two well-known handmade concept map evaluation methods that include the structural and propositional scoring method are chosen for comparison. These handmade methods can evaluate concept map meaningfully. Moreover, it is flexible for scoring because the human can understand the meaning of each proposition in concept map even the words of proposition do not appear in a learning material. So, the handmade methods are claimed they have reliability for evaluating concept map. To confirm our hypothesis, we designed the preliminary experiment in two learning comprehension situations. Those are learners' reading and instructor's teaching situations. In a preliminary experiment, the correlation between Kit-Build concept map and two handmade concept map evaluation methods in teaching situation has the marginal medium correlation. Even though this is a preliminary result, it suggests that Kit-Build concept map evaluation is reliable for evaluating concept map in the teaching situation.