Development of Sign Language Training Machine using Depth Sensor

Abstract

In this paper, we describe a development of Japanese sign language training machine using depth sensor that targets a beginner of Japanese sign language user and supports a learner by feeding back mistake information. This system consists of two parts, one is a input device of sign language and the other is a personal computer for study. That input device consists of flex sensors and accelerometer installed on gloves, and a depth sensor. The depth sensor (Xtion) is used to acquire the position of the hand and head, because this information is very important to analyze a sign language. Correct or incorrect determination s of input sign langu age motions are very important to achieve the sign language study accurately. Therefore, this thesis especially describes the correction judgment of sign language. For the internal processing, we defined fundamental motions and forms of a sign language, and modeled sign language motions by combination of fundamentals. These fundamentals are defined by some information of the shape of the hand, the inclination, the position and the direction of the movement. The number of sign language words necessary fo r communication with a hearing impaired is about 2,000 words. As the first step, we selected 231 words that an elementary conversation is possible and constructed the system. As a result, we are able to achieve correction judgments that have high accuracy. On the other hand, the problem in a present method is clarified, too.