Articulatory Movements from Speech for Pronunciation Training

Abstract

In this paper, we describe computer-assisted pronunciation training (CAPT) through the visualization of learner’s articulatory gesture. Typical CAPT systems evaluate pronunciation by using speech recognition technology, however, they cannot indicate how the learner can correct his/her articulation. The proposed system enables the learner to study how to correct pronunciation by adjusting the articulatory organs highlighted on a screen and comparing with the correctly pronounced gesture. In the system, a multi-layer neural network (MLN) is used to convert learner’s speech into the coordinate of a vocal tract using MRI data. Then, a CG generation process outputs articulatory gesture using the values of the vocal tract coordinate. Moreover we improved the animations by modifying vocal tract coordinate of important articulatory organ and training them in MLN. Lastly the comparison of the extracted CG animation from speech and the actual MRI data is investigated. The new system could generate accurately CG animations from English speech by Japanese as well as English native speech in this experiment.