The Sonification and Learning of Human Motion

Abstract

This paper examines how sonification can be used to help a student emulate the complex motion of a teacherinteractively with increasing accuracy and timing. The system captures the motion of a single joint (such as a handmotion or gesture) of a teacher in real-time. A 3-D motion path is then generated and recorded along with a referencesound. A student then attempts to perform the motion and thus recreate the teacher’s reference sound. The student’ssynthesized sound will dynamically approach the teacher’s sound as the student’s movement becomes more accurate.Several types of sound mappings which simultaneously represent time and space deviations are explored. For theexperimental platform, a novel system that uses low-cost camera-based motion capture hardware and open sourcesoftware has been developed. This work can be applied