Research Seminar - October 13, 2000

Abstract

Deaf communities in Australia use a sign language called Auslan. Building a 2-way translation system between Auslan and spoken English requires research in various areas such as gesture recognition, speech recognition and graphical sign display. We have been working on vision-based solutions towards building such a system.

Three separate systems are presented at this talk. Firstly, a fine-grain hand motion recognition system will be introduced. This system detects Auslan basic hand shapes and motion from the visual input and recognises them as a sign. It uses a combination of a 3D hand tracker for motion sensing and an adaptive fuzzy expert system for classification. Secondly, a visual lipreading technique will be presented. This technique uses a combination of image processing and acoustic speech recognition techniques. From a sequence of images, the speaker's mouth contour is tracked and used to recognise the inner mouth appearance which is achieved by using Cepstral image analysis on mouth region images in order to produce Higher Order Local AutoCorrelation (HLAC) features. These features are used for lipreading. Thirdly, a 3D head tracker that detects a speaker's head orientation will be presented. In automatic lipreading, the speaker's head movement can affect the mouth shape appearing in the captured images independently of the true mouth shape. Since such distortion can lead to incorrect recognition, the head tracker extracts 6 degrees-of-freedom head orientation parameters and then corrects the inaccurate 2D mouth dimensions detected from the images.

The talk will also discuss how these techniques can be integrated towards building a translation system.