UWEE Tech Report Series

The Vocal Joystick: A Voice-Based Human-Computer Interface for Individuals with Motor Impairments


Jeff A. Bilmes, Xiao Li, Jonathan Malkin, Kelley Kilanski, Richard Wright, Katrin Kirchhoff, Amarnag Subramanya, Susumu Harada, James A. Landay, Patricia Dowden, Howard Chizeck

vocal joystick, voice I/O, assistive devices, speech recognition, singing, motor impairments, information interfaces and presentation, input devices and strategies


We have developed a novel device we call the {\em Vocal Joystick} (VJ) whose goal is to enable individuals with motor impairments to make use of vocal parameters to control objects on a computer screen (buttons, sliders, etc.) and ultimately electro-mechanical instruments (e.g., robotic arms, wireless home automation devices). Standard spoken language can be quite inefficient for such continuous control. Existing assistive devices, such as sip-and-puff switches, can have low communication bandwidth. The Vocal Joystick, by contrast, allows users to exploit a large and varied set of vocalizations whose selection is optimized with respect to automatic recognizability, communication bandwidth, learnability, suitability, and ease of use. While this includes regular speech sounds (phones and syllables), the primary focus of the VJ project is on the variation of individual acoustic-phonetic parameters like pitch, loudness, vowel quality, voice quality, and variations thereof. In a VJ system, vocal characteristics (vowel quality, pitch, and loudness) are in real time processed using accurate parameter extraction methods (the VJ engine), transformed via adaptation and acceleration, and are converted into continuous control signals. We have also begun a data collection effort to obtain data relevant to training VJ statistical classifiers. The VJ engine is infrastructure, capable of being integrated into any suitable application. We have thus built several VJ applications, including voice web browsers, video games, and a controlled trajectory-following environment. Initial user studies indicate that the VJ is indeed viable technology.

Download the PDF version

Download the Gzipped Postscript version