7/27/2019 Gw Paper

http://slidepdf.com/reader/full/gw-paper 1/4

The Gesture Watch: An alternate gesture based interfacefor remote control of devices

Gaurav Gupta

Georgia Inst of TechAtlanta GA 30332gaurav@cc.gatech.edu

Jiasheng He

Georgia Inst of TechAtlanta GA 30332 jiazhe@cc.gatech.edu

Jung Soo Kim

Georgia Inst of TechAtlanta GA 30332 jszzang@cc.gatech.edu

ABSTRACT

This paper describes the Gesture Watch, a gesture based in-terface for the remote control of devices. By subscribing tothis interface, any device can become gesture-smart and thuswill not need a separate physical interface with the user. TheGesture Watch is designed to be embedded inside the wrist-watch of user, allowing it to be unobtrusive. It takes advan-tage of the developments in infra-red technology and ma-chine learning algorithms, allow for the robust sensing and recognition of gestures.

Author Keywords

The Gesture Watch, gesture, interface, control, remote

INTRODUCTION

With advances in micro-technology, mobile devices such ascell phones and MP3 players are growing smaller and morelightweight. An MP3 player could conceivably become thesize of a hearing aid that could fit snugly inside your ear.However, the problem then becomes input. How do you ad-vance to the next song when the MP3 player is so small thatit is inside your ear? In such a case, attaching buttons or ascroll wheel to the MP3 player would impractical because it

makes the device too big.

Figure 1. The size of the human hand is what constrains the new iPodShuffle to go smaller.

One solution for this problem is using remote controls. Withthe first television remotes appearing around 60 years ago,the concept was to not have to physically reach out and touchthe television to control it. Many novel ideas and conceptswere devised based on remote control of devices that madesome part of everyday life easier for humans. But whenremotes were designed, they had one inherent flaw: to re-motely control an appliance, a person would have to phys-ically hold another (i.e. the remote). When this is cou-pled with the fact that the remote control interfaces were not(and still are not) standardized, the average person in today’s

world may find himself overloaded by remotes. At a giveninstant of time, a person would have two or more remotes inhis proximity while he may need only one of them. Thus,the problem now becomes, can we design a universal remoteinterface that could serve as a common point of control for avariety of appliances?

RELATED WORK

In the past, many researchers have developed clever inter-

faces that could allow a person to control all devices in hisperiphery using a single interface. These include XWeb[1],

Light Widgets[2], Magic Wand[3], the FreeDigiter[4], the Ges-

ture Pendant[5], Smart Snakes[6] and the Gesture Pendant II.Of these, the FreeDigiter, the Gesture Pendant, and the Ges-ture Pendant II were developed at the Georgia Institute of Technology and relied on using infrared. The others variedin implementation.

Light Widgets used a system comprised of two cameras thatdetected the position and movement of hand on any surface.These hand gestures served as a control input to the XWebplatform. Another system that used the XWeb cross-modalplatform was the Magic Wand, which had lasers and an in-

expensive camera to generate control input. Smart Snakesmeanwhile recognized hand gestures from a 15-bit color videostream using genetic algorithms. Of the Georgia Tech cre-ations, the FreeDigiter was worn on the ear and could countthe number of fingers moving past the sensor whereas theGesture Pendant was worn on the neck and had a cameraringed by infra-red LEDs that illuminated the hand of theuser. This camera would successfully recognize the gesturebeing made. Our previous work, the Gesture Pendant II, usesfour proximity sensors arranged in a cross formation insteadof a camera to detect the hand gestures.

THE GESTURE WATCH

The Gesture Watch is designed to allow the control of differ-ent devices through the use of hand gestures. A user wearsthis device on his left arm just like a wristwatch and useshis right hand to perform gestures in the air at a moderateheight range of approximately 5 to 20 centimeters above thewatch face. The Gesture Watch then detects these gesturesand appropriate control signal is sent over to the appliancethat needs to be controlled. In contrast to the FreeDigiterthat could just count the number of fingers that went pastit, the Gesture Watch is specifically designed so that it canrecognize complex gestures made by the user. It enables the

1

7/27/2019 Gw Paper

http://slidepdf.com/reader/full/gw-paper 2/4

user to build up a rich library of different gestures that couldthen be used for a variety of actions.

Figure 2. 3D model of the Gesture Watch

The Gesture Watch consists of five infrared sensors whichare used to detect the proximity of user’s hand to each oneof them (Figure 2). Each sensor emits a positive signal whenit senses a hand in front of it, or emits no signal otherwise.When viewed through a given period of time, the sequenceof the combined outputs of these sensors specifies a ges-ture, with different sequences mapping to different gestures.

Thus, in principle, each exclusive sequence can indicate adistinct gesture. However, due to the imperfections in place-ment and sensing of the sensors coupled with the conicalviewing volume of infrared, some gestures might be ambigu-ous. The user might not be able to use the complete space of all possible combinations. Also, since the infrared sensorsgather data continuously, one of the major considerations issegmenting the true data (when a user is gesturing) from theinvalid data (infrared signals bouncing off from walls, furni-ture, book and other obstacles) to avoid false positives. Oursystem achieves this by using one of the sensors as a trig-ger switch. This sensor is mounted vertically on the frontedge of the watch so that it sweeps in a horizontal direction.The user flicks his wrist up to cover this sensor whenever

he needs to make a gesture. He then gestures with his otherhand over the four remaining sensors, which are mounted inthe horizontal plane of the user’s left arm and are facing up.This is done all while keeping his triggering wrist up. Tosignal the end of the gestures, the user flicks his wrist back down and the sensors data of this interval is sent over forrecognition. In order to prevent false triggering of the sen-sors by some object, a time windowing system is employed.If the trigger interval falls between 1 to 5 seconds, the sen-sor output is taken to be valid and is sent over for gesturerecognition. Otherwise it is ignored.

Example: A man walks into his living room, triggers hiswatch, and holds his hand for two seconds over the left sen-

sor. This signals that he wants to control his room lightingsystem. Then he makes an up gesture moving his hand overthe sensors perpendicular to his arm and going away fromthe body. This makes the room light up. Now wanting to lis-ten to music, this user holds his hand over the right sensor,activating his home entertainment system. With a circularclockwise gesture, he is able to increase the volume of hisMP3 player.

SYSTEM OVERVIEW

Figure 3. Block Diagram of the Gesture Watch

Figure 3 shows a high-level block diagram of the whole sys-tem. The output of infrared sensors goes to a PIC micro-controller which then sends it over to the Bluetooth chip. Aremote application listening for new gesture data receives itwirelessly through Bluetooth. This data is then processed by

a GT2K[7] enabled application that emits the gesture corre-sponding to the sensor outputs. This gesture is finally passedover to the control circuitry that does the related action. Thewhole interface is designed to promote scalability in terms of number of devices that can be controlled and the vocabularyof gestures that can be tagged. Using wireless communica-tion, multiple receivers can listen to the same data streamat any given instant of time and can independently take de-cisions. Also, instead of sending high level gesture tags,the Gesture Watch sends raw data over the channel so thateach device can map its gestures according to its own needs.Transmitting high-level gesture information would limit thenumber of control signals that devices can use by the vocab-ulary of the Gesture Watch and thus would not scale well formultiple heterogeneous devices.

HARDWARE

Figure 4. Prototype of the Gesture Watch

The Gesture Watch uses SHARP GP2Y0D340K proximitysensors that can detect objects in the range of 10-60 cm. Inpractice, this range is best from 5-20 cm. This is the dig-ital version of SHARP proximity sensors that simply out-puts a low when it detects an object and stays high otherwise. It requires around 4.5 to 5V Vcc to work and measures15x9.6x8.85 mm. The data from these sensors is sent overto PIC16LF873 microcontroller that packetizes this data andhands these packets over to the Taiyo YudenEYMF2CAMM Bluetooth module. These packets are thentransferred via Bluetooth to a remote receiver. Both the PICand the Taiyo Bluetooth modules run on 3.6V, and due to thedifference in the voltage requirements of the infrared sen-

2

7/27/2019 Gw Paper

http://slidepdf.com/reader/full/gw-paper 3/4

sors and PIC module, some voltage regulation circuitry isalso needed.

SOFTWARE

The software running on the receiver side does the actualwork of tagging the sensor data with its corresponding ges-ture. In our reference implementation, this software runson an Apple MacBook with Mac OS X. It employs Geor-

gia Tech Gesture Toolkit (GT2

K) to do the actual work of recognizing the gestures. As stated on the GT2K website:“GT 2K leverages Cambridge University’s speech recogni-tion toolkit, HTK, to provide tools that support gesture recog-nition research. GT 2K provides capabilities for training mod-els and allows for both real-time and off-line recognition.” .HTK employs Hidden Markov Models (HMM) for recogni-tion. Our software dumps raw segmented data to the GT2Kengine, which emits the corresponding gesture name that isthen used to perform the related action. In our implementa-tion, a user can successfully control iTunes software by hisgestures. Table 1 shows a subset of these gestures and thecorresponding actions.

Gesture CommandUp Go to previous track  

Down Go to next track  Clockwise Increase volume

Counter-Clockwise Decrease volume

Table 1. Subset of Gestures for iTunes control

Figure 5. A user gesturing on the Gesture Watch

APPLICATIONS

A gesture-based remote control device could possibly have

numerous applications. In addition to the iTunes controlsoftware, the authors were also able to successfully controla television set with a prior iteration of the Gesture Watch,namely the Gesture Pendant II.. This uses the similar sensorsto the Gesture Watch in an analogous configuration, with theexception being that its form factor is in the shape of a pen-dant hung from the neck rather than a wristwatch-like inter-face. The following part of the paper discusses this imple-mentation.

Overview

Figure 6. The Gesture Pendant II has four proximity sensors placeddiagonally and a Bluetooth module

The Gesture Pendant II is a prior version of the GestureWatch. It uses four proximity sensors that are similar tothose of the Gesture Watch in analogous configuration, but

its form is different. The Gesture Pendant II is shaped like apendant hung from the neck rather than a wristwatch-like in-terface. Figure 6 shows the Gesture Pendant II prototype. Ituses a combination of four infrared sensors and a Bluetoothplatform to detect a set number of gestures. In addition, theGesture Pendant II uses an on/off switch to segment eachgesture instead of using a separate proximity sensor that likethe Gesture Watch does. Users can switch between the Ges-ture Watch and Gesture Pendant II as they share the sameplatform and data interface. For example, one can wearthe Gesture Watch while he/she is jogging to control his/herMP3 player or wear the Gesture Pendant II to control variousappliances at home.

An Infrared Remote Control System

The gesture-based remote control device has many possibleapplications. In addition to the iTunes control software, wewere also able to successfully control a television set withthe Gesture Pendant II. With the Bluetooth module, one canonly control computers or mobile devices with Bluetooth ca-pability. However, Bluetooth can not be used to control otherhome appliances such as television, DVD player, radio, etc.This goal can be achieved with the infrared control systemby sending IR commands to the home appliances that use in-frared remote control. Figure 7 shows infrared codes that arerecorded from Samsung SRC1000 remote control. The re-mote control system can record IR commands from variousremote controls and send the commands back to the system.We used the USB-UIRT (Universal Infrared Receiver andTransmitter) infrared module which can detect 36-40 KHzfrequency range because most remote controls use the fre-quency of 38 KHz. Moreover, by using EventGhost, an opensource automation tool, we could also control Windows ap-plications with gestures.Table 2 shows a set of gestures thatwe use to control Windows movie player. Since the GestureWatch and the Gesture Pendant II share similar platform, thegestures are very similar and we can reuse the gestures tocontrol any applications.

3

7/27/2019 Gw Paper

http://slidepdf.com/reader/full/gw-paper 4/4

Figure 7. Recorded infrared remote control codes

Motion Gesture Name CommandA hand rotatingover the sensorsin a clockwisepattern

FF Fast Forward

A hand rotatingover the sensorsin a counter-clockwisepattern

REW Rewind

A hand blockstwo bottom sen-sors

POWER Turn on movie player

A hand blocksa right-bottomsensor

PLAY Play/Pause movie

A hand movesfrom bottom totop

VOLUP Increase volume

A hand movesfrom top to bot-tom

VOLDOWN Decrease volume

Table 2. A subset of gestures for Gesture Pendant II

Gesture Recognition

We used GT2K to train and recognize hand gestures. Fig-ure 8 shows a simple GT2K program that shows each of foursensors in the column. Figure 9 shows a confusion matrixfrom the training. We trained ten samples in each gesture.Among the ten gestures, seven were used for training andthree were used for testing. The matrix tells us that all threetest samples in each gesture are classified correctly. Aftertraining using the GT2K, a user can change to recognitionmode. In the recognition mode, whenever a user sends asample it returns the classified result and sends an IR com-mand according to the commands shown on Table 2.

By having IR remote system, we could successfully controlboth computer applications and home appliances. With theinfrared remote control system, we could expand the Ges-ture Watch and Gesture Pendant II’s ability to control home

appliances.

Figure 8. Simple Gesture Pendant II GT2K application

Figure 9. A confusion matrix from training result with 10 samples eachgesture

REFERENCES

1. Olsen Jr. , Jefferies, Nielsen, Moyes, Fredrickson,Cross-modal Interaction using XWeb.

2. Fails, Olsen Jr. , Light Widgets: Interacting in Every-daySpaces.

3. Fails, Olsen Jr. , MagicWand: The True UniversalRemote Control.

4. Metzger, Anderson and Starner (2004), FreeDigiter: AContactfree Device for Gesture Control.

5. Starner, Auxier, Ashbrook, and Gandy (2000), Thegesture pendant: A self-illuminating, wearable, infraredcomputer vision system for home automation control andmedical monitoring.

6. Heap and Samaria (1995), Real-Time Hand Tracking

and Gesture Recognition Using Smart Snakes.7. Westeyn, Brashear, Atrash and Starner (2003), Georgia

Tech Gesture Toolkit: Supporting Experiments in GestureRecognition http://gt2k.cc.gatech.edu/ 

4