Performing Mouse Events by Hand Gesture Recognition

T.K. Manglam (MS®,C.Tech) Punit Tiwan (MS®,C.Tech) Akhil Bansal (MS®,C.Tech)

Performing Mouse Events by Hand Gesture Recognition

Project taken as part of course “ Multimedia Systems “ under the guidance of Prof. Santanu Chaudhary .

Introduction

So what has been Done ?

• Various combinations of finger movements have been decided , where each combination corresponds to some particular mouse event .

• The presence of a particular combination is recognized to decide which mouse event to perform.

• To recognize finger movement , image segmentation has been used .

• For good segmentation results , color image segmentation has been used .

• For efficient segmentation , colored caps with different hues have been used .

So what has been Done ?

• Perform various mouse events by moving fingers • Various combinations of finger movements have been

decided , where each combination corresponds to some particular mouse event .

• The presence of a particular combination is recognized to decide which mouse event to perform.

• To recognize finger movement , image segmentation has been used .

• For good segmentation results , color image segmentation has been used .

• For efficient segmentation , colored caps with different hues have been used .

Functionalities developed :

• Move mouse pointer by hand gesture .• Right Click mouse event by hand gesture • Left Click mouse event by hand gesture .• Drag/Drop an item by hand gesture .

Specifications

• Platform : Windows7• Processor : Core2Duo• RAM : 2GB

The application has not been tested on lower

configurations , however it should work on system with lesser RAM and slower processor as well .

Hardware and Other Requirements

• A camera , which may be a web-cam fixed on laptop or an external web-cam .

• 2 caps of colors with different hue .

Tools Used

• MSDN library : The functions defined in <windows.h> have been used to move perform various events on mouse .

• OpenCV library : The funtions defined in OpenCV libraries have been used for quick reference .

Flow of Algorithm Implemented

• Learn colors of caps to be used for segmentation .

• Recognize the presence of each colored cap and segment the colored caps in each frame

• Determine the mouse event to be performed

• Find the displacement of colored cap in consecutive frames and move the cursor on screen .

Implementation of

Algorithm

Algorithm ImplementedStep 1 : Learn colors of caps to be used for segmentation .

• Initially start the camera using cvCameracapture( ) .

• Bring both caps in front of the camera and click an image using cvQueryFrame( ) .

• In the image , click at any point on the image of caps one by one .

• Determine the position of the point clicked on using GetCursorPos( ) .

• Find the hue component of the corresponding pixel in the image using setLimit( ).

Detecting Colors

Algorithm Implemented

Step 2 : Determine the presence of each colored cap and segment the colored caps in each frame

• Capture the image continuosly using while( ) loop .

• Change the image in HSV plane using cvConvertScale( )

• Find pixels which lies in the range of hue of a colored cap .

• Find the contour of the caps in the image using cvFindContour( ) .

Problems Faced

Step 1 : Learn colors of caps to be used for segmentation .Step 2 : Determine the presence of each colored cap and segment

the colored caps in each frame

• Segmentation using RGB space results in poor segmentation .

• Segmentation using connected components algorithm results in slower segmentation .

• Determining Hue of the colored caps using getPixel( ) of <windows.h> give different range of Hue .

Algorithm Implemented

Step 3 : Determine the mouse event to be performed

Assumption : First color is green color and second color is red color .

• Case 1 : Only green color present in image taken : Move mouse according to the displacement of green cap in present and previous image .

• Case 2 : Only red cap is present : Use left movement of red cap for Left Click , and right movement of red cap for Right Click

• Case 3.a : Both green and red cap present in image : No movement of mouse even if green cap is moved

• Case 3.b: Both green and red cap present in image and are nearby : Drag the mouse according to the displacement of caps .

Moving Mouse Pointer

Right Click and Left Click

Dragging and Object

Problems Faced

Step3 : Determine the mouse event to be performed

• A lot of combinations were tried before deciding on final combination .

• A nice observation is that finger in the image moves in the opposite direction

Algorithm Implemented

Step 4 : Find the displacement of colored cap in consecutive frames and move the cursor on screen .

• Stepwise velocity taken to make cursor movement look realistic .

• For very slow motion of finger , cursor doesn’t move .

• For very fast motion of finger , cursor doesn’t move .

Problems Faced

Step4 : Find the displacement of colored cap in consecutive frames and move the cursor on screen .

• A lot of combinations were tried to make mouse pointer movement look realistic .

• Since finger doesn’t have any support , so it keeps on displacing , and hence valid assumptions need to be tried to ignore small motion .

• If finger moves out of range of the camera, then bringing back finger cause motion of cursor in opposite direction , which might not be desirable .

Other Problems

• Improper Illumination Problem• Illumination Changes• Color of background shouldn’t

contain any portion having same color as that of caps .

• The finger movement needs to be constrained in the range of camera which sometimes is impalpable experience for first time user .

Future Aspects

• For our project we intend to use mobile camera for capturing images and using bluetooth to communicate with computer .

• Otherwise , Hand Gesture recognition to control events has a lot of scope e.g. streching an image by hand , playing games .

Is the code ready for commercial purpose ?

Not yet !!!! Since the most important component of a

software cycle ,i.e, testing has not been done rigorously .

However , we believe with little more hit and trial and testing , it can be made good enough for commercial purposes .

ThanK YoU