1

GESTURE BASED SYSTEM CONTROL

Developed by

Rohan C. Vardekar

rovardekar@gmail.com

2

INDEX Page No.

1)Introduction 3

2)Market Survey 4

3)Sensors Used 6

4)Integration and Communication 8

5)Block Diagram 12

6)Application 13

7)Conclusion 16

8)Future Scope 17

9)Challenges 17

10)References and Achievements 18

3

INTRODUCTION

The communication between humans and machines is defined by buttons. It has been more than century , but we are still using buttons. Even on the Computer's and the Smart phone's screen ,we can see lot of buttons.

The main aim of project is to provide more natural way of communicating with systems .This project focuses on implementing general human gestures to interact with systems . For example , on making action of opening the door, the door will get open automatically . Another example can be with specific action associated electrical device will ' Turn On', in the similar manner with reverse action the same device will 'Turn Off' . The applications are not limited with physical devices, we can also interact with computers with the help of gestures.

4

MARKET SURVEY 1)Kinect - Game developers may be abandoning Kinect , but the device is still the most immersive gesture-based controller available for a game console . It can sense where a user is standing, how the user is positioned and how he or she is moving.

Figure 1:Kinect Sensor as gaming console

2)Nymi Band - The main purpose of Nymi band is identification based on ECG. The Nymi band also consists of accelerometer for gesture based communication . But the gesture based application has not been launched by company yet . It is to be noted that the cost of Nymi band is high.

Figure 2 : Nymi Bamd

5

3)Star Band - Founded in 2013, Belgaum based company SenseGiz Inc. has come up with STAR, an affordable multi-purpose smart-device that can be clipped to clothing or worn as a band. This tiny device can track the user’s fitness and sleep quality, provide idle alerts, display time, deliver calls, texts or mobile app notifications, control smart-phone functions with simple gestures, trigger automatic alarm on loved ones’ phone numbers if they suffer a fall or crash, and has a panic button to get help in an emergency.

Figure 3: Star Band

4)MYO - The muscle-sensing armband from Thalmic Labs, is one of the more unique gesture control devices created . The company is backed by Intel and its product does not hinder the user experience with cameras or wires.

Figure 4 : MYO Band

6

SENSORS USED Accelerometer Sensor ,Compass Sensor and Flex Sensor are used in this project. Gesture is detected by combination of signals obtained by these three sensors.

1)Accelerometer Sensor - An accelerometer is an electromechanical device that will measure acceleration forces. These forces may be static, like the constant force of gravity pulling at your feet, or they could be dynamic - caused by moving or vibrating the accelerometer. By measuring the amount of static acceleration due to gravity, you can find out the angle the device is tilted at with respect to the earth. By sensing the amount of dynamic acceleration, you can analyze the way the device is moving.

Figure 5 : Accelerometer Sensor Orientation

In this project MMA7660FC with ±1.5 g 3-Axis Accelerometer which gives Digital Output(I2C) is used . It is a very low power, low profile capacitive MEMS sensor featuring a low pass filter, compensation for 0g offset and gain errors, and conversion to 6-bit digital values at a user configurable samples per second.

7

2)Digital Compass Sensor - The Honeywell HMC5883 is a surface mount multi-chip module designed for low field magnetic sensing with a digital interface for applications such as low cost compassing and magnetometry. Honeywell developed ASIC containing amplification, automatic degaussing strap drivers, offset cancellation, 12-bit ADC that enables 1° to 2° compass heading accuracy. The I2C serial bus allows for easy interface.

Figure 6 : Compass Sensor Orientation

3)Flex Sensor - It is special type of variable resistor. Resistance varies with bending of sensor. In order to achieve signal from bending , flex resistor is used with potential divider circuit. For this project 2.2 inch flex resistor is used.

Figure 7 : Flex Sensor Circuit diagram

NORTH 0

degrees

EAST 90

degrees

SOUTH 180

degrees

WEST 270

degrees

8

INTEGRATION AND COMMUNICATION

Accelerometer - In this project accelerometer is used to sense static acceleration i.e. acceleration due to gravitational force only. The accelerometer sensor is placed such that the negative Z axis points to the ground and mounted on back of palm.

Figure 8 : Accelerometer Sensor Mounting

Action X axis Reading Y axis Reading Z Axis Reading

0

0

Most +ve

0

0

Most -ve

9

0

Most +ve

0

0

Most -ve

0

Most +ve

0

0

Most -ve

0

0

Figure 9 : Various Gestures and associated accelerometer readings

Flex Sensor - is mounted and fixed on the back of the index finger .Thus the movement of finger can be tracked by the flex sensor. Now we can determine whether palm is closed or open depending upon the signal generated by flex sensor . In this case gesture will be only considered when the palm is closed .

Figure 10 : Close and open Palm can be detected by flex sensor

10

Compass Sensor - It is mounted perpendicular to the accelerometer sensor . We will be considering readings from compass only when Y axis readings are most positive or most negative . Another important point regarding compass sensor is , it is referenced with earth's magnetic field . Thus in order to maintain consistency in gestures ,a new reference is considered each time 'Palm Closed' is detected further deviation in angle due to gesture is calculated by taking difference between reference reading and actual reading . If we from North in clock wise direction the angle continuously increases but after completing 1 rotation and coming back to North , angle starts decreasing thus special algorithm is implemented to maintain continuity in gestures.

Figure 11 : (a)Hand moved to left to open the door (b)Original Position (c) Hand moved to right to close the door

Integrated Sensors Alignment - The final sensor alignment is as shown in the figure .It is to be noted that ,by adding no. of flex sensors for different fingers, total no. of actions can be increased.

Figure 12 : Complete integrated system (a) With glove (b) With Wrist band

11

Figure 13 : Side View of the gesture sensing system

First method implemented for interface was smart glove but it is more comfortable to use wrist-band and finger-band combination for mounting the sensors for gesture recognition.

Bluetooth module HC-05 is used along with this device in order to communicate with server. Gesture can be recognized by microcontroller connected with these sensors and Bluetooth module or by the main server.

There are two methods for defining gestures . First method is hard-core programming for gesture . Second method is by recording gesture with certain amount of tolerance. Here first method is implemented. In order to differentiate between command and normal human action, gesture will be only considered when palm is closed i.e. flex sensor reading reaches certain threshold . Flex sensor carries highest priority . Accelerometer sensor carries second priority. Compass sensor carries third priority . It is only used when Y axis reading for accelerometer is either most positive or most negative.

12

BLCOK DIAGRAM

Figure 14 : Block Diagram

Micro-controller BLE ADC

Acceler--ometer

Compass Sensor

Flex Sensor

Regulated Power Supply

VCC GND SDA SCL

13

APPLICATIONS 1) Door Control : Door lock can be opened or closed with the help of gesture .

Figure 15 : Door Figure 16 : Original Position

In order to open the door action of opening the door is performed. To perform the action , first tilt the hand into original position as shown above diagram and then turn it to left side by more than 30 degrees to open the door.

Similarly to close the door action of opening the door is performed. To perform the action , first tilt the hand into original position as shown above diagram and then turn it to right side by more than 30 degrees to close the door.

Figure 17 : Open the door Figure 18 : Close the door

14

2) Smart Box Control : Smart box is the device designed during this project consisting of three ports where any household electrical device can be connected and can controlled remotely using digital signal received from Bluetooth receiver.

Figure 19 : Smart Box

Port 1 Control : To turn on Port 1 Gesture performed so that Z axis reading of accelerometer will have maximum of positive reading than others. Similarly to turn off Port 1 reverse action is performed i.e. Z axis reading of accelerometer will have maximum of negative reading than others. In this compass sensor is not considered in gesture.

Figure 20 : Port 1 ON Figure 21 : Port 1 OFF

15

Port 2 Control : To turn on Port 2 Gesture performed so that X axis reading of accelerometer will have maximum of positive reading than others .It can be achieved by pointing knuckle upward . Similarly to turn off Port 2 reverse action is performed i.e. X axis reading of accelerometer will have maximum of negative reading than others. Note that, here again compass sensor is not considered in gesture.

Figure 22 : Port 2 ON Figure 23 : Port 2 OFF

16

CONCLUSION Gesture control provides better user experience . Any number actions can be defined as there is no limit on human gesture , but in order to avoid interference between gestures number of action should be limited . Gesture technology has bridged in the gap between physical and digital world . Using sensors instead of camera is better for gesture recognition as it has more consistency and lesser dependency over the environment. Combination of Flex Sensor , Accelerometer Sensor and Compass Sensor is most suitable for gesture recognition.

17

FUTURE SCOPE 1) This technology can also extended to control computers as substitute to mouse.

2) It will give more realistic feel to graphic designer and architects.

3) It is very much useful in robotic arm control in case of remote medical operations , Outer Space Missions , remote cleaning and painting operations.

4) This tool can also be used to provide audio feedback to actions for people who cannot talk .

5) This tool is useful for older people and physically disabled people so that they can control their house with gestures from distance.

6) This is an excellent tool for developers to design new and innovative applications to interact with their surroundings.

7) This technology can also be used as gaming console for more realistic experience.

CHALLENGES 1) Defining new action requires hard programming . No. of actions are limited ,then only we will be able to avoid interference between action .Thus it is required to design a way to record the gesture instead of programming it .This system should also tell whether new gesture is similar to any gesture from database.

2)Consistency is required in gestures just like the computer mouse where with limited number of actions we can operate various programs at the same time.

3)With limited number of gestures, we should be able to select which device to be operated i.e. in order to open the door same gesture will be there but we should be able to select which door to be opened.

4)Hardware needs to be minimized in size.

18

REFERENCES 1) Accelerometer Sensor datasheet->MMA7660FC.pdf

2)Compass Sensor datasheet->HMC5883.pdf

3)Bluetooth module datasheet->HC-05.pdf

ACHIEVEMENTS With this project I have won First rank in Intel IOT-hackathon , Pune in 2015.