GESTURE BASED WIRELESS SHADOW ROBOTGroup# 40

A Project SubmittedBy

1. Ratul, Ahasan Ulla ID: 13-23070-12. Sumnoon, Sharif Md. ID: 11-19449-23. Kabir, Sharif Raihan ID: 12-21365-2

4. Islam, Md. Nazrul ID: 12-21730-2

Under the Supervision ofDr. M. Tanseer AliAssistant Professor

American International University - Bangladesh

INTRODUCTION In the field of robotics, gesture controlling is becoming

popular lately. People need more user friendly and interactive robots for their work. Gesture detection makes thing easy for the users.

The project proposes a robotic body which mimics the action of human body

PROJECT GOALS

Develop the shadow code to read the sensor readings. Design and develop a skeleton structure for robot frame Implementation of control mechanism or Actuators and Drive

Systems. Using of Kinect Sensor data for the real time movement of

upper portion of body. Basic non-feedback walking sequence for the forward walking

of the Shadow Robot. Create save and replay mode the shadow commands . Implement a feedback IP camera at robot head.

PROJECT BLOCK DIAGRAM

THE KINECT SENSOR

1. Infrared (IR) projector.2. IR depth sensor.3. RGB camera.4. Group of tilt motors.5. Three-axis accelerometer.6. Microphone array.

KINECT SDK

• The Kinect SDK is synthesized set of libraries of software’s and the tools that is helpful for us for using the Kinect-based natural input.

COLOR STREAM Color image

formatResolution FPS Data

InfraredResol

uzion640x480

Fps30

640 x 480 30 Pixel format

is gray16

RawBayerRes

oluzion1280x9

60Fps12

1280 x 960 12 Bayer data

RawBayerRes

oluzion640x48

0Fps30

640 x 480 30 Bayer data

RawYuvResol

uzion640x480

Fps15

640 x 480 15 Raw YUV

RgbResoluzio

n1280x960Fps

12

1280 x 960 12 RGB

(X8R8G8B8)

RgbResoluzio

n640x480Fps1

5

640 x 480 15 Raw YUV

DEPTH STREAM

Depth image format Resolution Frame rate

Resoluzion640x480Fps30 640 x 480 30 FPS

Resoluzion320x240Fps30 320 x 240 30 FPS

Resolution80x60Fps 80 x 60 30 FPS

DEPTH IMAGE

THE BODY POSITIONING

The body positioning system incorporates depth mapping and decision tree analysis.

SKELETON IMAGE

JOINT ANGLES CALCULATION FROM SKELETON IMAGE

Skeleton joints tracking from Skeleton Image. Skeleton frame data to skeleton Arrays. Angle Calculation (ex: Inverse Kinematic

approach) using X,Y,Z co-ordinates of the joints that is saved in the arrays.

INVERSE KINEMATIC ANGLE CALCULATION

ARDUINO AND SERVO MOTORS

The Arduino Mega is used in our project for two purposes.

To generate servo signals from the received strings.

To calibrate the servo motors for initial position making and for creating walking sequence

GENERATING SERVO SIGNAL USING ARDUINO MEGA

END

START

If string Available

Read HC-05

If the slave String is entirely Complete

Break the string until “&” sign

Convert the rest of the piece into servo angle

Compare servo Id if it matches with saved Ids

Break it further until “:” for servo ID

NOYES

YES

NO

NO YES

CALIBRATION OF JOINT SERVOS USING ARDUINO

Arduino can be used to control up to 12 servos with minimum jitter. Two major parts are there in the application

The first part is the firmware, that needs to upload in the arduino. The second part is to install a simple software named “Serial servo

controller”.

COMMUNICATION BETWEEN MASTER AND SLAVE BLUETOOTH

To enable Wireless connectivity that enables windows pc bluetooth , C# language is used. Bluetooth can be programmed by C# in two ways.

Using Windows Sockets. That is programmed under windows sockets class.

Using Computer’s Serial Comport.

SENDING STRING DATA FROM PC BLUETOOTH

if (serial.IsOpen){try {String hexstring = ("1" + ":" + LShoulder + "&" + "2" + ":" + LElbow + "&" + "3" + ":" + RShoulder + "&" + "4" + ":" + RElbow);;serial.Writeln(hexstring);Thread.Sleep(1); }catch (Exception ex){para.Inlines.Add("Failed to SEND" + data + "\n" + ex + "\n"); }}

NON-FEEDBACK WALKING OF BIPED

The walking of the Shadow Robot follows 8 repetitive steps.

I. Stand still II. Lean Left III. Right Step forward

IV. Stand stillV. Lean RightVI. Left step forward

DESIGN PROTOTYPE AND SIMULATION

DESIGN PROTOTYPE AND SIMULATION

DESIGN PROTOTYPE AND SIMULATION

APPLICATIONS OF THE PROJECT

Artificial Intelligence. Industrial use. Simulation. Training & Education. Assistive living.Entertainment.

FUTURE WORK

Design A Dynamic Walk sequence with feedback from Gyro and accelerometer.

Finger and Face tracking. Making an Trained and Intelligent

Humanoid .

REFERENCES

[1] Jungong Han; Ling Shao; Dong Xu; Shotton, J., "Enhanced Computer Vision With Microsoft Kinect Sensor: A Review," in Cybernetics, IEEE Transactions on, vol.43, no.5, pp.1318-1334, Oct. 2013 doi: 10.1109/TCYB.2013.2265378.

[2] P. Baerlocher and R. Boilic, Inverse Kinematics Techniques for the Interactive Posture Control of Articulated Figures, PhD thesis, EcolePolytechniqueFederale de Lausanne, 2001.

[3] Machida, E.; Meifen Cao; Murao, T.; Hashimoto, H., "Human motion tracking of mobile robot with Kinect 3D sensor," in SICE Annual Conference (SICE), 2012 Proceedings of , vol., no.,pp.2207-2211,20-23Aug.2012.

[4] Xu, D. and Acosta, A. (2005) An Analysis of the Inverse Kinematics for a 5-DOF Manipulator. International Journal of Automation and Computing, 2, 114-124. http://dx.doi.org/10.1007/s11633-005-0114-1.

[5] Stephan Waldherr; Roseli Romero and Sebastian Thrun, A Gesture Based Interface for Human-Robot Interaction. , Autonomous Robots, September 2000, Volume 9, Issue 2, pp 151-173.

[6] W. Xu and E. J. Lee, “Continuous gesture recognition system using improved HMM algorithm based on 2D and 3D space”, International Journal of Multimedia and Ubiquitous Engineering, vol. 7, no. 2, (2012), pp. 335-340.

[7] Bhuiyan , M. & Picking, R. (2009). Gesture-controlled user interfaces, what have we done and what's next?, Proceedings of the Fifth Collaborative Research Symposium on Security, E-Learning, Internet and Networking (SEIN 2009), Darmstadt, Germany, 26-27 November 2009, pp59-60.

ANY QUARIES

For further information email: join2sharif@gmail.com

Thank You !