Development Of Control Software For Mobile Robot Using Image Processing and Artificial Intelligence

International Journal of Advance Foundation and Research in Computer (IJAFRC)

Volume 2, Issue 12, December - 2015. ISSN 2348 – 4853, Impact Factor – 1.317

72 | © 2015, IJAFRC All Rights Reserved www.ijfarc.org

Development Of Control Software For Mobile Robot Using

Image Processing and Artificial Intelligence

Mukul Pathak1,Shwetant Mohapatra2, Kshitij Kamalakar3,Prof. Uma Nagaraj

Dept. of Computer Engineering, MIT Academy Of Engineering, Savitribai Phule Pune University Pune, India [email protected], [email protected] [email protected]

A B S T R A C T

In this project our main aim is to design and develop the control software for the

deployment and alignment of the flippers/wheels mechanism of a stair climbing robot.

The robot platform is a differential drive, with skid steering system. The system is mounted

on a rugged chassis. Vision sensors are mounted on the robot. These are cameras which will

provide motion images of the robot’s surroundings. The application software will apply

image processing and artificial intelligence techniques to detect stairs at Real-time and align

the flipper/wheel at an appropriate distance from the stair. Use of canny edge detection

method to detect the edges of the stairs, smoothen the image and removing noise from the

image. Neural networking will be used to detect stairs and faults. Machine learning

technology to overcome faults in stairs and act accordingly from the saved experiences. This

will be Linux based application which will have support of OpenCV API.

INDEX TERM :Pattern Recognition, shape detection, automation and Security.Artificial

Intelligence, Image Processing, Neural Network, Opencv API, Mobile Robot.

I. INTRODUCTION

Autonomous ground robots have traditionally been restricted to single floors of a building or outdoor

areas free of abrupt elevation changes such as stairs. Although autonomous traversal of stairways is an

active research area for some humanoid and ground robots, the focus within the vision and sensor

community has been on providing sensor feedback for control of the mechanical aspects of stair

traversal, and on stair detection as a trigger for the initiation of autonomous climbing, rather than

on stair traversability. The restriction to a single floor presents a significant limitation to real-world

applications such as mapping of multi-floor buildings and rescue scenarios. Our work seeks a

solution to this problem and is motivated by the rich potential of an autonomous ground robot

that can climb stairs while exploring a multi-floor building. A comprehensive indoor exploration

system could be capable of autonomously exploring an environment that contains stairways,

locating them and assessing their traversability, and then engaging a platform-specific climbing routine

in order to traverse any climbable stairways to explore other floors. The physical properties of a stairway

may limit the platforms that are capable of climbing it. For example, a robot may not be able to climb

some stairways due to step height, and a ground robot may be restricted to stairways with a low pitch

due to its weight distribution. Our proposed approach is an effort to integrate the existing work in

autonomous stair climbing with autonomous exploration: a system to detect and localize stairways in

the environment during the process of exploration. With a map of the environment and estimated




locations and parameters of the stairways, the robot could plan a path that traverses the stairs in

order to explore the frontier at other elevations that were previously inaccessible. For example, a

robot could finish mapping the ground floor of a building, return to a stairway that it had previously

discovered, and ascend to the second floor to continue exploring if that stairway is of dimensions

(i.e. step height, width, pitch) that are traversable by that particular platform. Our proposed system

directly addresses the needs of an exploratory platform for solving the problem defined above. We

seek to answer the question Is this traversable? when a stairway is discovered during exploration.

The questions When should I traverse that? and How do I traverse that? are left for the path planner and

stair climbing routines, respectively. The system is composed of a stairway detection module for

extracting stair edge points using single camera. The physical properties can then be used by a path

planner to determine the traversability of stairs in relation to the specific robotic platform.

In this paper, we present a strategy for ascending-stair detection,approach, and traversal for an

autonomous tracked vehicle. Specifically, we focus on the minimum-sensing scenario in which only a

monocular camera are available on the robot. Our algorithm is divided in four phases (P1) Far-approach:

the robot determine possible ascending-stair locations and starts to navigate towards one of them.

(P2) Near-approach: the robot verifies alignment using analysis of the leading stair edge. (P3) Stair

alignment: the robot aligns itself to the ascending stairs. (P4) Stair traversal: the robot steers along a safe

trajectory up ascending staircase

II. LITERATURE SURVEY

The research conducted so far in development of software for mobile robot using image processing

and artificial intelligence are discussed in this section. The set of challenges outlined above span

several domains of research and the majority of relevant work will be reviewed in this section. In this

section, basic concept are discussed for better understanding of the project. The literature survey

regarding which methodology should be applied was taken considering the following existing algorithms

and models.

Mike Fair and David P. Miller[1] presented a stair climbing robot built around the chassis of a

Quest Access wheelchair. They replaced the control electronics and several sensors including a laser

scanner and a set of Sharp range sensors have been added. The original wheelchair hardware was

capable of negotiating a staircase, but it required the human operator to detect the staircase, put

the chair into stair-climbing mode, align the chair with the staircase, and then manually control the

rate of ascent or decent. The sequencing of the chair tilt and easy downs (the hydraulic actuators

which gently allowed the chair to transition from level ground to inclines ) was done automatically.

E.Mihankhah, A.Kalantari [2] presented a Autonomous Stair- case Detection and Stair Climbing for a

Tracked Mobile Robot using Fuzzy Controller. Theoretical analysis and implementation of

autonomous staircase detection and stair climbing algorithms on a novel rescue mobile robot are

presented in this paper. The main goals are to find the staircase during navigation and to implement a

fast, safe and smooth autonomous stair climbing algorithm. Silver is used here as the experimental

platform. This tracked mobile robot is a teleoperative rescue mobile robot with great capabilities

in climbing obstacles in destructed areas. Its performance has been demonstrated in rescue robot

league of international RoboCup competitions. A fuzzy controller is applied to direct the robot

during stair climbing. Controller inputs are generated by processing the range data from two

LASER range finders which scan the environment one horizontally and the other vertically

Daniel M.Helmick Stergios and Michael C Henry [3]designed for Small,tracked mobile robots for

general urban mobility have been developed for the purpose of reconnaissance and/or search and

rescue missions in buildings and cities. Autonomous stair climbing is a significant capability required for

many of these missions. In their paper they presented the design and implementation of a new set of


Volume 2, Issue 1

74 | © 2015, IJAFRC All Rights Reserved

estimation and control algorithms that increase the speed and effectiveness of stair climbing. They have

developed a Kalman filter that fuses visual/laser data with inertial measurements and provides

attitude estimates of improved accuracy at a high rate,and(ii) a physics based controller that

minimizes the heading error and maximizes the velocity of the vehicle during stair climbing.

Experimental results using a tracked vehicle validate the improv

estimation scheme over previous approaches

Sonda Ammar Bouhamed and Imene Khanfir Kallel, Dorra Sellami Masmoudi

staircase detection using ultrasonic sensor in (IJACSA) International Journal of Advanced Computer

Science and Applications, Vol. 4, No. 6 on 2013 . A new device is then proposed to enable them

to see the world with their ears. Considering not only system requirements but also technology

cost, they used, for the conception of our tool, ultrasonic sensors and one monocular camera to enable

user being aware of the presence and nature of

are involved in using only one ultrasonic sensor to detect staircases in electronic cane. The system

was evaluated on a set of ultrasonic signal where stair

multiclass SVM approach, recognition rates of 82.4.

III. PROPOSED SYSTEM

PARAMETERS FOR MOBILE ROBOT

1) Raw image captured from the camera.

2) Default Focal length of the camera.

3) Standard value of stair riser.

1.1 Stair Edge Detection:

In this section we shall present the state of the art for edge stair detection algorithms used by the

corresponding robots to climb stairs. In our project we use a robot which has a camera mounted

on it. A single molecular camera is used to capture the surrounding . A video is a sequence of image and

image is a binary representation of visual

where each element represents some value of pixel. Color digital images

are made of combinations of primary colors.

Every channel is represented in form of matrix. Element in each channel is used to store the intensity of

red , blue or green .Combination of these intensity provide the colour for every pixel. A gray scale

image has only 1 channel which is used to store the colour intensity between black or white

Using image processing we are analyzi

national Journal of Advance Foundation and Research in Computer (IJAFRC)

Volume 2, Issue 12, December - 2015. ISSN 2348 – 4853, Impact Factor

© 2015, IJAFRC All Rights Reserved



of improved accuracy at a high rate,and(ii) a physics based controller that


Experimental results using a tracked vehicle validate the improved performance of this control and

estimation scheme over previous approaches

Sonda Ammar Bouhamed and Imene Khanfir Kallel, Dorra Sellami Masmoudi [4] proposed a paper on



the world with their ears. Considering not only system requirements but also technology


user being aware of the presence and nature of potential encountered obstacles. In this paper, they


was evaluated on a set of ultrasonic signal where stair-cases occur with different sh

multiclass SVM approach, recognition rates of 82.4.

PARAMETERS FOR MOBILE ROBOT

camera.

camera.

Figure 1: Architectural Diagram



mera is used to capture the surrounding . A video is a sequence of image and

image is a binary representation of visual information. Image can be represented in the form of a matrix ,

where each element represents some value of pixel. Color digital images are made of pixels, and pixels

are made of combinations of primary colors. Color image consists 3-channel of red, blue and green.


ed , blue or green .Combination of these intensity provide the colour for every pixel. A gray scale

image has only 1 channel which is used to store the colour intensity between black or white

analyzing and manipulating the image to get the desired output. The RGB


4853, Impact Factor – 1.317

© 2015, IJAFRC All Rights Reserved www.ijfarc.org



of improved accuracy at a high rate,and(ii) a physics based controller that


ed performance of this control and

[4] proposed a paper on



the world with their ears. Considering not only system requirements but also technology


potential encountered obstacles. In this paper, they


cases occur with different shapes. Using a



mera is used to capture the surrounding . A video is a sequence of image and

Image can be represented in the form of a matrix ,

are made of pixels, and pixels

channel of red, blue and green.


ed , blue or green .Combination of these intensity provide the colour for every pixel. A gray scale

image has only 1 channel which is used to store the colour intensity between black or white color.

and manipulating the image to get the desired output. The RGB




image we get from camera , is to be first converted into gray scale image . The gray scale image is

then filtered so as to remove the noise. A filtering method is used to remove the extra pixels and to fill the

holes in a continuous curve. Then the filtered image is used to detect edges. A edge detection technique

Open CV is used to detect edges. The main purpose of the edge detection is to simplify the image

in order to minimize the data to be processed. There are three methods which can be used to detect

edges. First is Soble second is Laplacian and third is canny . In this project we are using canny edge .

Laplacian is more prone to noise and Sobel edge detection function requires the destination image to

have a larger depth than the source one. So we must make edge frame the 16-bit intermediate in this

case to store the result of the Sobel operation. A 16-bit signed image cannot be shown with the cv

Show Image function properly. So we have to convert it back to 8-bit unsigned format. To save memory

space and code lines, Canny edge detection is used. Using contour detection, the co-ordinates of boundary

are stored in a variable.

Processing on blurred image -If the image is blurred , then it can be rectified using the un-sharp masking

technique .In this technique a slightly blurred image is compared with the original image to detect the

presence of edges and the contrast is then slightly increase along the edges .

1.2 Stair Detection Using Neural Network:

In our method, we are using neural network to make a simple decision to check whether there is stair

or not. Back propagation is an algorithm in neural network that can be used to train a neural

network[5]. Training a neural network is the process of finding a set of weights and bias values so that,

for a given set of inputs, the outputs produced by the neural network are very close to some known

target values. Gradients are values that reflect the difference between a neural networks computed

output values and the desired target values. As it turns out, gradients use the calculus derivative of the

associated activation function. The gradients of the output nodes must be computed before the

gradients of the hidden layer nodes, or in other words, in the opposite direction of the feed forward

mechanism. Supervised learning which that each output unit is told what its desired response to

input signals is used in our system.

1.3 Distance calculation and alignment:

In order to determine the distance we are going to utilize triangle similarity [6]. In this we have stairs

with a known height of one step H. We then place this object some distance D from our camera. We take a

picture of our object using our camera and then measure the apparent height in pixels P. Let’s say the

focal length is F. D = (H x F) / P .Using contour detection we can calculate the x co-ordinate of two end

edges of a same horizontal boundary. If the values are different the robot is not aligned otherwise it is

aligned.

IV. PROJECT IDEA

In this project we have made a flow chart which will help us analyze and proceed according to the

processes or steps specified in the flow chart.


Volume 2, Issue 1

76 | © 2015, IJAFRC All Rights Reserved

The camera mounted on the ROV will act as the input for the program.

in operational the program runs as it starts capturing stream of images that is the video from the camera

mounted over the robot. The images

methods to outline the of stairs. Image filtering is applied and stairs are detected using contour detection

methods. Now as proposed in previous section, we will detect an approximate distanc

the robot after which robot will be able to climb up the stairs autonomously.

This project will be executed on the ROV DAKSH from the robotics group of the DEFENCE RESEARCH

AND DEVELOPMENT ORGANISATION'S R&DE(E) Pune

V. RELEVANT MATHEMATICS ASSOCIATED WITH THE PROJECT.

System Description:

Input: Image,Actual height of stair , height of stair in pixel , Focal length of camera.

Output: Stair recognition , movement of wheels ,distance between camera and stair.

Identify data structures, classes, divide and conquer strategies to exploit distributed/parallel/concurrent

processing, constraints.

Related Function:

$f_1$ = database creation for storing the images of types of stairs.

$f_2$ = Camera capture of stairs while robot is mobile

$f_3$ = image processing techniques on the frame captured

$f_4$ = mapping the processed image with the database to detect the type of stair it climbing on

Mathematical formula

Distance = (actual height of one step * Focal length)/pixel height of one step.

Success Conditions: stair detected accu

\item Failure Conditions: image inaccuracy , defect in camera.


Volume 2, Issue 12, December - 2015. ISSN 2348 – 4853, Impact Factor

© 2015, IJAFRC All Rights Reserved

Figure 2: Flow Chart

The camera mounted on the ROV will act as the input for the program. When the robot starts moving or


mounted over the robot. The images captured from this molecular camera then applies edge detection


methods. Now as proposed in previous section, we will detect an approximate distanc

the robot after which robot will be able to climb up the stairs autonomously.


AND DEVELOPMENT ORGANISATION'S R&DE(E) Pune

RELEVANT MATHEMATICS ASSOCIATED WITH THE PROJECT.



ures, classes, divide and conquer strategies to exploit distributed/parallel/concurrent

$f_1$ = database creation for storing the images of types of stairs.

$f_2$ = Camera capture of stairs while robot is mobile

$f_3$ = image processing techniques on the frame captured


Distance = (actual height of one step * Focal length)/pixel height of one step.

accurately, distance between the robot and camera

item Failure Conditions: image inaccuracy , defect in camera.


4853, Impact Factor – 1.317

© 2015, IJAFRC All Rights Reserved www.ijfarc.org

When the robot starts moving or


captured from this molecular camera then applies edge detection


methods. Now as proposed in previous section, we will detect an approximate distance of the stair from




ures, classes, divide and conquer strategies to exploit distributed/parallel/concurrent


distance between the robot and camera




VI. CONCLUSION

So far we have explored a big idea to automate a part of the manually controlled ROV DAKSH from the

Defense Research and Development Organization’s Research and Development Establishment (E), Pune.

Our work will help the robot’s to detect and climb stairs on their own. This will help in further more

exploiting the areas of automation and help the country in different needs. This project will help us in

understanding the concepts of Artificial intelligence, Image processing techniques and Neural networks.

VII. FUTURE WORK

Our project opens a lot of doors for further improvising the idea of automation amongst robots. Currently

working on ROV Daksh, we will be working on all other manually controlled robots of the DRDO. This will

help in all other robots of the organization to be somehow automated. Use of Neural network by us can

encourage in making the whole robot automated and hence save lives in the war-zone, terrorist attacks,

diffusing the bombs and other natural or man-made calamities.

VIII. APPLICATIONS AREA FOR PROJECT

Applications of the project are :

[1] Stair climbing autonomously

[2] In war zones

[3] In operations at the urban areas where buildings are high and requires robot to climb stairs

autonomously.

IX. AACKNOWLEDGMENTS

Our sincere thanks to all those who are guiding us in this

project. Our sincere thanks to our guide and mentor :-

� Prof.Uma Nagaraj, HOD, COMP, MIT-AOE, Pune

Our external Guides :

� Mr. Alok Mukherjee, Scientist, RDE(E), Pune

� Mr. Altaf Mirza Baig, Scientist, RDE(E), Pune

� Mr. Debjyoti Das, hi-tech robotic systemz ltd, Pune

X. REFERENCES

[1] S. Owald, J.-S. Gutmann, A. Hornung, and M. Bennewitz,“ From 3D point clouds to climbing

stairs: A comparison of plane segmentation approaches for humanoids ”.In Proc. of the

IEEE-RAS Int. Conf. On Humanoid Robots (Humanoids), 2011.

[2] Delmerico, J.A., Baran, D., David, P., Ryde, J., Corso, J.J. “ Ascending stairway modeling from dense

depth imagery for traversability analysis”. In: International Conference on Robotics and

Automation (ICRA). pp. 2283 2290 (2013).

[3] A. M. Johnson, M. T. Hale, G. C. Haynes, and D. E. Koditschek, “Autonomous legged hill and

stairwell ascent ” ,In IEEE International Workshop on Safety, Security, Rescue Robotics

(SSRR),pp. 134 142.November 2011




[4] J. Delmerico, J. Corso, D. Baran, P. David, and J. Ryde, “ Toward autonomous multi-floor

exploration: Ascending stairway localization and modeling ” , U.S. Army Research Laboratory,

Tech. Rep. ARL-TR, 2012

[5] R. B. Rusu and S. Cousins, “ 3D is here: Point Cloud Library (PCL)”, In IEEE International

Conference on Robotics and Automation (ICRA), Shanghai, China, May 9-13 2011

[6] Tung-Sing Leung , Medioni G. ,“ Real-time staircase detection from a wearable stereo system

” .In: Pattern Recognition (ICPR), 2012 21st International Conference on Nov. 2012,pp 3770 -

3773[7]

[7] Hernandez, D.C., Jo, K.H. “ Stairway tracking based on automatic target selection using

directional filters ” . In: Frontiers of Computer Vision (FCV). pp. 16 (2011)

[8] Cloix, S., Weiss, V., Bologna, G., Pun, T., Hasler, D. “ Obstacle and planar object detection

using sparse 3D information for a smart walker”. In: 9th International Conference on Computer

Vision Theory and Applications. vol. 2, pp. 305 312. Lisbon, Portugal (Jan 2014)

[9] Tseng, C.K., Li, I., Chien, Y.H., Chen, M.C., Wang,W.Y., “ Autonomous stair detection and climbing

systems for a tracked robot ”. In: System Science and Engineering (ICSSE), 2013 International

Conference on. pp. 201 204. IEEE (2013).

Documents

Development Of Control Software For Mobile Robot Using Image Processing and Artificial Intelligence