Undergraduate Research Opportunity Program(UROP) Project Report

Vision-based Autonomous Control and Navigation of a UAV

By

Wang Chao

Department of Computer Science

School of Computing

National University of Singapore

Nov 2014

Project No: U148150Advisors: Dr. Somchaya Liemhetcharat and Asst. Prof. Kian Hsiang LowDeliverables:

Report: 1 Volume

Abstract

In this report, we present the design and implementation of a real-time, vision-based naviga-tion system for an Unmanned Aerial Vehicle (UAV). The navigation system is integrated withautonomous flight control system, visual servoing, and visual detection of marker and object(a hula hoop). Based on the autonomous control system and visual servoing, the navigation isable to be done from an arbitrary initial position and orientation. Finally, the report concludesby presenting the flight trial performance, conclusions, and future work.

Keywords:Unmanned Aerial Vehicle, Autonomous control, Vision-based navigation

Implementation Software and Hardware:Parrot AR.Drone 2.0

Acknowledgement

This project is made possible through the help and support from everyone, including: my

advisors, parents and friends. Especially, please allow me to dedicate my acknowledgment of

gratitude toward the following significant advisors:

First, I would like to thank my advisor Prof. Low Kian Hsiang. Without his agreement,

this project will not be made possible. He kindly gives me valuable advice and makes me keep

consistent in this whole project.

Second, I would like to thank my co-advisor Dr. Somchaya Liemhetcharat for his most

support and encouragement. Without his scientific guidance and contributions, I would not be

able to finish this project. Moreover, he kindly reads my report and o↵ered invaluable detailed

advice.

Finally, I sincerely thank to my parents and friends, who provide moral support. Without

all of them the product of this project would not be possible.

List of Figures

3.1 Parrot AR.Drone 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4.1 Drone manoeuvres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.2 Performance of the Proportional Controller . . . . . . . . . . . . . . . . . . . . . 94.3 Performance of the PID Controller . . . . . . . . . . . . . . . . . . . . . . . . . . 114.4 UAV’s Control Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.1 Visual Servoing Marker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.2 Blobs and Bounding boxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.3 Calculation of the relative world coordinate with marker . . . . . . . . . . . . . . 155.4 The state transition diagram for marker tracking . . . . . . . . . . . . . . . . . . 16

6.1 UAV is aligned with the hula hoop . . . . . . . . . . . . . . . . . . . . . . . . . . 176.2 Fish-eye lens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.3 FOV increased after using fish-eye lens . . . . . . . . . . . . . . . . . . . . . . . . 186.4 Chessboard for calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.5 Distorted Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.6 Undistorted Image Using OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . . 206.8 Image processing results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.9 The state transition diagram for hula hoop detection and flying through . . . . . 24

7.1 Orientation Indication Marker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277.2 Orientation Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277.3 UAV and ground robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

8.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298.2 The state transition diagram for the final experiment . . . . . . . . . . . . . . . . 31

iv

List of Tables

7.1 Hoop angle detected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

v

Table of Contents

Title i

Abstract ii

Acknowledgement iii

List of Figures iv

List of Tables v

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Report Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Related Work 4

3 UAV Platform 6

4 UAV Autonomous Control 84.1 Proportional Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2 PID Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5 Visual Servoing Using Vertical Camera 135.1 Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.2 Marker Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.3 Marker Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.4 UAV Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6 Hula hoop Detection and Flying Through 176.1 Usage of Fish-eye Lens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.2 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.3 Image Undistortion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.4 Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.5 Position Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226.6 UAV Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

7 Hula hoop Detection While Not Aligned 257.1 Ellipse Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.2 Ground Robot Assistance Using Marker . . . . . . . . . . . . . . . . . . . . . . . 26

vi

8 Experiment Results 298.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298.2 UAV Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

9 Conclusion And Future Work 32

References 33

vii

Chapter 1

Introduction

In recent years, unmanned aerial vehicle (UAV) has become a major focus of robotics researches,

since it can extend human’s capabilities in a variety of areas especially for military application

like search-and-rescue and surveillance, or civilian application like transportation. Among all

kinds of UAVs, quadcopter is more stable and easier to control. For example, it is easier to

maintain a hovering position with a quadcopter than with a helicopter or airplane. We choose

quadcopter as a platform to design and implement an autonomous flight control system and

vision-based navigation system.

1.1 Motivation

Typical UAVs are controlled by humans and experiences are needed but still have low precision.

The missions of autonomous UAVs require flying at low speed to follow a path or to track an

object of interest and take a series of actions.

Imagine a scenario that there are some tunnels. Inside the tunnels are full of unknown

dangers. Since the ground environment inside the tunnels are unstructured, the best choice

would be dispatching a UAV to explore it. The UAV takes o↵ from a ground robot and starts

tracking it. Then ground robot will navigate the UAV to the entrance of the tunnel. Upon

arrival, the UAV will scan for the entrance and fly into the tunnel after confirmation of the

entrance using its on-board vision algorithm. Inside the tunnel, wireless signals are unreliable

1

and there may not be a consistent signal for a human operator to control the UAV. Therefore,

the UAV will perform autonomous flight using vision-based navigation system and complete a

series of missions. Finally, it flies out from the exit of the tunnel, then the ground robot will

navigate it to another tunnel and so on.

As you can see from the scenario, accurate autonomous flight control and vision-based

navigation are necessary to perform such tasks.

1.2 Challenge

UAVs are highly sensitive to control inputs. For example, the mechanical structurer of quad-

copter comprises four rotors attached to the four ends. To maintain a hover, the speed of each

rotor need to be same. Even a small disturbance can cause drifting and rotations. Feedback

is critical to stable the autonomous flight. As a result, a closed loop control system is needed

with low latency and precise prediction of future outputs.

UAVs can move quickly. Small and mid-sized quadcopters can accelerate in the range of 0.5

- 0.7g and can exhibit 40-60� per second angular velocity under normal operating conditions

(Amidi, 1996). To keep up with the quadcopters high degree of maneuverability, an on-board

vision system must be able to sample and process camera images at high frequency. On-board

image processing must be performed at frame rate (30 Hz) or higher for e↵ective vision-based

object tracking.

UAVs are always moving which can cause motion blur and sensor data is noisy. To make

the UAVs stable enough, some techniques must be used to filter noisy data and get usable data.

1.3 Contribution

Our UAV flight control system is fully autonomous. It is accurate and stable with high flexibility

as well. Our vision-based navigation system is fully on-board with fast and real-time image

processing (30 FPS). Furthermore, our system works both indoor and outdoor without relying

on any other external positioning devices like GPS and motion capture systems like VICON.

2

Moreover, a wide-angle lens is mounted on the front camera of the UAV which is an innovative

attempt. It successfully improves the FOV of our vision system and make it more reliable and

practical. All of this make our system promising and able to be developed further.

1.4 Report Organization

This report is divided into nine chapters. Chapter 2 presents related work that has been

done in autonomous control systems and vision-based navigation systems. Some issues are also

addressed. Chapter 3 provides some background information of the UAV platform we use.

Chapter 4 presents the design and implementation of the UAV autonomous control system.

Chapter 5 presents a visual servoing system using image processing and autonomous control.

Chapter 6 presents our navigation system and algorithm that allows the UAV to autonomously

detect a hula hoop and fly through it.

Chapter 7 addresses detection issues while the UAV and the hula hoop are not aligned. A

solution is provided as well.

Chapter 8 presents the final flight experiments using the vision-based autonomous control

and navigation system. Results are discussed and evaluated.

Chapter 9 presents conclusions and future work. Future work includes multi-UAV coordi-

nation and motion planning.

3

Chapter 2

Related Work

In recent years, unmanned aerial vehicle (UAV) and micro aerial vehicle (MAV) have been an

active area of research. They have been used for military applications but also are useful for

many civilian applications such as terrain and utilities inspection, disaster monitoring, environ-

mental surveillance, search and rescue, and tra�c surveillance. However, most of them are still

semi-autonomous and guided by human who is not on board. There are lots of challenges to

turn an UAV into fully autonomous control.

The control of UAV during autonomous flights relies on knowledge of variables like position,

velocity and orientation, which can be partly calculated using information provided by on-board

inertial sensors. However, the drift of inertial sensors leads to errors during time-discrete integra-

tion, making a steadily accurate estimation of the absolute pose nearly impossible. Therefore,

an absolute reference is needed.

An early autonomous navigation system for a model-scale helicopter (the Hummingbird)

was developed at Stanford University (Conway, 1995). The unique feature of this system was

the sole use of GPS as the navigation sensor replacing the Inertial Measurement Unit, which is

conventionally favored as the primary navigation sensor. However, indoors, the GPS will not be

available, where a visual tracking system could be applied. In (Gurdan, Stumpf, Achtelik, Doth,

Hirzinger, & Rus, 2007) a position controller for a quadcopter was implemented using an optical

tracking system by VICON in the Holodeck lab at MIT. This intelligent space is nevertheless

very expensive and not transportable. A notable vision-based technique used in autonomous

4

helicopter control, the visual odometer (Amidi, 1996), provides accurate navigational informa-

tion such as position and velocity which is then combined with inertial measurements. However,

outdoors, it still needs GPS for positioning.

While several techniques have been applied for vision-based control of UAVs, none of them

have shown an fast, fully on-board vision-based control and positioning system. Our approach,

presented here, di↵ers from prior approaches in three ways. First, our vision system is real-time

with low latency image processing (30 FPS). Hence, the UAV positioning can be done very fast

and accurately. Second, the front camera vision system has a FOV of 178� which can easily

detect object in a large range. Third, our autonomous control system works both indoors and

outdoors without relying on any other external positioning devices like GPS.

5

Chapter 3

UAV Platform

Figure 3.1: Parrot AR.Drone 2.0

Parrot AR.Drone is a radio controlled flying quadcopter built by the French company Parrot.

The drone is designed to be controlled by mobile or tablet operating systems such as the

supported iOS or Android within their respective apps or the uno�cial software available for

Windows Phone, Samsung BADA and Symbian devices.

Version 2.0 is the successor to the original drone, the AR.Drone 2.0 was unveiled at CES

Las Vegas 2012. A lot of improvements was made to its functionality. Technical specifications

are as follows:

• 1GHz 32 bit ARM Cortex A8 processor

6

• Linux 2.6.32

• 1GB DDR2 RAM at 200MHz

• Wi-Fi b g n

• 3 axis gyroscope 2000�/second precision

• 3 axis accelerometer +-50mg precision

• 3 axis magnetometer 6�/ precision

• Pressure sensor +/- 10 Pa precision

• Ultrasound sensors for ground altitude measurement

• Front camera: 720p sensor with 93�/ lens, recording up to 30 FPS

• Vertical camera: QVGA sensor with 64�/ lens, recording up to 60FPS

Thanks to the AR.Drone 2.0 on-board sensors, take-o↵, hovering, trimming and landing are

completely automatic. Some other basic control and automatic stabilization features are also

integrated. The AR.Drone 2.0 Library is also provided as an open-source library with high

level APIs to access the drone. However, the SDK provided is designed to run on an external

computer, but we adapted it to run on the drone. Our application is developed based on the

SDK and running on the drone as well.

7

Chapter 4

UAV Autonomous Control

Basically, manoeuvres are obtained by changing pitch, roll, and yaw angles of the AR.Drone

2.0.

Figure 4.1: Drone manoeuvres

The APIs of controlling pitch, roll, and yaw angles are provided by the SDK. Besides, some

other basic manoeuvres’ APIs are also provided including take-o↵, hovering, trimming, and

landing. However, these controls are too low-level to achieve autonomous flight. For example, if

the UAV finds a target ahead at one meter, the UAV need to move forward one meter precisely.

However, adjusting pitch angle directly is not su�cient to obtain the required performance.

As a result, some high-level controllers are needed for advanced manoeuvres. We consider two

frequently used controllers, namely Proportional Controller and PID Controller.

8

4.1 Proportional Controller

Proportional Controller is the most intuitive control method which is to adjust pitch angle

directly proportional to the distance to target. For example, if the UAV is far away from the

target, it can tilt a larger pitch angle so that it can move forward with large acceleration. While

it is approaching the target, the distance becomes smaller and smaller. At this time, it should

decrease accelerating rate. Hence, it needs tilt a smaller pitch angle. When it reach the target,

the pitch angle should be zero which means no acceleration. The experiment result is shown in

Figure 4.2.

0 2 4 6 8 10

0

20

40

60

80

Time(s)

Position(cm)

Move forward 50cm

target

UAV

Figure 4.2: Performance of the Proportional Controller

The manoeuvre being tested is to move forward 50 centimeters. The world coordinate was

updated using odometry from IMU sensor data. The experiment result shows that at first, the

UAV was approaching the target quickly. However, after it reached 50cm it still moving forward

but slow down. After it stopped, it started moving backward to 50cm but overshoot again. The

position of UAV is keep fluctuating around 50cm although the gap is getting smaller. It can be

seen that to make itself stop at 50cm will take a long time. Therefore, a more precise controller

is needed.

9

4.2 PID Controller

A proportional-integral-derivative controller (PID Controller) is a closed loop control based on

feedback mechanism. A PID controller calculates an error value as the di↵erence between a

measured process variable and a desired setpoint. The controller attempts to minimize the

error by adjusting the process through use of a manipulated variable.

u(t) = MV (t) = K

p

e(t) +K

i

Zt

0

e(⌧)d⌧ +K

d

d

dt

e(t) (4.1)

where Kp

is Proportional gain, a tuning parameter. Ki

is Integral gain, a tuning parameter.

K

d

is Derivative gain, a tuning parameter. e is the di↵erence between desired value and current

value. t is Time or instantaneous time (the present). ⌧ is Variable of integration; takes on

values from time 0 to the present t.

The proportional term produces an output value that is proportional to the current error

value. The proportional response can be adjusted by multiplying the error by a constant K

p

,

called the proportional gain constant. The Proportional Controller introduced in Chapter 4.1

is a PID controller with only proportional term. A high proportional gain results in a large

change in the output for a given change in the error. If the proportional gain is too high,

the system can become unstable. In contrast, a small gain results in a small output response

to a large input error, and a less responsive or less sensitive controller. Tuning theory and

industrial practice indicate that the proportional term should contribute the bulk of the output

change (Hjalmarsson, Gevers, Gunnarsson, & Lequin, 1998). However, a controller with only

proportional term will result in high instability with oscillation especially for a system with

inertia. As a consequence, derivative term needs to be considered.

The derivative of the process error is calculated by determining the change of the error over

time and multiplying this rate of change by the derivative gain K

d

. Derivative action can be

thought of as making changes slower as one changes too fast. Derivative control will slow the

response of the circuit. In doing so, it is able to partially compensate for the overshoot as well as

damp out any oscillations caused by integral and proportional control. Furthermore derivative

action predicts system behavior and thus improves settling time and stability of the system.

10

Now, the problem is how to define the process variable (PV) and the setpoint (SP) in our

system. Obviously, the manipulated variable (MV) is the pitch angle which is the only interface

provided. Basically, the change of pitch angle results in the change of the acceleration of UAV

proportionally rather than the speed. For the Proportional Controller in last chapter, even the

pitch angle is set to be zero, the UAV still has velocity caused by inertia. Thus, the PV and SP

should be in term of velocity. The way we define SP is proportional to the distance to target.

The larger the distance, the higher the SP (desired velocity). While the UAV reach the target,

the SP should be zero, so that it can stop right there without oscillation. The proportional gain

and derivative gain are tuned and optimized through hundreds of experiments. The experiment

result is shown in Figure 4.3.

0 2 4 6 8

0

50

100

150

200

250

300

Time(s)

Position(cm)

Move forward 300cm

target

UAV

Figure 4.3: Performance of the PID Controller

The improvements of PID controller is significant compared with Proportional Controller.

The UAV keeps approaching the setpoint and the overshoot and oscillation are reduced.

For moving left and right, we apply the same PID controller to adjust roll angle. For

turning and climbing, desired yaw and altitude are obtained by setting the AR.Drone’s yaw

and altitude changing rates. Therefore, P controllers were enough for these two degrees of

11

freedom. As you can see in the figure below, the control loop is fully autonomous now. With

this precise autonomous flight control, the UAV has gained the ability to perform some more

complicated tasks and higher level autonomous behaviors.

Figure 4.4: UAV’s Control Loop

12

Chapter 5

Visual Servoing Using Vertical

Camera

Visual Servoing is also known as vision-based robot control. It combines image processing and

control techniques, in such a way that the visual information is used within the control loop.

Basically, the task can be described as: given a target, we want to track a visual point feature

(centroid or a characteristic point on the target) in order to perform continuous monitoring

by keeping the UAV in flight above it. In other words, to control the position of the UAV

(hovering) autonomously, while keeping the target in the center of image plane. As the on-

board computational power of AR.Drone 2.0 is limited, the object must be easy to detect but

unique as well. What we use is a simple marker consists of two concentric circles with white

and black colors only (shown in Figure 5.1).

5.1 Image Processing

The first step is thresholding. Thresholding is to convert the original image to a dicrete color

image we want. For the marker which consists of only black and white color, the thresholded

image consists of only black and white.

The second step is segmentation. Segmentation is to find the contiguous regions in the

thresholded image with same color. Each of these region will be considered as a blob. The

13

Figure 5.1: Visual Servoing Marker

marker consists three blobs b

1

, b2

, and b

3

. Each blob will be detected with a bounding box

(Figure 5.2).

Figure 5.2: Blobs and Bounding boxes

5.2 Marker Detection

The coordinates of bounding boxes of each blobs on image will be calculated. The uniqueness

of the marker is that b1

is inside b2

, and b

2

is inside b3

. Additionally, they should have the same

center. The vision algorithm scans for the marker and compute the coordinate of marker’s

center in the image.

14

5.3 Marker Tracking

Figure 5.3: Calculation of the relative world coordinate with marker

Figure 7.1 explains how the relative position of the marker is calculated. Since the altitude

can be obtained from the UAV with ultrasound sensor, and the FOV (Field of View) of the

vertical camera is also given. Therefore the relative distance from the marker to the UAV in

real world can be easily calculated. Then the target position will be sent to control loop and

command will be sent to the UAV. The updating speed is around 30 FPS which is good enough

for tracking a low speed target.

5.4 UAV Behaviors

The overall tracking strategy is best described as a simple finite state machine (Figure 5.4) with

two states: search and track. Initially the UAV is in the search mode. The marker detection

algorithm scans for the marker. Once the marker is detected, the system transitions to the

track mode. In this mode, the marker detection algorithm sends the relative distance between

marker and UAV in the world coordinate. The marker tracking algorithm will send command

to UAV to make it track the marker. Similarly, if the target is lost during the track mode, the

15

system transitions back to the search mode.

Take o↵

Search

marker

Found

marker?

Track

marker

Lost

marker?

no

yes

no

yes

Figure 5.4: The state transition diagram for marker tracking

16

Chapter 6

Hula hoop Detection and Flying

Through

In addition to marker tracking, we want to make the UAV do some more interesting things

using the front camera and autonomous control techniques. Our aim is to make the UAV detect

a hula hoop and fly through the hula hoop autonomously and precisely. In this chapter, we

assume that the UAV and the hula hoop are aligned with each other (Figure 6.1). We will talk

about our solution while they are not aligned in the next chapter.

Figure 6.1: UAV is aligned with the hula hoop

17

6.1 Usage of Fish-eye Lens

Since the hula hoop is a large hoop with thin edge, the whole hoop must be visible in order to

detect it. However, the UAV’s front camera has only 90� FOV, which is too narrow and the

UAV will only be able to see a portion of the hoop while it gets close to the hoop. To increase

the FOV of the UAV, we mount a fish-eye lens on the original front camera (Figure 6.2).

Figure 6.2: Fish-eye lens Figure 6.3: FOV increased after using fish-eye lens

With the help of the fish-eye lens, the FOV of the UAV was increased to 178� (Figure 6.3).

The next task is to calibrate the camera and get the intrinsic parameters like focal length so

that the 3D position of object in the real world can be estimated.

6.2 Camera Calibration

For the camera calibration, we use OpenCV and classical black-white chessboard.

Figure 6.4: Chessboard for calibration

18

The aim of camera calibration is to find the camera matrix which is referred as

2

666664

f

x

0 c

x

0 f

y

c

y

0 0 1

3

777775(6.1)

The unknown parameters are f

x

and f

y

(camera focal length) and (cx

, c

y

) which are the

optical centers expressed in pixels coordinates. Basically, we need to take snapshots of the

chessboard with the front camera of UAV and let OpenCV solve the unknown parameters. To

get a good results, we need at least 10 good snapshots of the chessboard in di↵erent positions

from di↵erent angles.

6.3 Image Undistortion

However, using fish-eye lens causes a serious problem which is image distortion (Figure 6.5).

Figure 6.5: Distorted Image

From Figure 6.5, it can be seen that the chessboard in image was distorted. Especially

the distortion on the edge is more severe and apparent. Such significant distorted image will

result in wrong detection of object and estimation of position. Hence, an image undistortion is

necessary.

We use OpenCV to do undistortion and remapping. The formula given by OpenCV is as

follows:

x

corrected

= x(1 + k

1

r

2 + k

2

r

4 + k

3

r

6)

y

corrected

= y(1 + k

1

r

2 + k

2

r

4 + k

3

r

6)

(6.2)

19

So for any pixel point at (x, y) coordinates in the input image, its coordinates on the cor-

rected image will be (xcorrected

, y

corrected

) where k

1

, k

2

, k

3

are distortion coe�cients. However,

the undistortion seems not work well.

Figure 6.6: Undistorted Image Using OpenCV

From the image, it can bee seen that only the chessboard part was undistorted correctly.

Other part especially the edge is still distorted severely. After trying to tune some setup

parameters, we are still not able to get a good result. Then we do some researches regarding

to this problem. We find that the problem we have is probably because that the undistortion

model of OpenCV is based on pinhole camera which is di↵erent from cameras with wide-angle

lens such as fish-eye lens. Hence, in order to get a good undistortion result, we have to find

another way to do undistortion.

After doing a lot of experiments, OCamCalib (Omnidirectional Camera Calibration Toolbox

for Matlab) gives us a good result (Scaramuzza, Martinelli, & Siegwart, 2006a). This toolbox

is designed for catastrophic and fish-eye cameras up to 195 degrees (Scaramuzza, Martinelli, &

Siegwart, 2006b). We use OCamCalib toolbox to calibrate the fish-eye lens again and get

a mapping matrix which is able to correct distorted pixels with a mapping function (Rufli,

Scaramuzza, & Siegwart, 2008).

(xcorrected

, y

corrected

) = mapping(x, y) (6.3)

where (x, y) is the coordinates on distorted images, and (xcorrected

, y

corrected

) is the coordi-

nates mapped to the undistorted image.

20

(a) Distorted Image (b) Undistorted Image

(c) Distorted Image (d) Undistorted Image

Figure 6.7a is one of the chessboard image to do undistortion. Compared with Figure 6.7b,

the undistortion is acceptable and much better then the one using OpenCV. Figure 6.7c is the

image of hoop viewing from the UAV before undistortion. Compared with Figure 6.7d, the line

on the ground and the edge of white board become straight as expected. The improvement

using OCamCalib is significant. Although there is still distortion on the edge, but this is good

enough for us to proceed to the next step.

6.4 Image Processing

The goal of image processing is to locate and extract the hula hoop. Figure 7.3b shows an front

camera view of hula hoop.

(a) Image from Front Camera (b) Segmented Image

Figure 6.8: Image processing results

21

First, the original image is down-sampled to 180p. The reason to do down-sampling is that

object detection is real-time and the UAV always keeps moving. Thus the requirement of image

processing speed is high which must be at least 30FPS. The image we obtained from the camera

is 720p, its computational cost is too high to run on-board. Hence we reduce the image to 180p

to increase the e↵ectiveness of the algorithm.

Next, thresholding converts the color image to a discrete color image. As you can see from

the figures above, some bright orange color paper are stuck to the hula hoop. This makes the

UAV much more easier to distinguish between hula hoop and other objects present in the image.

At last, segmentation will find the contiguous regions with orange color. Figure 7.3a shows

that after segmentation, the orange hula hoop is extracted from the original image.

6.5 Position Estimation

To identify the center of the hoop in the image, we will do a circle fitting which aims to fit the

equation of a circle to the segmented image. The center coordinates and radius of the hoop in

pixels will be computed. Since the size of the hoop in real world can be measured and focal

length is given in camera matrix, the distance between any pixels in the image in real world is

able to be calculated. Thus the relative position of the hoop center to the UAV in real world

can be estimated.

However, in our experiments, we found that the position data is noisy. The noisy data

might due to the vibration of the UAV. Furthermore, the noisy data could cause more severe

vibration. My solution to this problem is to use a modified moving average filter. A moving

average filter smooths data by replacing each data point with the average of the neighboring

data points defined within the span. The equation is

y

s

(i) =1

N

(y(i) + y(i� 1) + ...+ y(i�N + 1)) (6.4)

where ys

(i) is the smoothed value for the ith data point, N is the number of neighboring data

points on left side of ys

(i) , and N is the span. My modified version is to remove the highest

and lowest part of data within the span before averaging. The highest and lowest part of data

22

have large variance which might caused by vibration. Another concern is that how large should

the span is. If the span is too large, the actual large change of position might be smoothed

wrongly. If the span is too small, the smooth e↵ect will not be obvious. As a result, we choose

the span to be same as the FPS rate which is 30. Since the UAV will not have a sudden change

of position in one second normally, it is reasonable to set the span to 30.

6.6 UAV Behaviors

The overall hoop detection and flying through strategy is described as a finite state machine

(Figure 6.9) with three states: search, approach, and fly through. Initially after taking o↵

the UAV is in the search mode. The image processing and hoop detection algorithm scans for

the hoop. Once the hoop is detected, the system transitions to the approach mode. In this

mode, the hoop detection algorithm sends the coordinates of the marker relative to the UAV’s

position and orientation to the hoop approaching algorithm. The hoop approaching algorithm

will send command to UAV to make it approach to the center of the hoop. If hoop is lost

during the approach mode, the system transitions back to the search mode. If the UAV finishes

approaching and is at the right position to fly through, the hoop approaching algorithm will

send command to UAV so that the UAV will fly through the hoop.

23

Take o↵

Search

hoop

Found

hoop?

Approach

hoop

Lost

hoop?

Ready to

go?

Fly

through

hoop

no

yesyes

no

no

yes

Figure 6.9: The state transition diagram for hula hoop detection and flying through

24

Chapter 7

Hula hoop Detection While Not

Aligned

The assumption we made in the last chapter is that the UAV is facing to the hoop plane

perpendicularly. However, most of the time, the UAV will not be aligned with the hoop. To

achieve the alignment, the UAV need to know the angle to the hoop plane.

7.1 Ellipse Fitting

Since the hula hoop is a circle from the view facing to it. If we view it from other directions,

the circle will become an ellipse. Hence, the angle to the hoop plane could be estimated using

ellipse fitting.

However, the experiment results show that the UAV does not always rotate to the correct

direction. This might because of angle detected is wrong. We collect some angle data while the

UAV is static (Table 7.1).

While the drone’s body is on each side of the hoop, three di↵erent sets of data on di↵erent

position are collected. The head of column is the position of the center of hoop in the image.

The angle detected is supposed to have no relationship with the position of the hoop center.

For each row, the angles detected are supposed to sum up to about 180�. However, only when

the center of the hoop is in the center of the image, the angle detected is correct. Our solution

25

Drone’s body

Center of hoop

in image

Left of hoop Middle of hoop Right of hoop

Left

31±2 30±2 27±2

27±2 21±2 16±2

17±2 13±2 11±2

Right

153±2 154±1 141±2

159±2 164±1 150±2

170±2 168±1 166±2

Center

65±2 90±2 109±2

31±2 90±2 154±2

14±2 90±2 168±2

Table 7.1: Hoop angle detected

to this problem is to make sure the angle detected is when the hoop’s center and image center

are match. To make sure this work we test it while the drone is flying. However, the angle

detected is incorrect although the two centers are match. After doing some research, we find

that the problem might be due to motion blur. Therefore, we have to find another solution to

this problem which is to use a marker to indicate the orientation of the hoop.

7.2 Ground Robot Assistance Using Marker

The orientation indication marker (Figure 7.1) consists of two non concentric circles.

The two centers of the non concentric circles form a straight line which intersects the vertical

line of the image. Angle ✓ of Figure 7.2 indicates the orientation di↵erence between the marker

and drone. Then the turning algorithm will send command to the drone to rotate until the

di↵erence become zero so that they will align with each other. At the same time, the marker

is also an visual servoing marker. The drone will be able to track and align with it. While the

drone is not facing to the hoop, the ground robot will lead the drone to face to the hoop. Then

26

Figure 7.1: Orientation Indication Marker

Figure 7.2: Orientation Calculation

the drone will only need to do the hoop detection and then going through.

27

(a) Ground robot with marker

(b) UAV is ready to fly through

Figure 7.3: UAV and ground robot

28

Chapter 8

Experiment Results

8.1 Experiment Setup

We use TurtleBot 2 as the ground robot platform for tracking purpose. TurtleBot is controlled

by a joystick and it has moving forward/backward and turning functions. A marker is placed

on the TurtleBot so that the UAV is able to track the TurtleBot (i.e., marker).

Figure 8.1: Experiment Setup

A marker is on the ground behind the hoop. The marker points to west. At two meters

west of it, another marker is placed on the ground pointing to south.

29

8.2 UAV Behaviors

Basically, the overall UAV behaviors can be described as a finite state machine (Figure 8.2).

First, the UAV takes o↵ from the TurtleBot and starts tracking it.

Next, the TurtleBot will navigate the UAV to the front of the hula hoop. After that, the

TurtleBot turns to face the hula hoop which indicates the orientation of the hoop for the UAV.

The UAV will align with the TurtleBot, and in doing so, the UAV is aligned with the hula hoop.

Then the UAV starts to detect the hula hoop and fly through. At the same time, the

TurtleBot will go to the end of the route to prepare. After the UAV finishes flying through the

hoop, it should be able to find a marker pointing to west. Hence, it will hover on the marker

and turn to align with it. After finish turn, it will perform next task — fly forward to find next

marker.

Similarly, as the next marker points to south, the UAV will hover on the marker and turn

to face south. Then it will fly forward with climbing up and down. After finished, it will start

hover until the TurtleBot navigates it to the next hoop and perform other tasks. Since we only

have one hula hoop, the TurtleBot will navigate the UAV to the same hoop and fly through

again with other tasks following.

8.3 Discussion

We conduct the same experiments thirteen times to make sure our system works as expected.

Through our observation, the average performance is satisfactory. The UAV is able to track

the ground robot consistently. For flying through the hoop, the UAV is able to fly through

accurately and fast. However, some drifting problems are present especially while the UAV

is turning. The drifting problem may cause the UAV to lose the ground robot and marker

temporarily. A demo video will be presented during my presentation.

30

Take o↵

Track

TurtleBot

Ready to go?

Detect and

fly through

the hoop

Track and

align with

the marker

Go to next

marker

Track and

align with

the marker

Fly forward,

climb up

and down

Track

TurtleBot

Detect and

fly through

next hoop...

no

yes

Figure 8.2: The state transition diagram for the final experiment

31

Chapter 9

Conclusion And Future Work

We have presented the design and implementation of a real-time vision-based system for au-

tonomous control and navigation. The autonomous control algorithm is accurate and stable

with high flexibility. The usage of wide-angle lens improves the vision system. The vision

algorithm is fast, robust, and computationally inexpensive.

Observation from thirteen flight experiments shows that our system works accurately and

repeatably. At present, the detection of the hoop while not aligned relies on the indication of

ground robot. In the future, we plan to improve the vision algorithm so that the UAV is able

to detect the orientation of the hoop itself. Also, we plan to improve the landing algorithm so

that the UAV is able to land on a moving target.

In the future, we plan to focus our attention on multi-UAV coordination and motion-

planning. Since complex tasks will require more than one UAV and more than one ground

robot, the coordination and motion-planning is important to prevent collision between UAVs

and to allocate the tasks e�ciently.

32

References

Amidi, O. (1996). An autonomous vision-guided helicopter. PhD thesis, Carnegie MellonUniversity.

Conway, A. R. (1995). Autonomous control of an unstable model helicopter using carrier phasegps only. PhD thesis, Stanford University.

Gurdan, D., Stumpf, J., Achtelik, M., Doth, K.-M., Hirzinger, G., & Rus, D. (2007). Energy-e�cient autonomous four-rotor flying robot controlled at 1 khz. 2007 IEEE InternationalConference on Robotics and Automation (pp. 361–366), IEEE, Roma, Italy, April, 2007.

Hjalmarsson, H., Gevers, M., Gunnarsson, S., & Lequin, O. (1998). Iterative feedback tuning:theory and applications. Control Systems, IEEE, 18 (4), August, 1998, 26–41.

Rufli, M., Scaramuzza, D., & Siegwart, R. (2008). Automatic detection of checkerboards onblurred and distorted images. Proceedings of the IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS 2008), Nice France, September, 2008.

Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006a). A flexible technique for accurateomnidirectional camera calibration and structure from motion. Proceedings of the FourthIEEE International Conference on Computer Vision Systems (ICVS 2006), New York,NY, January, 2006.

Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006b). A toolbox for easily calibratingomnidirectional cameras. Proceedings to IEEE International Conference on IntelligentRobots and Systems (IROS 2006), Beijing China, October, 2006.

33