Upload
mohamed-elawady
View
294
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Project Activity - October 2013 B31XP Robotics Project Module Heriot-Watt University VIBOT Promotion 7 (2012-2014)
Citation preview
Detecting and Avoiding Frontal Obstacles from a Monocular
Camera for Micro Unmanned Aerial Vehicles
1Robot Project
WakaWaka Group
Professor: Yvan Petillot
Team Members : H.Kidane , I.Sadek , M.Elawady
Heriot Watt University
School of Electrical and Physical Sciences
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 2
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 3
Introduction
GoalDetecting and avoiding frontal obstacles using Ar-Drone2
Robot Project 4
Introduction
Ar.Drone It is a rotating rigid structure with 6 degree of freedom. The two pair of rotors
rotate in different directions
Robot Project 5
HD Camera 720P , 30FPSVery Light and High
Resistance Plastic
Specific
Propeller
Ultrasound Sensor
Indoor Weight :420g
Price : $300
• Applications:
MAVs play an important role in many applications (i.e.
search, monitoring, rescue, surveillance, etc)
- Able to maneuver rapidly and adequately.
- less dangerous for people.
- Provide real time data to the operator.
• Limitation:
- Limited payload to carry additional sensors.
- Depend on monocular camera.
- Obstacles can’t be observed directly using this camera.
Robot Project 6
Introduction
Outline
• Introduction
• Related Work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 7
Related Work
• Paper1: Learning Monocular Reactive UAV Control in Cluttered Natural EnvironmentsStephane Ross, Narek Melik-Barkhudarov, Kumar Shaurya , Shankar Andreas Wendel, DebadeeptaDey, J. Andrew Bagnell, Martial Hebert
• Paper2: First Results in Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned
Aerial VehiclesTomoyuki Mori , Sebastian Scherer
• Paper3: Autonomous quad rotor flight with vision-based obstacle avoidance in virtual environmentAydın Eresen, Nevrez Imamoglu, Mehmet Önder Efe
Robot Project 8
Robot Project 9
Related Work-Paper1
The system observes a human expert driving the drone
Video Stream
Visual Features
Expert
Input
Unsupervised Learner
Control Command
http://robotwhisperer.org/bird-muri/
Robot Project 10
Related Work-Paper1
Example shows the learning process where learner in this frame
gives wrong results (white line), while the expert provides the correct command
(red line).
Robot Project 11
Related Work-Paper2
This method relies on the relative size change of an object in two
consecutive frames
Robot Project 12
Related Work-Paper2
Position of obstacle
Confirm scale with template matching
Discard key-points (smaller or same size)
Discard mismatch (Distance threshold)
Matching in consecutive frames
Generate surf key points
Robot Project 13
Related Work-Paper3
Take Snap Shot
Image Pre-processingGoal
Achieved
Object Detection
Generate PathYaw Angle
Landing
Controller
No
Yes
Robot Project 14
Related Work-Paper3
• Image pre-processing: resizing and de-blurring
• Object detection: optical flow (Horn and Schunk)
Search
WindowTemplate
Google earth environment
Robot Project 15
Related Work-Paper3
Result
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 16
17
Methodology – Detection
Un-successful Works
Robot Project
18
Methodology – Detection
Semi-Dense Optical Flow
Robot Project
Outline
Check image de-blurring (variance of laplacian)
Robot Project 19
BlurredNot Blurred
Correct Incorrect
Not Blurred (7 Images) 6 1
Blurred (8 Images) 8 0
https://www.mathworks.co.uk/matlabcentral/fileexchange/27314-focus-
measure/content/fmeasure/fmeasure.m
20
Methodology – DetectionBlock Diagram
Level 1
Robot Project
Image at
frame x
[1]
Image at
frame x+k
[2]
(Optional)
Pre-processing
image resizing
and sharpening
[1] Compute
symmetric
feature locations
within step range
Optical-flow
Algorithm
[Gunnar
Farneback]
Mismatch points
removal
[euclidean distance]
(Optional)
[2] Region of interest
(ROI) column selection
[25%]
[2] Split image into
five regions
[FL,NL,CN,NR,FR]
Calculate
average/median
euclidean distance
for each region
Find region with
maximum value
last 5 max ==
current max
&&
current max
>= threshold
yesObstacle
direction
[left/right]
No
No
obstacle
21
Methodology – DetectionBlock Diagram
Level 0
Robot Project
Image at
frame x
Image at
frame x+k Obstacle
direction
[left/right]
No obstacle
Detection
Algorithm OR
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 22
Robot Project 23
Takeoff
Fly Forward
Detected?
Process
video to detect obstacle
Fly sideways
Land/wait_joy_cmDestination/
joy_active?
Yes
Yes
No
No
Avoidance
Robot Project 24
ROS driver for Parrot AR.Drone
Avoidance
• "ardrone_autonomy” developed in Autonomy Lab of Simon Fraser University
Avoidance
• Information from Ar.Drone will be published in ardrone/navdatatopic
• ardrone/navdata
– Battery percent
– Drone state
– Orientations/tilt magnitudes
– pressure
– etc
Robot Project 25
Receiving data from AR.Drone
”ardrone_anatomy“
Avoidance
• ROS camera interface topics to
capture Images/video from Drone
– ardrone/image_raw
– ardrone/front/image_raw
– ardrone/bottom/image_raw
Robot Project 26
Receiving data from AR.Drone
”ardrone_anatomy“
Avoidance
• Drone will takeoff, land, or
emergency stop/reset by
publishing an Empty ROS
messages to the ff topics
– ardrone/ takeoff
– ardrone/land
– ardrone/reset
Robot Project 27
Sending commands to AR.Drone
Avoidance
• To fly the Drone after takeoff,
publish a message of type
geometry_msgs::Twist to the
cmd_vel
• geometry_msgs::Twist expresses
velocity in free space broken into its
Linear and angular
Robot Project 28
Sending commands to AR.Drone
Robot Project 29
Waka_Controller ardrone_driver
/cmd_vel
/ardrone/takeoff
/ardrone/land
/ardrone/reset
/ardrone/front/image_rawWaka_Image_Proce
Avoidance
Autonomous flying controller
Robot Project 30
Avoidance
Integrating with Joystick
/joy
Waka_Controller ardrone_driver
/cmd_vel
/ardrone/takeoff
/ardrone/land
/ardrone/reset
/ardrone/front/image_rawWaka_Image_Proce
Joy_node
Robot Project 31
Sideway velocity for 1s
Avoidance
geometry_msgs::Twist /cmd_vel
Forward Velocity
linear.x: 1m/s (move forward)
linear.y: 0
linear.z: 0
Obstacle in half left
linear.x: 0
linear.y: -2/s move right
linear.z: 0
Obstacle in half right
linear.x: 0
linear.y: 2m/s move left
linear.z: 0
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 32
• Setup:
- Intel Core™ i7-3630 QM Processor
- Clock speed : 2.40 / 3.40 Turbo GHz
- 3rd level cache : 6 MB
- Running OS: Linux (Ubuntu)
Robot Project 33
Experiments
Robot Project 34
Experiments
Control training
virtual obstacle
Online Obstacle
Avoidance test
Detection training
using offline video
35Robot Project
Offline Online
Correct Incorrect Total Correct Incorrect Total
Indoor 7 5 12 7 3 10
Outdoor 8 4 12 - - -
Experiments
Results
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 36
• Optical flow algorithm gives better detection results
comparing with feature-based algorithms
• Control part for avoidance reacts as expected
• Accuracy is reduced due to inaccurate measurement
of time to collusion
Robot Project 37
Conclusion
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 38
39Robot Project
Multi-sensor data / multi-detectors for robust time-to-collision estimation
• Frontal camera with Optical flow is used
Optical flow comparisons across all frames
• One comparison at current frame
Find de-blurring kernel for wiener/lucy algorithm
• Neglect blurring images
Path planning and follow m-line to goal
• Fly forward
Future Work
40Robot Project