PR inRobotics
1/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Pattern Recognition in Mobile RoboticsCSE 455/555 Introduction to Pattern Recognition
Dr. Julian Ryde
SUNY at Buffalo
January 21, 2011
PR inRobotics
2/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Abstract/Lecture Overview
A more realistic set of motivating application examplesbased in Mobile Robotics
Motivations for mobile roboticsChallenges of mobile roboticsExamples involving pattern recognitionNot going to talk about Industrial/static robotics
Even thought there are many applications of patternrecognition in this area of robotics
PR inRobotics
3/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
What is Robotics?
DefinitionsIntelligent connection of perception to action
Perception - Acquisition and interpretation of sensordataIntelligence?
Decision makingLearning
Actuation - Effecting changes in the physical worldLocomotionManipulation
Roles of pattern recognition in roboticsPattern recognition also has a role to play in intelligencePattern recognition forms part of the interpretation
PR inRobotics
4/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Robotics Areas
StaticIndustrial roboticsControlled environmentsMore commercially successful
MobileIndoor Robotics
Personal roboticsOffice robots
Remote telepresenceDomestic robots
Roomba - most successful consumer mobile robot?Field Robotics
MiningMilitary
Health careHome care for the elderly (Especially Japan)
Wherever it is Dirty, Dull or Dangerous
PR inRobotics
5/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Willow Garage
Long term funded company that does not need to makea profitAdheres to the open source development modelRobot Operating System (ROS)OpenCV
Computer VisionMachine Learning algorithms
PR2 RobotPR2 Beta Overview Video
PR inRobotics
6/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Sensors
Internal sensorsGlobal position system - GPSInertial Measurement Unit - IMU
Vision basedStandard CamerasOmnidirectional
Range sensorsTime of flightTriangulation based
Structured lightStereo vision
UltrasonicKinect
Structured light 3D depth sensorReturns RGBD - Red, Green, Blue and DepthInexpensive at 150$
PR inRobotics
7/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Pattern Recognition in 3D Data versus Pictures
More difficult for humansMore suitable for computers?
3D data is more independent of observer positionMany objects are rigid or articulated so 3D shape isimportant and invariantFor features invariant is good
implies reliable/repeatable extraction
Camera images susceptible toillumination changesloss of information in conversion from 3D to 2D
PR inRobotics
8/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Considerations specific to mobile robotics
AutonomyThere can be no guarantee of continuous connection.Therefore a certain level of autonomy is required forreliable continuous operation.
Power consumptionRestricted processing power
Especially for indoor and smaller robots that run onbatteries
Real-time constraintsPedestrian recognition has to be fast and reliable forautomotive applicationsBatch algorithms not really suitable
For long term operation robot ultimately has to processdata as fast as it receives it.
PR inRobotics
9/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Localisation and Mapping
LocalisationFor most applications need to have an awareness oflocationAugment the robots workspace with features
Possible but expensive for factories and other controlledenvironments
MappingLearning a model of the operatingworkspace/environmentAids localisationAids path planning
Simultaneous Localisation and Mapping (SLAM)The process of both inferring location from a map whilstsimultaneously updating the map with new observedinformation.
PR inRobotics
10/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Simultaneous Localisation and Mapping(SLAM)
SLAM is vital for indoor robotsInterestingly not used for the iRobot roomba
Outside not as necessary where GPS coverage isavailableVarious flavours of SLAM
MetricTopological
Matching2D/3D scan matchingImage
PR inRobotics
11/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Loop Closing
Closing of an open loop in a large (80m by 25m) cyclicenvironment from Konolige and Gutmann (1999)Improving map accuracy through loop closing
Have I been here before?Recognising previously visited locations
PR inRobotics
12/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Scan matching
Two scans taken from different posesFind features present in each scanDetermine the correspondence between featuresUse this information to align the scansThis gives the pose change between scansMuch research done with 2D range scansExample with 3D scans or an indoor environment
Scan matching Ilustration
PR inRobotics
13/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
3D mapping indoors
3D map fly around video
PR inRobotics
14/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Probabilistic robotics
Return not only an answer but some idea of theprobability distribution
For example the pose distribution can be multimodalMany ways of representing these probablity densityfunctionsThus you have some idea of how likely the result is tobe correct
Algorithms in robotics need to be robustRobust statistics e.g. median
contrast with sufficient statisticsInsensitive to outliers
Sufficient statistics e.g. meanMathematically analyticComputational convenientSensitive to outliers
PR inRobotics
15/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
DARPA Grand and Urban Challenges
Grand Challenge$2 million Prize awarded to Stanford Racing Team132 mile desert course in just under 7 hours
Urban challengeFollow on from the easier grand challengeAutonomous vehicle drive through an urbanenvironment
Drive in trafficManeuvers merging, passing, parking, negotiatingintersections
Featured the first autonomous car crashReliable perception and interpretation vitalFinal highlights
DARPA Urban challenge highlights video
Key EnablersVelodyne 3D laser scannerNear to far learning
PR inRobotics
16/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Learning for mobile autonomous systems
Pattern recognition enables systems to learn andgeneralise from new data.Interaction with humans provides plenty of opportunityfor supervised learning.Amazon’s mechanical turk
Humans working for the robots!DARPA urban challenge videos
near to far learning in images e.g. using laser data andcorresponding vision results to train and so classify farimage pixels.
Self supervised learningTraining data might arrives at a later time for somecasesTraining data provided by others sensors
e.g. Near to far learning
PR inRobotics
17/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Near to Far Learning
Image and research from Raia HadsellOnline machine learning
Either laser or stereo provides reliable near range 5mclassification (driveable or not based on smoothness)Then used as training data for appearance basedclassifier for pixels beyond 5m
PR inRobotics
18/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Solutions in Perception Challenge
GoalRigid objects recognized and their 6DOF posedetermined with the Kinect sensor
Kinect sensorBest solutions likely to combine conventional imageinformation with depth informationIdeal opportunity to put into practice what you learn inthis course
PR inRobotics
19/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Field robotics
Work undertaken at the Australian CommonwealthScientific and Industrial Research Organisation(CSIRO)Autonomous earth moving
PR inRobotics
20/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Autonomous Skid Steer loader (Bobcat)
Earth moving taskMove soil material from one location to another
Ultimate goal is the design the terrain you want in aCAD package and have it replicated in the physicalworld by autonomous excavation and earth movingmachines
PR inRobotics
21/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Reliable Human/Novelty Detection
Learn the operational environmentHighlight those areas that have changedEssential for safety reasons
If unknown objects detected or the environment hasbeen tampered with need to potentially stop operation
Handle noise, how much variation constitutes change?
PR inRobotics
22/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Some active research questions related toPattern Recognition
SLAMMonocular camera based visual SLAMSLAM with RGBD sensors
Correspondence problemFinding corresponding points in multiple imagesFor stereo vision
Object recognition in 3D data3D shape based retrieve and lookupView and occlusion invariant hashingLocality sensitive hashing
Object recognition from a moving camera sensorDifferent to object recognition in random imagesBackground subtraction for moving cameras
PR inRobotics
23/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Recap
What is (mobile) robotics?Example applications involving pattern recognition
LocalisationMapping
Scan matchingLoop closing
Terrain classification for path planningNear to far learningSelf supervised machine learning
Field roboticsAutonomous Earth moving
Current work in the research communitySolutions in Perception Challenge
PR inRobotics
24/24
Dr. JulianRyde
Introduction
Challenges
ExampleApplications
Summary
Questions for consideration?
What is holding back mobile robotics?Was the lack of low cost depth sensors, but now withthe Kinect?Low cost manipulator platforms? The PR2 is $400,000
Motivated by the recent debut of the Kinect how canexisting computer vision pattern recognition algorithmsbe applied to RGBD data?