Steps Toward Developing an Intelligent Robotics Course Kutztown University

Steps Toward Developing an Steps Toward Developing an Intelligent Robotics CourseIntelligent Robotics Course

Kutztown UniversityKutztown University

PACISE 2011PACISE 2011April 9, 2011April 9, 2011

Oskars J. RiekstsOskars J. Rieksts

Jeffrey W. MintonJeffrey W. Minton

2011 Kutztown University 2

Acknowledgments – FundingAcknowledgments – FundingCS Department & LAS CollegeCS Department & LAS CollegeTYCO Electronics FoundationTYCO Electronics FoundationKutztown University FoundationKutztown University FoundationKU Faculty Professional KU Faculty Professional

Development CommitteeDevelopment CommitteeKU Undergraduate Research KU Undergraduate Research

CommitteeCommitteePASSHE Faculty Professional PASSHE Faculty Professional

Development CouncilDevelopment Council

2011 Kutztown University 3

MotivationsMotivationsRodney BrooksRodney BrooksDARPA Grand ChallengeDARPA Grand ChallengeREU @ AuburnREU @ AuburnHardware-software synthesisHardware-software synthesis

““Hardened” hardwareHardened” hardwareCentral role of softwareCentral role of software

2011 Kutztown University 4

MotivationsMotivationsApplied A.I.Applied A.I.Applied cognitionApplied cognitionMobilityMobilityAssistive roboticsAssistive robotics

Robotic wheelchairsRobotic wheelchairs

2011 Kutztown University 5

OutlineOutlineEarly effortsEarly efforts

Hacking Roomba & other venturesHacking Roomba & other venturesResearch Experience for Research Experience for

UndergraduatesUndergraduatesCreate and MindstormsCreate and MindstormsCaptain KURKCaptain KURKTrinity firefighting contestTrinity firefighting contestMyro and PythonMyro and PythonReading-Berks Science FairReading-Berks Science Fair

2011 Kutztown University 6

Coming into FocusComing into FocusEmerging goalsEmerging goals

IntelligenceIntelligenceVisionVisionCommuncationCommuncationPhilosophical issuesPhilosophical issues

2011 Kutztown University 7

Point of convergencePoint of convergenceCourse in intelligent roboticsCourse in intelligent robotics

ControlControlVisionVisionCommuncationCommuncationInvestigate cognitive issuesInvestigate cognitive issues

2011 Kutztown University 8

Course design objectivesCourse design objectivesRobot isRobot is

SituatedSituatedEmbodiedEmbodied

Shared environmentShared environmentHumanHumanMachineMachine

Shared communicationShared communicationFramework for investigationFramework for investigation

2011 Kutztown University 9

Robot is situatedRobot is situated

Operates within an environmentOperates within an environmentEmbedded in the worldEmbedded in the worldChief knowledge source:Chief knowledge source:Data stream drawn from Data stream drawn from

environmentenvironment““The world is its own best The world is its own best

representation” – Brooksrepresentation” – Brooks

2011 Kutztown University 10

Robot is embodiedRobot is embodiedEntity 1Entity 1stst, agent 2nd, agent 2nd

Self reliantSelf reliantSecretary of State modelSecretary of State model

Self-seated conceptual frameworkSelf-seated conceptual frameworkWorld concepts grounded in sensor suiteWorld concepts grounded in sensor suiteBehavior set emanates from actuator Behavior set emanates from actuator

suitesuiteDecision apparatus grounded in Decision apparatus grounded in

sensor/actuator suitessensor/actuator suites

2011 Kutztown University 11

Robot is embodiedRobot is embodiedEames: Eames: Design is a plan for arranging Design is a plan for arranging

elements to accomplish a particular elements to accomplish a particular purposepurpose

Elephants don’t play chess (Brooks)Elephants don’t play chess (Brooks)““Mind” is to fit the bodyMind” is to fit the body

Design on need to basisDesign on need to basisKnowKnowThinkThinkDoDo

2011 Kutztown University 12

Shared environmentShared environmentShared conceptual Shared conceptual frameworkframework

Robot’s Robot’s Derivative of human’sDerivative of human’sSimplified in structureSimplified in structure

Shared factbaseShared factbaseRobot’s Robot’s

Subset of human’sSubset of human’sSimplified ontologySimplified ontology

Vision basedVision based

2011 Kutztown University 13

Common communication frameworkCommon communication frameworkGrounded inGrounded in

SharedSharedEnvironmentEnvironmentConceptual frameworkConceptual frameworkOntologyOntology

Vision as main sensorVision as main sensor

Intersecting language constructsIntersecting language constructs

2011 Kutztown University 14

Projected course topics/activitiesProjected course topics/activitiesSpecialized robot control softwareSpecialized robot control softwareRobot control architectureRobot control architectureBasics of image processingBasics of image processingSpecialized image processing Specialized image processing

softwaresoftwareBasics of communication theoryBasics of communication theoryKey issues of cognitive roboticsKey issues of cognitive robotics

2011 Kutztown University 15

Slow, steady progressSlow, steady progress

Browning:Browning:Reach should exceed graspReach should exceed grasp

Rapid prototypingRapid prototypingIterative developmentIterative development

2011 Kutztown University 16

Structure of the classStructure of the classFour teamsFour teamsRolesRoles

Team leaderTeam leaderDesignerDesignerCoderCoderDocument guruDocument guruHistorianHistorianTest designerTest designerTest administratorTest administratorHardware specialistHardware specialist

2011 Kutztown University 17

PACT demonstrationPACT demonstrationPennsylvania Association of Pennsylvania Association of

Council of TrusteesCouncil of TrusteesVery early in learning curveVery early in learning curveNavigation within a “corral”Navigation within a “corral”Well receivedWell received

Gateway navigationGateway navigationBull fighting robotBull fighting robot

PR for robotics @ PASSHEPR for robotics @ PASSHE

2011 Kutztown University 18

Jeff – gateway navigationJeff – gateway navigation

2011 Kutztown University 19

Investigate cognitive issuesInvestigate cognitive issuesSight and touchSight and touch

George StrattonGeorge StrattonSpatial harmony of sight & touchSpatial harmony of sight & touch

SpaceSpaceBenjamin KuipersBenjamin KuipersSemantic spatial hierarchySemantic spatial hierarchy

MetricalMetricalTopologicalTopologicalHybridHybrid

2011 Kutztown University 20

Investigate cognitive issuesInvestigate cognitive issuesLanguage and meaningLanguage and meaning

Stevan HarnadStevan HarnadSymbol grounding problemSymbol grounding problem

Language and shared spaceLanguage and shared spaceAmichai KronfeldAmichai KronfeldShared referent problemShared referent problem

2011Kutztown University

21

Investigate cognitive issuesInvestigate cognitive issuesEmbodied cognitionEmbodied cognition

Randall BeerRandall BeerSituated, embodied, dynamicalSituated, embodied, dynamical

Principle of interactivismPrinciple of interactivismMark BickhardMark BickhardEmergence of representational Emergence of representational

contentcontent

2011 Kutztown University 22

Illustration – Shared Referent ProblemIllustration – Shared Referent ProblemFrom KronfeldFrom Kronfeld

Steve & wife Bev are at a partySteve & wife Bev are at a partySteve: Bob & his wife are certainly Steve: Bob & his wife are certainly

enjoying the partyenjoying the partyBev: Sally is not his wife!Bev: Sally is not his wife!

NoteNoteSteve & Bev share a referent, SallySteve & Bev share a referent, SallyDespite misidentification no Despite misidentification no

communication problems arise!communication problems arise!

2011 Kutztown University 23

The KLO TriadThe KLO TriadFor communication involving For communication involving

referentsreferentsFor each communicant – this triadFor each communicant – this triad

Object (the referent)Object (the referent)KB – representation of objectKB – representation of objectLanguage – reference to objectLanguage – reference to object

For successful communicationFor successful communicationReferents must matchReferents must matchFor HMC (Human-machine For HMC (Human-machine

communication)communication)Human & machine KBs do Human & machine KBs do notnot match match

2011 Kutztown University 24

Establishing co-refrenceEstablishing co-refrencePer Frege’s distinctionPer Frege’s distinctionExtensional approachExtensional approach

Pointing or equivalentPointing or equivalentIntensional approachIntensional approach

Language aloneLanguage aloneHMC goalHMC goal

Minimize extensionalMinimize extensionalMaximize intensionalMaximize intensional

2011 Kutztown University 25

The Trinity RobotThe Trinity Robot

2011 Kutztown University 26

The Trinity RobotThe Trinity RobotVEX robotics kit for chasisVEX robotics kit for chasisVEX included motors for locomotionVEX included motors for locomotionSonar rangefindersSonar rangefindersWeb-camWeb-camArduino to control motors and sensorsArduino to control motors and sensorsNetbook to control Arduino and Netbook to control Arduino and

process imagesprocess images

2011 Kutztown University 27

The Trinity RobotThe Trinity RobotInaccurate motor controlInaccurate motor controlSonar signals bounce inside Sonar signals bounce inside

cornerscornersProvide inaccurate measurementsProvide inaccurate measurements

Viewing angle of web-cam too Viewing angle of web-cam too smallsmall

2011 Kutztown University 28

BrobiBrobi

2011 Kutztown University 29

Brobi – hardwareBrobi – hardwareCreate by iRobotCreate by iRobotPlatform built onto Create cargo Platform built onto Create cargo

bay to accommodate equipmentbay to accommodate equipmentWeb-cam with increased viewing Web-cam with increased viewing

angleangleIR rangefinderIR rangefinder

IR light does not bounce like soundIR light does not bounce like soundArduinoArduino

2011 Kutztown University 30

Brobi – softwareBrobi – softwareOpenCV and PythonOpenCV and PythonConsultation – John SpletzerConsultation – John Spletzer

LehighLehighLittle Ben in Urban ChallengeLittle Ben in Urban Challenge

MATLABMATLABImage processingImage processingCreate APICreate APIA programming languageA programming language

2011 Kutztown University 31

Image processingImage processing““A picture is worth a thousand A picture is worth a thousand

words”words” need to extract discrete objects need to extract discrete objects

from images to identify themfrom images to identify themK-Means clusteringK-Means clustering

2011 Kutztown University 32

Goal: identify green ballGoal: identify green ball

2011 Kutztown University 33

K-Means ClusteringK-Means ClusteringCluster sets of dataCluster sets of data

Into user-defined number of Into user-defined number of segmentssegments

Number of segments referred to as KNumber of segments referred to as KClusters defined by MEAN of all Clusters defined by MEAN of all

values in clustervalues in cluster

K-Means AlgorithmK-Means Algorithm

K-Means ExampleK-Means Example

K-Means using k = 6

K-Means using k = 5

2011 Kutztown University 36

Natural Language ProcessingNatural Language Processing

““Go to the green ball”Go to the green ball”The meanings behind words must be The meanings behind words must be

inferredinferred““Go,” conceptually can represent Go,” conceptually can represent

many thingsmany thingsTake a turn in a gameTake a turn in a gameThe Chinese strategy gameThe Chinese strategy gameTravel to a locationTravel to a location

Determine the concept being referred Determine the concept being referred toto

Conceptual parsingConceptual parsing

2011 Kutztown University 37

Conceptual ParsingConceptual Parsing

Words are mapped to conceptsWords are mapped to concepts

ConceptsConceptsRules define set of related conceptsRules define set of related conceptsOne concept may have many One concept may have many

separate rule setsseparate rule sets

Concept TreeConcept Tree

2011 Kutztown University 39

Assessment – PlatformAssessment – PlatformBest hardware platform to dateBest hardware platform to dateOpens up many avenues of course Opens up many avenues of course

developmentdevelopmentSoftware for robot control systemsSoftware for robot control systemsRobot control architecturesRobot control architecturesImage processingImage processingCommunicationCommunicationCognitive robotics issuesCognitive robotics issues

2011 Kutztown University 40

Assessment – MATLABAssessment – MATLABCan do all 3 thingsCan do all 3 things

Sensor/actuator interface with Sensor/actuator interface with Lehigh APILehigh API

Image acquisition & processingImage acquisition & processingRobot control system programmingRobot control system programming

Good IDEGood IDEGood documentationGood documentation

17 pdf files; 64 mB17 pdf files; 64 mBOnline documentationOnline documentation

2011 Kutztown University 41

Assessment – MATLABAssessment – MATLABGood IDEGood IDEStrong user communityStrong user community

BlogsBlogsDiscussion boardsDiscussion boardsGood code segmentsGood code segments

Good programming languageGood programming languageMatrix optimizedMatrix optimizedAdvance features, e.g., lambda, Advance features, e.g., lambda,

applyapply

2011 Kutztown University 42

Assessment – CreateAssessment – CreateUsefulUsefulStableStableUbiquitousUbiquitousNegativesNegatives

Battery short lastingBattery short lastingDoes not travel in straight lineDoes not travel in straight line

2011 Kutztown University 43

Assessment – AccessibilityAssessment – AccessibilityCost not prohibitiveCost not prohibitive

Create – $130Create – $130$220 with battery and charger$220 with battery and charger

Netbook ~ $330Netbook ~ $330Arduino ~ $30Arduino ~ $30MATLABMATLAB

$900 for 10 seat license$900 for 10 seat license$130 for student version with image $130 for student version with image

acquisitionacquisition

2011 Kutztown University 44

Future directionsFuture directionsSoftware repositorySoftware repositoryLearningLearning

BayesianBayesianGenetic algorithmsGenetic algorithmsNeural networks, etc.Neural networks, etc.

Control architecuresControl architecuresBehavior-basedBehavior-basedHybridHybridBlackboard?Blackboard?

2011 Kutztown University 45

Future directionsFuture directionsExperiment with other sensorsExperiment with other sensors

Touch {whiskers}Touch {whiskers}OdorOdorSound {whistle, a la Sound of Sound {whistle, a la Sound of

Music}Music}HeatHeatOdorOdorLight/brightnessLight/brightness

Wireless interfaceWireless interface““Watson, come here. I need you”Watson, come here. I need you”

2011 Kutztown University 46

Future of robotics at KUFuture of robotics at KUStrong student interestStrong student interestRejuvenated courseRejuvenated course

New syllabusNew syllabusApprovedApproved400 level – a mixed blessing400 level – a mixed blessingNot yet scheduledNot yet scheduledIn rotation?In rotation?PoliticsPolitics

2011 Kutztown University 47

Further InformationFurther InformationJeff’s graduate thesisJeff’s graduate thesis

Do You See What I'm Saying: Do You See What I'm Saying: Relating Language and Vision to Relating Language and Vision to Create Interaction Between Humans Create Interaction Between Humans and Robotsand Robots

Delves further into concepts Delves further into concepts discussed herediscussed here

https://github.com/jeffminton/Thesis

2011 Kutztown University 48

Questions?Questions?

2011 Kutztown University 49

The EndThe End

2011 Kutztown University 50

Extra slidesExtra slides

2011 Kutztown University 51

TemplateTemplateFont 44Font 44

font 40font 40Font 44Font 44

font 40font 40Font 44Font 44

font 40font 40Font 44Font 44

font 40font 40

2011 Kutztown University 52

AssessmentAssessmentSensors can be addedSensors can be addedOffers new areas of explorationOffers new areas of exploration

Heat sensorsHeat sensorsSound sensorsSound sensorsOdor sensorsOdor sensors

2011 Kutztown University 53

Unsuccessful segmentationUnsuccessful segmentation

2011 Kutztown University 54

Successful segmentationSuccessful segmentation

2011 Kutztown University 55

TemplateTemplateFont 44Font 44

font 40font 40Font 44Font 44

font 40font 40Font 44Font 44

font 40font 40Font 44Font 44

font 40font 40