View
2
Download
0
Category
Preview:
Citation preview
Achieving Next-Level Robotic Intelligence with SoS Implications
14th International Conference onSystem of Systems Engineering
Sheraton Anchorage20 May 2019
Associate Director, Robotics
Dr. Edward Tunstel, FIEEEtunstel@ieee.org
President
2 E. Tunstel
Robots are here
3 E. Tunstel
High-Tech
4 E. Tunstel
Transdisciplinary
5 E. Tunstel
Robotic intelligence pipeline
§ Advances to date are impacting individual pipeline elements§ How to effectively integrate the pipes to realize practical
intelligent robots and robotic systems is unclear.
§ I do not have all of the answers…§ Will pull the thread on the issue as food for thought:
Ø Some select robotics research topics and applications Ø Some next-level intelligence considerations for each
SENSE
ACTPLAN
INTEGRATED INTELLIGENCE
PER
CEI
VE
PLAN
ACT
CO
NTR
OL
LEARN INTERACT
REASON
6 E. Tunstel
Early Automata at APL (1960s)
HopkinsBeast
APL News, March 1964
Ferdinand
E. Tunstel
7 E. Tunstel
Fast forward…2018Got robot? …and Internet?
§ RoboEarth, WWW for robots: giant repository where robots can share information; learn from each other about their behavior & environment –similarly, Robo Brain
§ Cloud Robotics – cloud computing centered around shared robot computing resources (central to robotics within Industry 4.0)
§ Robot-App Store – an apps marketplace intent on enabling apps sharing among robots
http://www.robotappstore.com/
8 E. Tunstel
Robots and IoT
§ Robots as systems connected to IoT in and around intelligent homes and buildings (including hotels, universities and the workplace) – delivery and other services
StarshipTechnologies (UK)
Amazon Scout
PepsiCo & Robby Technologies
Alibaba
Singapore
9 E. Tunstel
Robots and IoT
§ Robots as systems connected to IoT in manufacturing and logistics environments, future (smart) factories, precision agriculture, etc
§ Robots as the truly physical or apparently tangible nodes in Cyber-Physical Systems
IoT-enabled smart robot: KUKA Connect
10 E. Tunstel
Robotics Research Context
Modeling &Simulation
Hardware+ Software
Theoreticalfoundation
Practicalexperience
Mobility AutonomousNavigation Manipulation Sensing &
Perception
Learning, Adaptation, Deliberative planning, Robust field operation, Systems-thinking,
Transdisciplinary ideas…
...
Breakthroughs
I N N
O V
A T
I O
N
RoboticsResearch
InnovationSpace
11 E. Tunstel
Command Sequencing
Engineering Assessment
Uplink
CommandSequences
Downlink
Telemetry
Science Team
scienceactivities
autonomousexecution
• Best health knowledge
•Recommendations• engineering & image data• science data
Spirit / Opportunity
Semi-autonomous operations from Earth
Intelligence and Autonomy• Mission intelligence (science/exploration) is largely human while remote autonomy
is necessarily robotic• Sequencing and analysis teams plan and assess robotic activities using their
perception of the rover surroundings and knowledge of rover state and behavior
12 E. Tunstel
Mars Rover Technology & Robotic Intelligence
§ Autonomous navigation§ Local/global waypoint planning§ Dense stereo vision§ Autonomous terrain assessment§ Visual odometry§ Goal-driven visual servoing§ Robotic arm motion planning§ Precision arm placement§ Terrain sampling and handling§ Autonomous fault response§ Cmd sequencing & visualization
M. Maimone, J. Morrison, JPL
13 E. Tunstel
Next up…https://mars.nasa.gov/mars2020/
§ Faster navigation§ More modern path planning
(RRT, particle filters)
14 E. Tunstel
Next-Level rover intelligence: Mars Sample Return
§ Richer set of capabilitiesØ Sample acquisition & handlingØ Sample fetch & retrievalØ Lander detection & rendezvous
§ Mobility & manipulation capability maturation for sample caching andfetch rovers (prototypes demonstrated in field tests > a decade ago *)
§ Mars 2020 rover is representative of the sample-caching rover in Mars sample return mission concepts
§ Extreme terrain access (cliffs, caves, subsurface lava tubes, etc)
JPL
Schenker, Huntsberger, Pirjanian, Baumgartner and Tunstel, "Planetary Rover Developments Supporting Mars Exploration, Sample Return and Future Human-Robotic Colonization," Autonomous Robots, Vol. 14, 2003.
*
15 E. Tunstel
Intelligent Co-Robots
Beyond mobile sensing and toward environment manipulation and intuitive & physical HRI
Background imgae: http://gp-email.brtapp.com
16 E. Tunstel
Intelligent Co-Robots – Comms. network topology§ UGVs, UAVs, and a ground
link to the user are connected using a wireless mesh Mobile Ad hoc NETwork (MANET)
§ Mini-MACSS uses a COTS comms module that cannot directly link to the Wave Relay mesho One UAV assigned to act as
bridge to mesh (Mini-MACSS is outside of MANET)
17 E. Tunstel
DARPA Robotics Challenge Tech Exposition 2015
MULTI-ROBOT SEARCH & SAMPLINGIN INCREASINGLY CONSTRAINED ENVIRONMENTS
“Russian Doll” scenario
UGV à UAV à micro-UGV
§ A unique demonstration scenario that focused our development of underlying capabilities in key IRAD areasØ Autonomous UAV and UGV mobility/navigationØ Intelligent co-robots and human-robot teamingØ Dexterous manipulationØ Robot vision and perceptionØ Data fusion, distribution, and display
RoboSally
Pelican
Mini-MACCS
Moore, J., Wolfe, K.C., Johannes, M.S., Katyal, K.D., Para, M.P., Murphy, R.J., Hatch, J., Taylor, C.J., Bamberger Jr., R.J. and Tunstel, E., "Nested Marsupial Robotic System for Search and Sampling in Increasingly Constrained Environments," 2016 IEEE Intl. Conf. on Systems, Man, and Cybernetics, Budapest, Hungary, pp. 2279-2286. Oct. 2016.
18 E. Tunstel
Mini-MACSSMiniature Autonomous Crawling Surveillance System
19 E. Tunstel
Human Interface
Operator Console
Keyboard and Mouse
Multi-monitor Display
State Info. & Situational Awareness: maps, state estimates, video, point clouds
User Intent: motion primitives, desired poses
User Confirmation and/or Refinement of Plan
Planned Behavior
Remote Robot Team
UGV: Robo-Sally
UAV: Modified AscTec Pelican
Miniature Ground Vehicle Joystick
20 E. Tunstel
DRC Tech. Expo. demo scenario
21 E. Tunstel
Demo video
https://www.youtube.com/watch?v=Hvh20ySwgPw
22 E. Tunstel
Next-Level manipulation intelligence§ Enhancing perception capabilities beyond
the visual modality
§ Moving beyond object recognition and grasping to knowledge and reasoning about object properties
§ Dense tactile arrays / e-skin, and sensor processing (e.g., neuromorphic)
CEA LIST, FranceKing’s College, London
23 E. Tunstel
Beyond ISR toward environment manipulationand intuitive & physical HRI
Autonomy for Marsupial Robot Team
24 E. Tunstel
Autonomous Team
Planning
Collision-free motion planning
Marsupial-team motion planning
Whole-body planning
Collaborative mapping
Autonomous Team Planning
Vision and Object Recognition
https://www.youtube.com/watch?v=7Gz7yjYqEEM
P.G. Stankiewicz, S. Jenkins, G.E. Mullins, K.C. Wolfe, M.S. Johannes and J.L. Moore, "A Motion Planning Approach for Marsupial Robotic Systems," 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, Sept. 2018.
25 E. Tunstel
Next-Level multi-robot system intelligence
§ Realizing smart behavior for not only singular robots but for multi-robot systems
§ Tactical behaviors with greater, situational intelligence
§ Coupling w/high-level reasoning systems/ architectures
§ Move beyond swarm intelligence to enable multi-functional swarms that do more than distributed sensing – e.g., manipulation of their environment
26 E. Tunstel
User Interaction & Interfaces
Next Generation First ResponderDHS
27 E. Tunstel
Head-Up, Hands-Off HRI
Current robots require oversight from a trained operator, resulting in reduced
alertness and higher cognitive workload for one or more human teammates. This results in a less effective team.
A conversational human-robot interface using standard, unambiguous military language would allow more intuitive robot C2 and would reduce
the human cognitive burden. This results in more efficient and safer operations.
“Head-Down, Hands-On” “Head-Up, Hands-Off”
28 E. Tunstel
Motivating Use Cases
29 E. Tunstel
Multi-modal and Robot agnostic
{Voice Cmds} {Gestures} {BMI} {AR}
{Behaviors}
BRUCE
Set of common high-level
commands
Other Robots
Multi-modal Head-Up, Hands-Off HRI
BMI ARVoice Text
Gesture
30 E. Tunstel
EWT-30
DARPA RP-2009 (to present):Modular Prosthetic Limb§ New approach to limb systems closely
mimicking human natural form and movements (developed for amputees)
§ Control options range from non-invasive or minimally-invasive techniques to use of sets of wireless implants (targeted muscle reinnervationto thought control)
§ Modular design (hand only, hand+ forearm, whole arm); all power, actuation, and control self contained
§ APL served as lead & systems integrator for a large multi-institutional team
31 E. Tunstel
EWT-31
Thought control…
§ “Breakthrough: Robotic limbs moved by the mind”
§ Scott Pelley reports on movement of robotic limbs using human thought; reception of sensory feedback from a robotic hand
§ Subject, Jan Scheuermann, suffers from a genetic disease that severs the brain-body connection (spinocerebellar degeneration)
http://www.jhuapl.edu/newscenter/stories/
CBS’60 MinutesDec. 2012
32 E. Tunstel
EWT-32
Targeted muscle reinnervation…
§ Breakthrough: Robotic prosthetic attachment to implant in bone
§ Subject, John Matheny, lost arm to cancer in 2008§ Implant inserted in marrow space of bone replaces constricting and
potentially uncomfortable harnesses§ Prosthetic guided by brain signals via nerve reassignment surgery
https://hub.jhu.edu/2016/01/12/prosthetic-limb-more-mobility-apl/
33 E. Tunstel
Robotic Exoskeletons
§ Human Augmentation (able bodied users)Ø Provides user augmentation or assistance to
accomplish generalized tasking
§ Technical ChallengesØ Other than engineering (power, actuation,
controls, ergonomics, physical constraints)
Ø Control software that can assist operators in highly dynamic behaviors.
Ø Exoskeleton control to seamlessly sense and interpret the operator’s intended behavior and then introduce assistive power that coordinates with the operator's motions.
34 E. Tunstel
Applications
§ Health CareØ Functional RestorationØ AssistanceØ Rehabilitation
§ IndustrialØ LogisticsØ Load CarriageØ Heavy Machine Operations
§ Military & Law EnforcementØ Operator Assist Carrying Heavy LoadØ Improve Operator Performance Under
Stressed ConditionsØ Tactical ProtectionØ Heavy Load Carrying
35 E. Tunstel
Next-Level interaction / interface intelligence
§Smart human-collaborative robots that are responsive to intuitive, physical, and brain-interfaced interaction
§Human-robot fusion – Neural interfaces, BMI, etc
§ Tighter feedback loops with brain, EMG signals, etc enabling the robotics to make smart decisions and take appropriate shared control actions leveraging the human as a sensor & supervisory controller
36 E. Tunstel
Modular Open Systems Architecture:Example: AEODRS Standards-based Common Open Architecture
Autonomous BehaviorsVisual SensorsManipulator End Effector Master Comm Link Power
System
InfrastructureUGV
DismountedUGV
TacticalUGV
AEODRS standardized wireless link
Handheld OCU Common OCU
AEODRSElectrical Interface
AEODRSPhysical Interface
AEODRSLogical Interface
AEODRS standardized
interfaces
System Capabilities
Interchangeable and Interoperable
37 E. Tunstel
AEODRS – A Modular Open System
38 E. Tunstel
AEODRS – A Modular Open System
39
AEODRS Capability Modules and Distributed Architecture
Implication for next-level robotic intelligence…but intelligence is non-trivial to modularize in a similar manner
Softweare/Hardware System Test Bed
40 E. Tunstel
Next-Level systems intelligence
§ MOSA provides interoperability, but that is not enough; need “conversability” or dialog as well, enabling next-level robotic intelligence to interoperate with humans & IoTØperhaps borrowing techniques from MAS wherein
agents communicate, not solely to pass data and messages, but particularly in order to achieve their goals
§ Greater leverage and improved adaptation of cognitive architectures (ACT-R, Soar, DUAL, CLARION, etc)
§ Better ways to modularize intelligence
41 E. Tunstel
Robot Learning – Next-Level
§ Need to advance from learning for X (perception, control, etc) Ø autonomous/developmental learningØ knowledge/skill transfer
Next-level intelligence for robot learning moves toward:
§ Autonomous learning (e.g. book by P Angelov)
§ Empirical machine learning (new book. P. Angelov)
§ Broad learning (P. Chen)
§ Robot memetics
PER
CEI
VE
PLAN
ACT
CO
NTR
OL
LEARN INTERACT
REASON
42 E. Tunstel
Robot Memetics
Forthcoming book (colleagues and I):
§ Concept of memes and memetics as elements of collective robotic intelligence and implications for emergence of a hybrid community of humans and intelligent robots
§ Illustration of robot memetics ideas in the context of a space exploration scenario (development and operation of a human-robot settlement on Mars)
§ New ways of thinking about how to realize higher levels of intelligence and learning in robots and robotic communities
43 E. Tunstel
44 E. Tunstel
National GeographicIllustration by Jason Treat, NGM Staff, and Dylan ColeSources: Robert Braun, Georgia Institute of Technology; NASA/JPL
45 E. Tunstel
NASA
46 E. Tunstel
Achieving Next-Level robotic intelligence
Sharp focus on
§ Increased robustness
§Cognitive facilities
§More sophisticated behavior
§System-/colony-level situational awareness
47 E. Tunstel
Need for Basic Robotic-SE
§ Systems engineering (SE) for research products should serve to bridge the research-applications gap
§ Like systems engineering, robotics is inter-disciplinary; thus can benefit from structured approach SE offers
§ A more structured development stage, during or following fundamental research, will allow us to field robots and expect them to be reliable for extended periods
SE – Building robotic systems right
48 E. Tunstel
Some Robotic-SE Considerations
Robotics practice needs more attention to:§Requirements definition/analysis
Ø Customer needs and objectivesØ Requirements V&V (Does design meet requirements? Can it perform
the required mission?)
§Capability characterizationØ What are the capabilities?Ø How reliable/risky are they, and under what conditions?Ø Can the customer trust the system? (Big issue for autonomy!)Ø What specific investment(s) will make it better?
§SpecificationsØ What is the customer really getting? What’s on the spec sheet?Ø Is it supported by statistically significant test results?
49 E. Tunstel
More Robotic-SE Considerations
Robotic-systems development also needs:§Performance evaluation/benchmarking
Ø Metrics/measures of performance over life-cycleØ How does the system measure up to the state of the art?Ø Methods to evaluate performance to facilitate system comparisons
§Test planning/logisticsØ System tests in relevant operating environmentsØ Why is the chosen test environment relevant? To what degree?Ø Support equipment? Costs? Environment regulations?
§People-oriented collaborationØ Complex/large systems call for more personnel from different groups
or organizations (“people skills” – communication, social, etc)Ø Beyond the documents, people facilitate subsystem interface and
configuration control, their negotiation, and cross-system transparency
50 E. Tunstel
SoSE Implications / Challenges
§ Robots (like people) will fail...how to engineer the capacity to self-recover, recover with human assistance without re-programmingØ autonomicity, Ø conversational co-troubleshooting (explainable AI) Ø real-time re-directioning / re-purposing
§ How to evolve from moving data across interfaces to moving information, intelligence, and contextual dialog
§ How to construct design and concept exploration environments that enable non-experts to configure, use and re-use robotic systems as needed
SoSE – Right systems & Interactions
51 E. Tunstel
Robotics & Systems Foci in the IEEE SMC SocietySELECTED ROBOTICS & SYSTEMS TECHNICAL COMMITTEES
§ Autonomous Bionic Robotic Aircraft
§ Bio-mechatronics and Bio-robotics Systems
§ Brain-Machine Interface Systems
§ Computational Cybernetics
§ Cyber-Medical Systems
§ Intelligent Learning in Control Systems
§ Model-Based Systems Engineering
§ Robotics and Intelligent Sensing
§ Shared Control
§ Systems of Systems
§ Unmanned Maritime Systems Engineering
http://ieeesmc.org/
IEEE SMCis where it all
comes together!
Join Us!
52 E. Tunstel
Acknowledgments
Research and real-world applications discussed are based
on past work as PI or contributor, or owing to influence on
or exposure to work by fellow group members at NASA JPL
(Advanced Robotic Controls Group; Robotic Intelligence
Group, Mars rover project teams) and Johns Hopkins APL
(Intelligent Systems Center).
53
5353
Thank You!
Sunset as imaged by the Spirit rover from a hilltop on the surface of Mars
Q U E S T I O N S ?
Recommended