A Quest for Human Robot Cohabitation Robot Cohabitation - Trive… · Prof Trivedi, UCSD Fujitsu World Tour 2017 2 Safe, Stress-free, Efficient, Enjoyable Driving Learning by Li Lo:

Prof Trivedi, UCSD Fujitsu World Tour 2017 1

Mohan M. Trivedi LISA: Laboratory for Intelligent and Safe Automobiles

University of California at San Diego http://cvrr.ucsd.edu/LISA

A Quest for Human Robot Cohabitation

Connecting AI Technologies to Real-World Needs Panel


Safe, Stress-free, Efficient, Enjoyable Driving

Learning by Li Lo: Looking-in and Looking-out

Understanding human behavior is essential for humans It may very well be essential for robotic vehicles as well Make humans and vehicles form a distributed cognitive system to accomplish mutually beneficial goals

How to make two intelligent systems, one robotic and another human

understand each other and collaborate?

LISA Publications http://cvrr.ucsd.edu/publications/index.html

Key Points: Humanizing Automated Vehicles

•  �Driving� = Environment + Vehicle + Driver •  Autonomous Driving in the “Real World “ => Humanized Robotic Systems •  Should be Proactive: highly reliable, safety-time critical operation

•  Situational Awareness and Intent Predication – needs Holistic Perception •  Holistic Perception:

- Looking-Out: Lanes, Vehicles, Pedestrians and “Surround” -  Looking-In: Occupant and “Driver” -  Looking In and Out: Attention, Intentions, Activities, “Awareness”

•  Key Challenges: - Multilevel Signal-> Semantics Representation and Analysis - Multisensory and Distributed Systems and Software - Communication – speed, reliability, fail safe - Human-Centric (Distributed) System Architectures - Robustness - Performance and Evaluation Metrics, Protocols, Standards

LISA Publications http://cvrr.ucsd.edu/publications/index.html


Humans in

vehicle cabin

Humans

in surround vehicles

Humans

around vehicle

Safe to deploy airbag? Distracted driver?

Noticed pedestrian?

Pedestrian intent?

Pedestrian trajectory? New traffic rules?

Distracted pedestrian?

My neighbor’s intent?

Acknowledge right of way?

Distracted neighbor?

Ready to take over? Hands on wheel?

“Humanizing” Intelligent Vehicles: LISA Research Agenda

Eshed Ohn-Bar, Mohan Trivedi, "Looking at Humans in the Age of Self-Driving and Highly Automated Vehicles", IEEE Transactions on Intelligent Vehicles, 2016.

Vehicle Environment Driver

Vision for Intelligent Vehicles: LISA-A Test bed 2013


Aim: The early detection of an intended maneuver using driver, vehicle, and surround information.

Looking In and Out: Lane Change Intent Prediction [Discovery Channel, 2013]

Doshi, Morris, Trivedi, IEEE Pervasive Computing 2011

•  When should the driver merge? •  Should he accelerate or decelerate? How much? •  Is it ok to change lanes? If yes Left/right? If so, how? •  Can an “Intelligent” system assist with this?

Predictive Driver Assistance


Robust, Reliable, Generalizable, Practical, Market ready in 2017!

Intelligent Merge- Break Assist, Attention Monitoring CES Week, San Francisco Media Event, Jan 2014

LISA-Audi Urban Intelligent Assist, 2011-2014

Vehicle Trajectory Learning Flowchart

B. Morris, M. Trivedi, "Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011


Panoramic Trajectory Analysis

M Kristoffersen, J. Dueholm, R. Satzoda, M. Trivedi, T. Moeslund, "Trajectories and Maneuvers of Surrounding Vehicles with Panoramic Camera Arrays,” IEEE Transactions Intelligent Vehicles, 2016

Proposed Approach – Integration of Regions Looking at Humans in Surrounding Vehicles ).

Surround Vehicle Behavior Analysis

Intent prediction

M Kristoffersen, J. Dueholm, R. Satzoda, M. Trivedi, T. Moeslund, "Trajectories and Maneuvers of Surrounding Vehicles with Panoramic Camera Arrays,” IEEE Transactions Intelligent Vehicles, 2016


Proposed Approach – Integration of Regions Looking at Humans Around the Vehicle ).

Pedestrian path prediction Pedestrian body pose and fine-grained classification

T. Gandhi, M. Trivedi, "Image Based Estimation of Pedestrian Orientation for Improving Path Prediction", IEEE Intelligent Vehicles Symposium, June 2008

Proposed Approach – Integration of Regions Looking at Humans Around the Vehicle: Detections ).

Akshay Rangesh , Mohan Trivedi, “Pedestrians and their Phones", IEEE ITSC 2016, IEEE T-IV 2017


Proposed Approach – Integration of Regions Looking at Humans inside the Vehicle ).

Head and hand cues integration Gaze zone classification

Secondary tasks Interactions and Activities

S. Martin, K. Yuen, M. Trivedi, "Vision for Intelligent Vehicles and Applications (VIVA): Face Detection and Head Pose Challenge," IEEE Intelligent Vehicles Symposium, 2016. N. Das, E. Ohn-Bar, M. Trivedi, "On Performance Evaluation of Driver Hand Detection Algorithms: Challenges, Dataset, and Metrics,” IEEE ITSC 2015.

Proposed Approach – Integration of Regions ). Privacy and Safety: Balancing the Scale

Sujitha Martin, Ashish Tawari and Mohan M. Trivedi, “Towards Privacy Protecting Safety Systems for Naturalistic Driving Videos,” IEEE Transactions Intelligent Transportation Systems, 2014.


LISA @ CES, Jan 2016 LISA SafeShield @ CES 2016

R. Satzoda, Sean Lee, Frankie Lu, M. Trivedi, "Snap-DAS: A Vision-based Driver Assistance System on a Snapdragon Embedded Platform,” IEEE Intelligent Vehicles Symposium, 2015.

Holistic Distributed Cognitive Systems Perspective: Learning from Naturalistic Driving Studies, Predictive, Attentive Systems

Long-Term Goals: Human cohabitation with intelligent robots

Open Issues: Fail-safe, Control transitions, Performance Metrics, standards, evaluations, multi-agents, cooperation, etc. etc.

Big Picture: Safe, Stress-free, Efficient, Enjoyable Driving

Four Point Summary

Thanks ! more information: [email protected] cvrr.ucsd.edu/lisa

Documents

A Quest for Human Robot Cohabitation Robot Cohabitation - Trive… · Prof Trivedi, UCSD Fujitsu World Tour 2017 2 Safe, Stress-free, Efficient, Enjoyable Driving Learning by Li Lo: