On IT Technologies

& Self-Driving Cars

Emil M. Petriu

School of Electrical Engineering and Computer ScienceUniversity of Ottawa

By ANJALI SINGHVI and KARL RUSSELL UPDATED July 12, 2016

“” After Joshua Brown, 40, of Canton, Ohio, was killed driving a Tesla Model S in the first fatality involving a self-driving car, questions have arisen about the safety of the car’s crash-avoidance Autopilot system. Tesla told Senate investigators that a “technical failure” of the automatic braking system played a role but maintains that Autopilot was not at fault.”

“ How the Accident Happened

The Tesla Model S crashed in northern Florida into a truck that was turning left in front of it. The Tesla then ran off the road, hitting a fence and a power pole before coming to a stop. “

NY Times:

Inside the Self-DrivingTesla Fatal Accidenthttp://www.nytimes.com/interactive/2016/07/01/business/inside-tesla-accident.html

“ How Tesla’s Technology Works

The Tesla uses a computer vision-based vehicle detection system, but according to the company, it is not intended to be used hands-free and parts of the system are unfinished. The accident may have happened in part

because the crash-avoidance system is designed to engage only when radar and

computer vision systems agree that there is an obstacle, according to an industry executive with direct knowledge of the system. “

Ford T : [engine & wheels] + [hand cranked to

start, brut_force_steering & breaks, manual horn, manual gear_box] + [human _diver]

Evolution of Car Technologies

[engine & wheels] + [starter, electronic_ignition, power_assited steering & breaks, automatic_gear_box] + [human _diver]

[engine & wheels] = [starter, electronic_ignition, power_assited_steering&breaks, automatic_gear_box, automatic_breaks, cruise_control, rear_cameras, collisin_avoidance_sensors, GPS] = [human _diver]

Autonomous car : [engine & wheels] + [starter, electronic ignition, power assisted steering & breaks, automatic gear box, automatic breaks, cruise control, all around cameras, collision avoidance_sensors, GPS] + [robotic_driver]

…

…

“An autonomous car (driverless car, self-driving car, robotic car) is a vehicle that is capable of sensing its environment and navigating without human input Autonomous cars can detect surroundings using a variety of techniques such as radar, lidar, GPS, odometry,

and computer vision. Advanced control systems interpret sensory information to identify appropriate navigation paths, as well as obstacles and relevant signage. Autonomous cars have control systems that are capable of analyzing sensory data to distinguish between different cars on the road, which is very useful in planning a path to the desired destination.”

According to https://en.wikipedia.org/wiki/Autonomous_car :

Impact of driverless cars | Driverless car market watchwww.driverless-future.com/?cat=26Aug 24, 2016 - In May BMW announced that they would have a self-driving car on the market within 5 years. Next came Uber, which acquired autonomous ...

Autonomous vs. automated “ Autonomous means having the power for self-governance. Many historical projects related to vehicle autonomy have in fact only been automated (made to be automatic) due to a heavy reliance on artificial hints in their environment, such as magnetic strips. Autonomous control implies good performance under significant uncertainties in the environment for extended periods of time and the ability to compensate for system failures without external intervention.[As can be seen from many projects mentioned, it is often suggested to extend the capabilities of an autonomous car by implementing communication

networks both in the immediate vicinity (for collision avoidance) and far away (for congestion management). By bringing in these outside influences in the decision process, some would no longer regard the car's behaviour or capabilities as autonomous; for example Wood et al. (2012) writes "This Article generally uses the term "autonomous," instead of the term "automated." The term "autonomous" was chosen because it is the term that is currently in more widespread use (and thus is more familiar to the general public). However, the latter term is arguably more accurate. "Automated" connotes control or

operation by a machine, while "autonomous" connotes acting alone or

independently. Most of the vehicle concepts (that we are currently aware of) have a person in the driver’s seat, utilize a communication connection to the Cloud or other vehicles, and do not independently select either destinations or routes for reaching them. Thus, the term "automated" would more accurately describe these vehicle concepts".

According to https://en.wikipedia.org/wiki/Autonomous_car#Classification :

Classification.

“A classification system based on six different levels (ranging from driver assistance to fully automated systems) was published in 2014 by Society of Automotive Engineers (SAE), an automotive standardisation body.[This classification system is based on the amount of driver intervention and attentiveness required, rather than the vehicle capabilities, although these are very closely related.SAE automated vehicle classifications:Level 0: Automated system has no vehicle control, but may issue warnings.Level 1: Driver must be ready to take control at any time. Automated system may include features such as Adaptive Cruise Control (ACC), Parking Assistance with automated steering, and Lane Keeping Assistance (LKA) Type II in any combination.Level 2: The driver is obliged to detect objects and events and respond if the automated system fails to respond properly. The automated system executes accelerating, braking, and steering. The automated system can deactivate immediately upon takeover by the driver.Level 3: Within known, limited environments (such as freeways), the driver can safely turn their attention away from driving tasks.Level 4: The automated system can control the vehicle in all but a few environments such as severe weather. The driver must enable the automated system only when it is safe to do so. When enabled, driver attention is not required.Level 5: Other than setting the destination and starting the system, no human intervention is required. The automatic system can drive to any location where it is legal to drive.In the United States, the National Highway Traffic Safety Administration (NHTSA) released in 2013 a formal classification system. The NHTSA abandoned this system when it adopted the SAE standard in September 2016.”

According to https://en.wikipedia.org/wiki/Autonomous_car#Classification :

* Tesla's Autopilot can be classified as somewhere between NHTSA levels 2 and 3. At this level, the car can act autonomously but requires the full attention of the driver, who must be prepared to take control at a moment's notice

Car Technologies (system integration):

[Mechanical & electric] => [Mechanical & Electric]

=> [Mechanical & Electric & Electronics]

=> [Mechanical & Electric & Electronics & Computer_HW]

=>[Mechanical & Electric & Electronics & Computer_HW+SW]

=> [Mechanical & Electric & Electronics & Computer_HW+SW & AI]

Electrical, Computer, Software

Engineering Design & Risk Problems

“Engineering design integrates mathematics, natural sciences, engineering sciences, and complementary studies in order to develop elements, systems, and processes to meet specific needs. It is a creative, iterative, and open-ended process, subject to constraints which may be governed by standards or legislation to varying degrees depending upon the discipline. These constraints may also relate to economic, health, safety, environmental, societal or other interdisciplinary factors.

[CEAB – Accreditation Criteria and Procedures, sections 3.4.4.3 and

3.4.4.4]

“Digital electronics or digital (electronic) circuits

are electronics that handle digital signals – discrete bands of analog levels – rather than by continuous ranges as used in analog electronics. All levels within a band of values represent the same information state. Because of this discretization, relatively small changes to the analog signal levels due to manufacturing tolerance, signal attenuation or noise do not leave the discrete envelope, and as a result are ignored by signal state sensing circuitry.”…..

Digital Electronics and Computer

According to https://en.wikipedia.org/wiki/Digital_electronics

“ Properties

An advantage of digital circuits when compared to analog circuits is that

signals represented digitally can be transmitted without degradation due

to noise. For example, a continuous audio signal transmitted as a sequence of 1s and 0s, can be reconstructed without error, provided the noise picked up in transmission is not enough to prevent identification of the 1s and 0s. … In a digital system, a more precise representation of a signal can be obtained by using more binary digits to represent it. While this requires more digital circuits to process the signals, each digit is handled by the same kind of hardware, resulting in an easily scalable system. In an analog system, additional resolution requires fundamental improvements in the linearity and noise characteristics of each step of the signal chain.

Computer-controlled digital systems can be controlled by software, allowing new functions to be added without changing hardware. Often this can be done outside of the factory by updating the product's software. So, the product's design errors can be corrected after the product is in a customer's hands.”

Digital Electronics and Computer (…continued)

According to https://en.wikipedia.org/wiki/Digital_electronics

“Properties (continued)…. The noise-immunity of digital systems permits data to be stored and retrieved without

degradation. In an analog system, noise from aging and wear degrade the information stored. In a digital system, as long as the total noise is below a certain level, the information can be recovered perfectly.Even when more significant noise is present, the use of redundancy permits the recovery of the

original data provided too many errors do not occur. …Most useful digital systems must translate from continuous analog signals to discrete digital signals. This causes quantization errors. Quantization error can be reduced if the system stores enough digital data to represent the signal to the desired degree of fidelity. The Nyquist-Shannon sampling theorem provides an important guideline as to how much digital data is needed to accurately portray a given analog signal.In some systems, if a single piece of digital data is lost or misinterpreted, the meaning of large blocks of related data can completely change. … Digital fragility can be reduced by designing a digital

system for robustness. For example, a parity bit or other error management method can be inserted into the signal path. These schemes help the system detect errors, and then either correct the errors, or at least ask for a new copy of the data. In a state-machine, the state transition logic can be designed to catch unused states and trigger a reset sequence or other error recovery routine.Digital memory and transmission systems can use techniques such as error detection and correction to use additional data to correct any errors in transmission and storage.”

Digital Electronics and Computer (…continued)

According to https://en.wikipedia.org/wiki/Digital_electronics

S.R. Das, C.V. Ramamoorthy, M.H. Assaf, E.M. Petriu, W.-B. Jone, “Fault Tolerance in Systems Design in VLSI using Data Compression under Constraints of Failure Probabilities,” IEEE Tr. Instrum. Meas., Vol. 50, No. 6, pp.1725 –1747, 2001 [IEEE Donald G. Fink Prize Paper Award, 2003].

12

Software

• Software is pervasive in our lives, in most systems surrounding us - we take it for granted!

• US $500 Billion world-wide in 1995

• This includes critical systems that affect our well-being and our lives

• Many reported stories of poor software engineering practices leading to catastrophes

• External Characteristics (of interest to stakeholders)– Usability– Efficiency– Reliability– Maintainability– Reusability

• Internal Characteristics (impact maintainability and reliability)– Comments– Code Complexity: Nesting depth, branches, complex programming– Modularity

13

Software (… continued)

• Patient Protection and Affordable Care Act (a.k.a. ObamaCare, 2013):

– Incorrect functionality, unable to handle the load due to poor specifications, flawed design, poor coding, poor testing, security flaws, time constraints.

• Soyuz spacecraft’s descent from the ISS on May 3rd 2003

– Halfway back to Earth, for no apparent reason, the computer had suddenly begun searching for the ISS as if to dock with it.

• Ariane 5 Flight 501:

– The space rocket was destroyed. Cause: poor specifications, usage testing, and exception handling.

• Therac-25:

– Radiation therapy and X-ray machine killed several patients. Cause: unanticipated, non-standard user inputs.

• NASA mission to Mars (Mars Climate Orbiter Spacecraft, 1999):

– Incorrect conversion from imperial�metric leads to loss of Mars satellite.

– Therac-20:

• independent protective circuits for monitoring electron-beam scanning

• mechanical interlocks for policing machine and ensuring safe operation

– Therac-25:

• AECL decided to rely on computer to control similar features (…. save $$)

���� 6 incidents of massive overdoses (June 1985 – Jan. 1987)

resulting in 3 deaths

– 500 rads (radiation absorbed dose) = whole-body radiation causing death in 50% of cases

– one patient estimated to have received 1-2 doses (localized area) ranging betw. 15 000-20 000 rads

------------------

– Reused design features & modules from Therac-6 for Therac-25 (!!)

– Software developed by 1 person; very little software documentation produced during development

– Similar problems later detected with Therac-20 software, but hardware safety interlocks prevented accidents and injuries.

Therac-25 [Leveson, N. and Turner, C.S. (1993) Computer 26 (7): 18-41]

Environment Sensing and Perception

Context-Based

Rules of

Assessment

Environment{SOI(j) / j=1,2,… }SOI = Subject Of Interest

Distributed Sensor Data-Collection Network

Environment-Aware Multimodal Information Extraction =>Parameterization of the SOI’s Structural and Behavioral Features of Interest, as

e.g Formal Language (FL) Vocabulary Descriptors

Environment and Context-Aware Scalable System for

Situation Assessment

{SA(l) / l=1,2, …} SA =Exteroceptos

Robotic Car (actuators & proprioceptors &

INTELLIGENT CONTROL)

Context-Aware Multimodal Knowledge Extraction and Representation =>

Scalable World Model (e.g. represented in FL terms)

Interdependency & Behavioral Models

FL Grammar

Fuzzy Cognitive Maps

Environment-Aware Multi-Sensor Data Fusion

Robotics = Intelligent Connection of the Perception to Action

The multi-sensor fusion system has to (i) organize data collection and signal processing from different types of

sensor, (ii) produce local and global representations using the multi-sensor

information, and (iii) integrate the information from the different sensors into a

continuously updated model of the monitored system.

The specific objectives of the proposed research program are: * Development of the Mission-Critical Multi-sensor Data Fusion

Architecture* Development of Multi-Sensor Fusion Algorithms for Context-Aware

Decision Support

A multi-sensor fusion framework is needed to manage in a consistent way the usage of multiple sensor resources, while supporting user’s trust, workload, attention and situation awareness.

The multi-sensor fusion architecture may be based on the mission-critical JDL Data Fusion Model developed by the Joint Directors ofLaboratories Data Fusion Group .

This architecture has five functional levels.* level 0 “Signal/Feature Assessment” and level 1 “Entity Assessment”essentially asses the measurement data;* level 2 “Situation Assessment”;* level 3 “Impact Assessment” essentially asses the informationrecovered from data;* level 4 “Performance Assessment” provides sensor managementfunctions for process refinement.* a supplementary knowledge-management level 5 “User Refinement”is used delineate the human from the computer in the processrefinement and allow for the adaptive decision of who can query andrespectively access the information and the collected data in order tosupport cognitive decision support and actions.

Soft Computing AI

In Isaac Asimov's science-fiction short story, "Sally" (1953), autonomous

cars have "positronic brains" and communicate via honking horns and slamming doors, and save their human caretaker.

Intelligent control does not use classic analytic model-based techniques, but soft computing-based adaptive and learning techniques

Looking for a model to prove that algebraic operations with analog variables can be performed by logic gates, Professor J. von

Neuman advanced in 1956 the idea of

representing analog variables by the mean

rate of random-pulse streams [J. von

Neuman, “Probabilistic logics and the

synthesis of reliable organisms from

unreliable components,” in Automata

Studies, (C.E. Shannon, Ed.), Princeton, NJ,

Princeton University Press, 1956].

BIO-INSPIRED NEURAL NETWORKS

Biological Neurons

Neurons are rather slow (10-3 s) when compared with the modern electronic circuits. ==> The brain is faster than an electronic computer because of its massively parallel structure. The brain has

approximately 1011 highly connected neurons

(approx. 104 connections per neuron).

Dendrites carry electrical signals in into the neuron body. The neuron body integrates and thresholds the incoming signals.The axon is a single long nerve fiber that carries the signal from the neuron body to other neurons. A synapse is the connection between dendrites of two neurons.

Memories are formed by the modification of the synaptic strengths which can change during the entire life of the neural systems.

Body

Axon

Dendrites

Synapse

Analog/Random-Pulse and Random-Pulse/Digital Conversion

• E.M. Petriu, K. Watanabe, T. Yeap, "Applications of Random-Pulse Machine Concept to Neural Network Design," IEEE Trans. Instrum. Meas., Vol. 45, No.2, pp.665-669, 1996, • E. Pop, E.M. Petriu, "Influence of Reference Domain Instability Upon the Precision of Random Reference Quantizer with Uniformly Distributed Auxiliary Source," Signal Processing (EURASIP), North Holland, Vol. 5, pp.87-96, 1983

Auto-associative memory NN architecture

P1, t1 P2, t2 P3, t3

Training setRecovery of 30% occluded patterns

30

P

30x1

30x30

n

30x1

a

30x1W

)*Hardlim ( P Wa =

Neural Network for Pattern Recognition

photo by Peter Thornton, uOttawa Gazette

Thank You !