ART OF AUTONOMOUS DRIVING VALIDATION...A game engine would give a near real-word environments, which can be used to create multiple scenarios. This ground truth data which is much

Art of Autonomous Driving Validation

[email protected] © Tata Elxsi 2019 1

ART OF AUTONOMOUS DRIVING VALIDATION

Chanjal Prakash, Tata Elxsi



TABLE OF CONTENTS

ABSTRACT ...................................................................................................................................................... 3

INTRODUCTION ............................................................................................................................................. 4

CHALLENGES IN IMPLEMENTING AUTONOMOUS DRIVING SOLUTIONS ..................................................... 7

THE NEED FOR PRECISE VISION..................................................................................................................... 9

CONCLUSION ............................................................................................................................................... 10

ABOUT TATA ELXSI ...................................................................................................................................... 11

REFERENCES ................................................................................................................................................ 12



ABSTRACT Autonomous driving planning and control algorithms have to ensure high levels of safety on all scenarios for which they are exposed to. Autonomous driving systems rely upon sensors and embedded software for localization, perception, motion planning and execution. Hence the authenticity is put on test by a large number of scenarios such as the non-deterministic behavior of traffic participants and by inaccurate sensors and actuators. Billions of miles of road testing would be required to validate the safety of the autonomous driving systems & its algorithms. This paper explains how these challenges are addressed by creating simulated environment using gaming technology. This method helps in generating photo realistic 3D simulation which matches the real camera images/video. Using this simulation platform, sensor simulation can be integrated into a closed-loop simulation environment that interacts with traffic simulators, enabling thousands of driving scenarios to be executed virtually. This not only helps in reducing the total effort and time required for realizing autonomous features but also ensures end-to-end safety in deep-learning-based autonomous driving systems.



INTRODUCTION

A study prepared by Strategy Analytics predicts that autonomous vehicles will generate a $7 trillion revenue stream by 2050 while saving an estimated 585,000 lives per decade and freeing more than 250 million hours of commuting time per year. The greatest problem that needs to be tackled before the introduction of Level 4 or Level 5 autonomous. There could be an endless number of scenarios that may be a hindrance to the vehicle such as any pedestrian, animal, vehicle or any other objects that could appear abruptly on the road, change of weather, lighting, haze, etc. which could obstruct the field of view of sensors.

Figure 1: Autonomous Driving Market Outlook Source: Frost & Sullivan

https://info.microsoft.com/rs/157-GQE-382/images/K24A-2018%20Frost%20%26%20Sullivan%20-%20Global%20Autonomous%20Driving%20Outlook.pdf



Currently, autonomous driving systems address these challenges with machine learning/deep learning software that can be trained to recognize patterns without having to be specifically programmed for every possible situation that could arise. Since, automotive manufacturers are developing advanced driver assisted system, because of the safety and reliability offered by autonomous driving technology. This would ultimately leads to fully autonomous driving.

Figure 2: Levels of autonomous driving : Source Birmingham

Taking the driver out of the control loop creates major challenges in validating the capabilities of the sensors that are now responsible for the perception of the driving environment. As the complexity of driver assistance systems increases, so does the required effort for testing and evaluation. Exhaustive physical vehicle tests for validation would be economically infeasible.

Advanced Driver Assisted System (ADAS) algorithms, currently are validated using the software V-cycle

defined by ISO 26262. The autonomous driving needs closed-loop simulation where the virtual vehicle is

driven in a realistic virtual world by real autonomous vehicle algorithms. Creating simulation environment

of autonomous vehicle in a very detailed manner impose enormous challenge that requires ultimate

precision over various elements:

The characteristics, state, behaviors and mechanical properties of the vehicle

Obstacles and other vehicles

Road and its various conditions

The environmental conditions, visibility, properties of the material, climatic conditions and road

signage

https://www.birmingham.ac.uk/news/thebirminghambrief/items/2016/11/driving-the-revolution.aspx



EASE OF VALIDATIONON A SIMULATED ENVIRONMENT

Figure 3: Challenges of testing in simulation or the physical world. Testing in simulation requires determining

In-short, validating algorithms in a simulated environment could limit the cost of autonomous driving

systems. Major challenges that may arise for test planning and control systems are:

Need to identify all the relevant scenarios to be tested

Several millions of test kilometers need to be driven

Issues need to be reproduced to solve them

Fixed issues need repetitive test, which increases the expense



CHALLENGES IN IMPLEMENTING AUTONOMOUS

DRIVING SOLUTIONS In real-world validation and testing of autonomous driving solutions, the test vehicle with the AD

algorithm is driven through various scenarios and conditions in the real world, for long distances and

durations, in order to collect data for training and testing the AD algorithm as well as to analyze its

performance. This approach has drawbacks to it, which are described below.

Artificial Intelligence-based autonomous driving algorithms in these vehicles, need to drive

through millions of kilometers, even then the volume of scenarios to the vehicle needs to be tested

is not practical to create.

These scenarios must include traffic conditions of various intensity, different road structure and

surface types, dynamic weather phenomenon, variations in temperature and lighting and the day-

night cycle. The creation of such scenarios in a real vehicle in the real world would be expensive

and time-consuming.

The test vehicle should have a dynamic interaction with various other actors and events, like road

accidents, road conditions, construction works, pedestrians and animals etc. Creating these

scenarios in real-world is enormously challenging.

Since the autonomous vehicle is exposed to real-world without actual driver, there is a huge

amount of health and safety risk which needs to be considered. Figure 3 shows the frequency of

disengagements in AD test vehicles and hence the risk they pose during real-world testing.



Figure 4 Number of AD disengagements in vehicles tested in California from Dec 2017 to Nov 2018

The creation of edge conditions where the AD vehicle is about to collide with a pedestrian or with

another closed-loop would have obvious risks. If Ad technology were to fail or disengage, it could

lead to catastrophic outcomes.

The faults obtained during the validation of autonomous vehicles need to be reproduced and after

fixing the fault, the same scenario validation has to be repeated. This is a direct increase in the

effort and time required for validation.

To train and validate scenarios in a fleet of the autonomous vehicle, a huge number of scenarios

need to be created and these have to be repeated multiple times, which is economically infeasible.

Figure 4 compares the miles per disengagement for AD test vehicles in California during the period

from December 2017 to November 2018. This comparison along with the number of

disengagements from Figure 3 provides us with an estimate of the amount of distance that was

covered for the testing of these vehicles.

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300



Figure 5. Miles per disengagement for AD vehicles tested in California from Dec 2017 to Nov 2018

THE NEED FOR PRECISE VISION There are many development steps involved for performing simulated tests in the automotive industry.

Early tests are performed as SIL (Software In the Loop)-tests. SIL-tests represent the environment of the

tested software as a software simulation. This allows early dynamic testing of the software component.

HIL (Hardware In the Loop)-tests add more realism by executing parts of the system on real hardware.

A more practical solution to Autonomous Driving validation and testing would be to train, test and validate

these algorithms in a virtual vehicle in a simulated environment. This will enable the creation of test

scenarios to be created in a controlled virtual world, which would otherwise be impossible to recreate in

the real world.

A game engine would give a near real-word environments, which can be used to create multiple scenarios.

This ground truth data which is much more reliable, can be used in a closed loop validation of autonomous

driving algorithms in which a virtual vehicle can be exposed to.

This closed-loop simulation approach could generate virtual world model and driving scenarios, physically

accurate sensor models, autonomous driving algorithms and vehicle dynamics. This allows virtual testing

in HiL, SiL, ViL, PiL, MiL and DiL applications.



CONCLUSION Autonomous driving which is mostly depended upon artificial intelligence, needs to be tested rigorously well before it is exposed to the public. Validation of AD algorithms is not only proven to consume enormous effort and time but also possess health and safety risk elements. This is due to various elements such as huge amount of data required, scenarios which need to be created & recreated in the real-world, rare case generation etc. These challenges may be overcome by making use of a game engine technology to create near real world scenarios and vehicle dynamics. This enables rendering of 3D simulation, which helps attain higher efficiency and accuracy. Sensor models and vehicle dynamics, which is simulated would reduce the real world risk to minimal. This results in an accelerated, economical, efficient, accurate and risk-free solution to AD testing and validation The autonomous driving validation suite developed by Tata Elxsi, helps to create simulated environment and vehicle dynamics. It helps OEM’s and Tier 1s in saving effort and time for AV testing with paramount precision. The solution has the ability to automatically create scenarios based on a set of rules or parameters defined by the user. The tool does so using a generation algorithm, which is fine-tuned so that each generated scenario is logical, realistic and unique. The solution facilitates realistic and ideal sensor modeling of multiple sensors types – Ultrasonic, RADAR, LIDAR, Camera, GPS, IMU and Wheel-tick.



ABOUT TATA ELXSI

Tata Elxsi is a global design and technology services company, headquartered in Bangalore. It addresses the automotive, broadcast and communications, consumer electronics and healthcare industries. This is supported by a network of design studios, development centers, and offices worldwide.

Tata Elxsi works with leading OEMs and suppliers in the automotive and transportation industries for R&D, design and product engineering services from architecture to launch and beyond. It brings together domain experience across Infotainment, Autonomous Driving, Telematics, Powertrain, and Body electronics, along with technologies such as Artificial Intelligence, Analytics, Cloud and IoT.

Tata Elxsi has been investing in research and development for Advance Driver Assisted Systems and

Autonomous Driving for over a decade and have developed multiple solutions which accelerates the

realization of autonomous driving system.



REFERENCES

1. Accelerating the Future: The Economic Impact of the Emerging Passenger Economy, Roger

Lanctot, June 2017

2. The Last Driving License Holder, Mario Herger, December 2019

3. Will autonomous cars change the role and value of public transportation?, Freemark, 2017

The Transport Politic

4. Autonomous Driving, McKinsey Center of Future mobility, 2019

5. An Efficient Method for Testing Autonomous Driving Software Against Nondeterministic

Influences, Pascal Marcel Minnerup, May 2017

6. Automotive Software Engineering, 5th ed. Wiesbaden: Springer Fachmedien, J. Schauffele and T.

Zurawka, 2013

Documents

ART OF AUTONOMOUS DRIVING VALIDATION...A game engine would give a near real-word environments, which can be used to create multiple scenarios. This ground truth data which is much