Verification and validation challenges for Connected ......Verification and validation challenges for Connected Autonomous Vehicles (CAVs) Verification Futures 2019 ... Scenario based

© HORIBA MIRA Ltd. 2019© HORIBA MIRA Ltd. 2019

Tim Edwards

Senior Consultant, CAV Technologies. HORIBA MIRA

Verification and validation challenges for Connected Autonomous Vehicles (CAVs)

Verification Futures 2019

13th June 2019. Reading, UK


What is an Autonomous Vehicle?

2

SAE J3016


What is a Connected Autonomous Vehicle (CAV)?

• Real-time information

• e.g. Rich mapsConnect

• Peer-to-peer communication & negotiation

• e.g. Traffic signal prioritisation

• e.g. C-ACC / platooning

Cooperate

• External management intelligence

• e.g. Remote automated parking

• e.g. Fleet supervision

Control

3

Requirements: Integrity, latency, availability

■ Different definitions have very different requirements and suit different technologies


Why develop CAVs?

4


Jekyll & Hyde

5

What is all this fuss

about CAV verification?

Don’t worry, we can

apply existing practices

from automotive and

other sectors

Ahh no… this is

unprecedented we need

completely new tools and

approaches


Challenges of autonomy

Complexity

Integrity

CAV

Diversity

6

• Complex distributed software

• High availability requirement

• Introduction of Machine Learning

• Wireless communications and external dependencies

• Scenario definition & parameterisation

• Road networks

• Weather

• National / international variations

• Other road users

• Acceptance criteria

• Test coverage

• Cost / time / recursion

• High severity failure modes

• Consideration of Safety of the

Intended Functionality (SOTIF)

• Misuse cases

• Maintenance & in-service support


Systems engineering lifecycle

7

Transport System

Requirements

Vehicle

Requirements

AD System

Requirements

AD System

Verification

Full Vehicle

Verification

Operational

System

Linking both sides

using advanced

verification tools

and DVPs


Scenario based testing

■ Scenarios are a mechanism for mapping

use cases or requirements to test cases

■ Operational Design Domain (ODD) bounds

the intended scope of the system operation

and informs the environmental

requirements

■ Functional decomposition is recommended

to constrain and target the scenarios

8

Requirements Scenarios TestCases

User

requirements

ODD

Behavioural requirements

Environmental requirements


Test distribution

9

Test Gaps

We can’t test all the requirements at this level:

■ Test equipment not capable

■ System integration level

Correlation

Some element of the system or environment was

approximated - we devise tests to carry out at later

stages to seek correlation

PG

HIL

SIL

Motivation:

(1) Test early (Reduce rework cost) (2) Test virtually (Reduce test cost)


Integrated CAV Test Eco-System

10

■ Public environments

■ Virtual environments

■ Controlled environments

■ Cyber-physical environments

SaM

UKCITE & Midlands

Future Mobility

TIC-IT

Off Highway Driving

City Circuit

Park-IT


Public environment

*N. Kalra, S. Paddock; “Driving to Safety. How Many Miles of Driving Would

It Take to Demonstrate Autonomous Vehicle Reliability?”, RAND (2016) 11

■ Capture real scenarios and failure modes

- Particularly for perception systems

■ Highly automated driving trials (SAE L4)

already live – under UK Code of Practice

■ Not a viable means for safety assessment

- Billions of miles to make a statistical

argument about system failure rates*


Virtual

12

■ Required to address the volume of test scenarios

■ Accuracy and realism of models can be a limiting factor

■ Different levels of abstraction for different test activities


Controlled environments

■ Characterise models, and demonstrate correlation of simulation results

■ Validate system integration and demonstrate functionality in controlled conditions

13


Proving ground developments (Park-IT and TIC-IT)

14

Urban, interurban, highway and controllability scenarios

© HORIBA MIRA Ltd. 2019© HORIBA MIRA Ltd. 2019 15

Proving ground test automation


Proving ground test automation


A hybrid approach: Vehicle-in-the-loop

17


Acknowledgements

18


Summary

■ V&V forms part of a robust, requirements driven, systems engineering methodology

■ Challenging to adequately define scenarios that reflect the real-world

- … and to define acceptable levels for test coverage

■ All test activities have limitations and dependencies, which need to be addressed collectively

- Simulation is critical - only practical way to address scale - but still just one piece of the puzzle

- Development of test tools, assets, and facilities to exercise CAV technologies at component,

system and vehicle level – in simulation, laboratory and proving ground conditions

■ Here to engage with experts in other sectors and learn from previous experiences


HORIBA MIRA Ltd.

Watling Street,

Nuneaton, Warwickshire,

CV10 0TU, UK

T: +44 (0)24 7635 5000

F: +44 (0)24 7635 8000

www.horiba-mira.com

Firstname SecondnameQualifications / Affiliations

Job Title

Direct T: +44 (0)24 7635 5xxx

M: +44 (0)7xxx xxxxxx

E: [email protected]

Contact Details

20

HORIBA MIRA Ltd.

Watling Street,

Nuneaton, Warwickshire,

CV10 0TU, UK

T: +44 (0)24 7635 5000

F: +44 (0)24 7635 8000

www.horiba-mira.com

Tim EdwardsBEng (Hons), MPhil, CEng, MIET

Senior Consultant

Connected and Autonomous Vehicles

Direct T: +44 (0)24 7635 5484

M: +44 (0)7787 280164

E: [email protected]

Documents

Verification and validation challenges for Connected ......Verification and validation challenges for Connected Autonomous Vehicles (CAVs) Verification Futures 2019 ... Scenario based