Thesis - publications.lib.chalmers.sepublications.lib.chalmers.se/records/fulltext/193048/193048.pdf · training ob ject classi ers. ords: Keyw e, Automotiv e ctiv A, y Safet Semi-Autonomous

Thesis for the Degree of Do tor of Philosophy

Computational Veri� ation Methods

for Automotive Safety Systems

Jonas Nilsson

Department of Signals and Systems

Chalmers University of Te hnology

Göteborg, Sweden 2014

Computational Veri� ation Methods for Automotive Safety Systems

Jonas Nilsson

ISBN 978-91-7385-973-8

© Jonas Nilsson, 2014.

Doktorsavhandlingar vid Chalmers tekniska högskola

Ny serie nr 3654

ISSN 0346-718X

Department of Signals and Systems

Me hatroni s Group

Chalmers University of Te hnology

SE�412 96 Göteborg

Sweden

Telephone: +46 (0)31 � 772 1000

Jonas Nilsson

Telephone: +46 (0)70 � 266 5397

Email: jonas.nilsson�volvo ars. om

jonas.nilsson� halmers.se

Typeset by the author using L

A

T

E

X.

Chalmers Reproservi e

Göteborg, Sweden 2014

to Moni a

Abstra t

This thesis onsiders omputational methods for analysis and veri� ation

of the lass of automotive safety systems whi h support the driver by mon-

itoring the vehi le and its surroundings, identifying hazardous situations

and a tively intervening to prevent or mitigate onsequen es of a idents.

Veri� ation of these systems poses a major hallenge, sin e system de isions

are based on remote sensing of the surrounding environment and in orre t

de isions are only rarely a epted by the driver. Thus, the system must

make orre t de isions, in a wide variety of tra� s enarios. There are two

main ontributions of this thesis. First, theoreti al analysis and veri� ation

methods are presented whi h investigate in what s enarios, and for what

sensor errors, the absen e of in orre t system de isions may be guaranteed.

Furthermore, methods are proposed for analyzing the frequen y of in or-

re t de isions, in luding the sensitivity to sensor errors, using experimental

data. The se ond major ontribution is a novel omputational framework

for determining the errors of mobile omputer vision systems, whi h is one

of the most widely used sensor te hnologies in automotive safety systems.

Augmented photo-realisti images, generated by rendering virtual obje ts

onto a real image ba kground, are used as input to the omputer vision

system to be tested. Sin e the obje ts are virtual, ground truth is readily

available and varying the image ontent by adding di�erent virtual obje ts

is straightforward, making the proposed framework �exible and e� ient.

The framework is used for both performan e evaluation and for training

obje t lassi�ers.

Keywords: Automotive, A tive Safety, Semi-Autonomous Vehi les, Veri-

� ation, Performan e Evaluation, De ision Making, Augmented Reality.

i

ii

A knowledgments

Writing a Ph.D. thesis is a bit like hiking in the mountains. The view is

beautiful and inspiring but as one strives to the top, there is some real

physi al pain involved. You pin your hope on the summit resting right

behind the next rest, only to �nd yet another rest. Anyhow, for a person

appre iating ardiovas ular exer ise, the journey has been truly enjoyable.

Many people have a ompanied me on this journey and deserve my deep-

est gratitude. My a ademi supervisor, Dr. Jonas Fredriksson, has always

been available for dis ussions (often on topi s related to ross- ountry ski-

ing), and has helped me not to loose sight of the bigger pi ture. Thank you

for your ommitment. My industrial supervisor, Dr. Anders Ödblom, has

ontinuously supported me using his impe able eye for details. For this I

thank you. Together I think we have �nally understood what this proje t

is really about. Thank you also to Prof. Jonas Sjöberg for wel oming me

into his resear h group and o asionally allowing me to beat him in a ra e.

From the many olleagues at Volvo Cars and Chalmers ontributing to

a stimulating work environment, I would like to mention a few. My urrent

and former managers, Georgios Minos and Peter Janevik respe tively, for

giving me the freedom needed to work e� iently in between a ademia and

industry. Dr. Mohammad Ali, for fruitful ollaboration and enri hing dis-

ussions. The room got awfully quiet without you. Co-authors Adeel Zafar,

Patrik Andersson and Prof. Irene Gu, I hope we an ollaborate again in

the future. Dr. Erik Coelingh, Dr. Mattias Brännström and Jonas Ekmark,

for indire t ontributions through your on�den e and support. Dr. Fredrik

Persson, Feng Liu and David Hultberg, for your ommitted work on VCAV.

Thank you also to Vinnova, SAFER and Volvo Cars for funding this proje t.

Last but most important I thank my family, starting with my siblings

Fredrik and Pernilla for proofreading this thesis. Moni a, for your everlast-

ing support and your omplementing talents, bringing balan e to our family.

Ludvig and Valdemar, for showing me there is joy in every detail in life and

for being inexhaustible sour es of ideas on new forms of transportation.

Jonas Nilsson

Göteborg, January 2014

iii

iv

Contents

Abstra t i

A knowledgments iii

Contents v

I Introdu tory Chapters

1 Introdu tion 1

1.1 Aims and Obje tives . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.4 List of Publi ations . . . . . . . . . . . . . . . . . . . . . . . 4

2 Automotive Safety Systems 7

2.1 Tra� A ident Causation . . . . . . . . . . . . . . . . . . . 8

2.2 Vehi le Dynami s Control . . . . . . . . . . . . . . . . . . . 10

2.3 Driver Assistan e . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3.1 Sensor Te hnology . . . . . . . . . . . . . . . . . . . 13

2.3.2 De ision-Making and Interventions . . . . . . . . . . 14

2.4 Autonomous Driving . . . . . . . . . . . . . . . . . . . . . . 15

2.5 System E�e tiveness . . . . . . . . . . . . . . . . . . . . . . 16

2.6 Veri� ation Challenges . . . . . . . . . . . . . . . . . . . . . 18

3 Veri� ation Methods 21

3.1 Performan e Metri s . . . . . . . . . . . . . . . . . . . . . . 21

3.2 Method Properties . . . . . . . . . . . . . . . . . . . . . . . 22

3.3 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.3.1 System Models . . . . . . . . . . . . . . . . . . . . . 25

3.3.2 Pro ess Models . . . . . . . . . . . . . . . . . . . . . 27

3.4 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.4.1 Real Driving . . . . . . . . . . . . . . . . . . . . . . . 28

v

Contents

3.4.2 Closed-Loop Simulations . . . . . . . . . . . . . . . . 30

3.4.3 Data Replay . . . . . . . . . . . . . . . . . . . . . . . 30

3.4.4 Theoreti al Methods . . . . . . . . . . . . . . . . . . 31

3.5 Method Comparison . . . . . . . . . . . . . . . . . . . . . . 31

4 Summary of In luded Papers 35

5 Con luding Remarks 41

5.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.2 Dire tions of Future Resear h . . . . . . . . . . . . . . . . . 42

Referen es 45

II In luded papers

Paper 1 Worst Case Analysis of Automotive Collision Avoid-

an e Systems 55

1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 57

2.1 Collision Avoidan e Systems . . . . . . . . . . . . . . 57

2.2 Performan e Evaluation of Collision Avoidan e Systems 58

3 De�nitions and Problem Formulation . . . . . . . . . . . . . 59

3.1 General CA System Des ription . . . . . . . . . . . . 59

3.2 Problem Formulation . . . . . . . . . . . . . . . . . . 59

4 System Des ription . . . . . . . . . . . . . . . . . . . . . . . 61

5 S enario Des ription . . . . . . . . . . . . . . . . . . . . . . 65

6 Worst Case Performan e . . . . . . . . . . . . . . . . . . . . 67

6.1 Desired Time of Intervention . . . . . . . . . . . . . . 68

6.2 Time of Intervention . . . . . . . . . . . . . . . . . . 69

6.3 Maximum Predi tion Errors . . . . . . . . . . . . . . 70

6.4 Robust Avoidan e S enarios . . . . . . . . . . . . . . 71

7 Numeri al Results . . . . . . . . . . . . . . . . . . . . . . . . 72

7.1 Longitudinal S enarios . . . . . . . . . . . . . . . . . 73

7.2 Lateral S enarios . . . . . . . . . . . . . . . . . . . . 78

7.3 Robust Avoidan e S enarios . . . . . . . . . . . . . . 81

8 Con luding Remarks . . . . . . . . . . . . . . . . . . . . . . 84

Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Paper 2 Veri� ation of Collision Avoidan e Systems using

Rea hability Analysis 91

1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

vi

Contents

2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 92

3 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . 93

4 Problem Formulation and Proposed Approa h . . . . . . . . 96

5 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

5.1 Coordinate Systems . . . . . . . . . . . . . . . . . . . 98

5.2 Relative Motion . . . . . . . . . . . . . . . . . . . . . 99

5.3 Threat Assessment . . . . . . . . . . . . . . . . . . . 99

6 Veri� ation . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

6.1 Admissible Set . . . . . . . . . . . . . . . . . . . . . 101

6.2 Computing the Safe and Unsafe Sets . . . . . . . . . 101

6.3 Veri� ation . . . . . . . . . . . . . . . . . . . . . . . 101

6.4 Error Robustness . . . . . . . . . . . . . . . . . . . . 102

7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102


Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Paper 3 Sensitivity Analysis and Tuning for A tive Safety

Systems 109

1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

2 De�nitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

3 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

3.1 Performan e Evaluation . . . . . . . . . . . . . . . . 115

3.2 Input Requirements . . . . . . . . . . . . . . . . . . . 119

3.3 Fun tion Tuning . . . . . . . . . . . . . . . . . . . . 120

4 Appli ation Example . . . . . . . . . . . . . . . . . . . . . . 120

5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . 121

5.1 Input Requirements . . . . . . . . . . . . . . . . . . . 122

5.2 Fun tion Tuning . . . . . . . . . . . . . . . . . . . . 124

6 Con lusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Paper 4 Performan e Evaluation Method for Mobile Com-

puter Vision Systems using Augmented Reality 131

1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

2 Performan e Evaluation Framework . . . . . . . . . . . . . . 133

3 Case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

3.1 System Des ription . . . . . . . . . . . . . . . . . . . 136

3.2 S enario Des ription . . . . . . . . . . . . . . . . . . 136

3.3 Implementation . . . . . . . . . . . . . . . . . . . . . 136

3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 138

4 Dis ussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

4.1 Dis ussion on Method . . . . . . . . . . . . . . . . . 139

vii

Contents

4.2 Dis ussion on Case Study Results . . . . . . . . . . . 140

5 Con lusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Paper 5 Reliable Vehi le Pose Estimation using Vision and

Single-Tra k Model 147

1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 149

2.1 Visual Pose Estimation . . . . . . . . . . . . . . . . . 149

2.2 Ground Vehi le Pose Estimation . . . . . . . . . . . . 150

3 Vehi le and Camera Modeling . . . . . . . . . . . . . . . . . 151

3.1 Coordinate Systems . . . . . . . . . . . . . . . . . . . 152

3.2 Pinhole Camera . . . . . . . . . . . . . . . . . . . . . 154

3.3 Lo al Vehi le Motion . . . . . . . . . . . . . . . . . . 155

4 Bundle Adjustment Framework . . . . . . . . . . . . . . . . 157

4.1 Parametrization . . . . . . . . . . . . . . . . . . . . . 158

4.2 Error Modeling . . . . . . . . . . . . . . . . . . . . . 158

5 Re onstru tion Algorithm . . . . . . . . . . . . . . . . . . . 160

5.1 Image Feature Extra tion and Mat hing . . . . . . . 161

5.2 In remental Estimation of Initial Pose . . . . . . . . 161

5.3 Lo al Bundle Adjustment . . . . . . . . . . . . . . . 162

6 Experimental Validation Method . . . . . . . . . . . . . . . 162

6.1 Data Colle tion and Des ription . . . . . . . . . . . . 163

6.2 Performan e Metri s . . . . . . . . . . . . . . . . . . 165


7.1 Bundle Adjustment Performan e . . . . . . . . . . . 167

7.2 Sensitivity to Initial Estimate . . . . . . . . . . . . . 174

7.3 Sensitivity to Model Parameters . . . . . . . . . . . . 177

7.4 Moving Obje ts . . . . . . . . . . . . . . . . . . . . . 180

8 Con lusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

Paper 6 Pedestrian Dete tion using Augmented Training Data189

1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 190

3 Pedestrian Dete tion . . . . . . . . . . . . . . . . . . . . . . 191

3.1 Histogram of Oriented Gradients . . . . . . . . . . . 192

3.2 Support Ve tor Ma hine . . . . . . . . . . . . . . . . 192

3.3 Sliding Window Dete tion . . . . . . . . . . . . . . . 193

4 Augmented Data Generation . . . . . . . . . . . . . . . . . . 193

5 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . 194

5.1 Real Training Data . . . . . . . . . . . . . . . . . . . 196

viii

Contents

5.2 Real Test Data . . . . . . . . . . . . . . . . . . . . . 197

5.3 Augmented Training Data . . . . . . . . . . . . . . . 197



Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

ix

x

Part I

Introdu tory Chapters

Chapter 1

Introdu tion

Road tra� a idents are a global problem of epidemi proportions. A -

ording to the World Health Organization (WHO), road tra� injuries are

the leading ause of death globally for young people aged 15 − 29, and

the eight leading ause of death in total, [1℄. In the developed ountries

primarily, road tra� a idents have been on the agenda in the past few

de ades. Governments have invested in infrastru ture and passed laws to

improve road safety. The automotive industry has put emphasis on design-

ing systems that prote t the o upants of the vehi le in ase of a rash, so

alled passive safety systems. Passive safety innovations in lude seat belts,

rumple zones and airbags.

In the 1970s, the introdu tion of Anti-lo k Braking Systems (ABS)

marked a �rst milestone for a tive safety systems, i.e. systems whi h a -

tively intervene to prevent or mitigate onsequen es of a idents. In re ent

years, a tive safety systems whi h monitor the surrounding environment,

using remote sensing te hnologies, have been introdu ed to the market.

By using information on the surrounding tra� environment, systems an

identify hazardous situations, e.g. when the driver has failed to observe a

rossing pedestrian and a ollision is imminent. If and when hazardous

situations are dete ted, the system an a tively intervene to prevent an

a ident either by informing the driver of the up oming danger or by au-

tonomously performing an evasive maneuver su h as Automati Emergen y

Braking (AEB).

This thesis on erns the problem of verifying that a given a tive safety

system a ts orre tly in the wide variety of possible tra� s enarios. There

are two major reasons why this is a hallenging task. First, the variations

in operating onditions are essentially unlimited, a fa t easily a knowledged

when re�e ting and omparing a snowy ountry road in northern Sweden to

downtown Tokyo. Se ond, in orre t de isions by highly intrusive systems,

like AEB, an only be a epted on very rare o asions.

1

Chapter 1. Introdu tion

1.1 Aims and Obje tives

The aim of the work presented in this thesis is to develop omputational

methods for e� ient veri� ation of automotive safety systems. In this on-

text, omputational veri� ation methods are de�ned as methods whi h pre-

di t system performan e by performing omputations with re orded exper-

imental data and/or mathemati al models as input.

In a tive safety systems, de ision fun tions use input from sensors to

de ide how to appropriately support the driver. A vital part of a tive safety

system performan e is the ability to make orre t de isions, also in the

presen e of sensor measurement errors. Consequently, three obje tives are

formulated, namely to develop methods that

I. For a given a tive safety de ision fun tion, identify tra� s enarios

where the fun tion makes in orre t de isions

II. For a given a tive safety de ision fun tion, quantify the robustness to

input errors

III. Generate virtual sensor data with su� ient quality for analysis and

veri� ation

The �rst two obje tives are addressed by Papers 1-3, while the third obje -

tive is treated in Papers 4-6.

1.2 Delimitations

This thesis is on erned with semi-autonomous vehi les where a tive safety

systems monitor the tra� situation and intervene if needed to ensure safety.

Obje tives I and II are delimited to evaluating the orre tness of the inter-

vention de ision as opposed to the hoi e and exe ution of the intervention.

With regards to the same two obje tives, only tra� s enarios with single

moving obje ts are onsidered. In Obje tive II, we primarily onsider in-

put errors whi h are bounded and systemati , where systemati means that

they depend on the spe i� tra� situation. Obje tive III is on erned with

e� iently determining said input errors and is delimited to omputer vision

sensors, whi h is one of the dominating te hnologies used in a tive safety

appli ations.

1.3 Thesis Outline

The thesis is divided into two parts. Part I serves as an introdu tion to

Part II by presenting ba kground information and related work. Part II

2

1.3. Thesis Outline

ontains six s ienti� papers that onstitute the base of the thesis.

Part I provides ontext to the appended papers and is organized as fol-

lows. In Chapter 1, the topi of the thesis is introdu ed and aims, obje tives

and delimitations are des ribed. Chapter 2 gives an overview of in-vehi le

safety systems with a strong emphasis on a tive safety systems. In Chap-

ter 3, an overview of methods for system veri� ation is provided. Chapter 4

brie�y summarizes the papers in luded in Part II while Chapter 5 presents

the main s ienti� ontributions and gives suggestions for future resear h.

3

Chapter 1. Introdu tion

1.4 List of Publi ations

This thesis is based on the following publi ations:

Paper 1

J. Nilsson, A. Ödblom and J. Fredriksson, Worst Case Analy-

sis of Automotive Collision Avoidan e Systems, submitted for

possible journal publi ation.

Paper 2

J. Nilsson, J. Fredriksson and A. Ödblom, Veri� ation of Colli-

sion Avoidan e Systems using Rea hability Analysis, submitted

as invited paper to the 19th IFAC World Congress, Cape Town,

South Afri a, 2014.

Paper 3

J. Nilsson and M. Ali, Sensitivity Analysis and Tuning for A tive

Safety Systems, in Pro eedings of the 13th International IEEE

Conferen e on Intelligent Transportation Systems, 2010, pages

161-167, Madeira Island, Portugal.

Paper 4

J. Nilsson, A. Ödblom, J. Fredriksson, A. Zafar and F. Ahmed,

Performan e Evaluation Method for Mobile Computer Vision

Systems using Augmented Reality, in Pro eedings of the IEEE

Virtual Reality Conferen e, 2010, pages 19-22, Waltham, Mas-

sa husetts, USA.

Paper 5

J. Nilsson, J. Fredriksson and A. Ödblom, Reliable Vehi le Pose

Estimation using Vision and Single-Tra k Model, submitted for


Paper 6

J. Nilsson, P. Andersson, I. Gu and J. Fredriksson, Pedestrian

Dete tion using Augmented Training Data, submitted to the

22nd International Conferen e on Pattern Re ognition, Sto k-

holm, Sweden, 2014.

4

1.4. List of Publi ations

Other Publi ations

In addition to the publi ations above, the following publi ations by the

thesis author are related to the topi of this thesis:

J. Nilsson, J. Fredriksson, and A. Ödblom, Bundle Adjustment

using Single-Tra k Vehi le Model, in Pro eedings of the IEEE

International Conferen e on Roboti s and Automation, 2013, pp.

2888-2893.

J. Nilsson, A. Ödblom, J. Fredriksson, and A. Zafar, Using Aug-

mentation Te hniques for Performan e Evaluation in Automo-

tive Safety, in Handbook of Augmented Reality, 1st ed., B. Furht,

Ed. Springer, 2011, pp. 631-649.

J. Nilsson, On Performan e Evaluation of Automotive A tive

Safety Systems, Li entiate Thesis R014/2010, ISSN 1403-266X,

Chalmers University of Te hnology, Göteborg, Sweden, 2010.

J. Nilsson and A. Ödblom, On Worst Case Performan e of Col-

lision Avoidan e Systems, in Pro eedings of the IEEE Intelligent

Vehi les Symposium, 2010, pages 1084-1091, San Diego, Califor-

nia, USA.

A. Ödblom and J. Nilsson, Augmented Vision in Image Sequen e

Generated from a Moving Vehi le, Patent pending, EP2639771,

European Patent O� e, 2012.

J. Nilsson, Operating Method and System for Supporting Lane

Keeping of a Vehi le, Patent granted, US8428821, U.S. Patent

and Trademark O� e, 2007.

J. Nilsson, Operating Method and System for Supporting Lane

Keeping of a Vehi le, Patent granted, EP2188168, European

Patent O� e, 2007.

5

6

Chapter 2

Automotive Safety Systems

The over 1 million annual fatalities aused by road tra� a idents are

merely the tip of the i eberg, e.g. the WHO estimates that road tra�

a idents also lead to between 20 and 50 million non-fatal injuries ea h

year, [1℄. On top of that, the e onomi burden linked to road tra� a idents

is signi� ant. In 1998, a rude estimate of the annual global ost was found

to be in the order of US$500 billion, [2℄.

There are large regional di�eren es a ross the world as the variations

in vehi le safety, infrastru ture and driver edu ation are substantial. Re-

markable progress has been made in the developed ountries during the last

de ades, as an be seen in Figure 2.1. Improved vehi le design, road infras-

tru ture investments and road safety poli ies have ontributed to redu ing

the risk of getting killed in tra� , in most developed ountries, by more

than 40% sin e 1990, [3℄.

Su ess in redu ing fatalities has spurred stakeholders in road safety to

set more and more ambitious goals, as des ribed in [6℄. The most ambitious

goal possible, i.e. a vision of zero fatalities in road tra� , has been expressed

in road safety poli ies in Sweden and the Netherlands. The urrent and

future automotive safety systems dis ussed in this hapter have the potential

to ontribute signi� antly to this goal.

We ategorize automotive safety systems into passive safety systems,

whi h prote t the vehi le o upants when ollision has o urred, and three

types of a tive safety systems, whi h are designed to prevent a idents.

The �rst ategory of a tive safety systems, vehi le dynami s ontrol sys-

tems, prevent unwanted dynami al behaviours su h as instability. Driver

Assistan e (DA) systems monitor the vehi le surroundings to assist the

driver. In a not too distant future, Autonomous Driving (AD) systems may

take omplete responsibility for the driving task. The line between these

ategories is by no means sharp, as exempli�ed by the Roadway Departure

Prevention Assist (RDPA) system des ribed in [7℄ whi h in orporate both

7

Chapter 2. Automotive Safety Systems

1970 1975 1980 1985 1990 1995 2000 2005 20100

5

10

15

20

25

30

Year

Fatalities/100

000Inhab

itan

ts

United States

United Kingdom

Germany

Netherlands

Sweden

Figure 2.1: Histori al road tra� fatalities, obtained from [4℄, for some of

the developed ountries. As a referen e, low- and middle-in ome ountries

have annual road tra� fatalities of 18.3 and 20.1 per 100 000 inhabitants

respe tively, [1℄. The sharp in rease in fatalities for Germany in 1990 is an

e�e t of the reuni� ation of Germany, [5℄.

stability ontrol and ollision avoidan e in a single framework.

In this hapter, the di�erent ategories of a tive safety systems are de-

s ribed, after �rst providing some ontext by brie�y dis ussing the auses

of a idents. In the �nal part of the hapter, the e�e tiveness of these sys-

tems is reviewed followed by a dis ussion on the hallenges asso iated with

system veri� ation, whi h is the ore problem addressed in this thesis.

2.1 Tra� A ident Causation

To e� iently prevent a idents, the auses of a idents need to be under-

stood. A ommon approa h for identifying a ident auses is to study a -

ident statisti s. Figure 2.2 shows the a ident distribution in terms of

major rash types, obtained from [8℄. There are numerous ways to lassify

a idents, e.g. by gender, age, type of vehi le, time of day or weather on-

ditions. Extensive reports with a ident lassi� ations based on national

a ident statisti s are published ontinuously, see e.g. [9℄ for the U.S. or [10℄

for Sweden.

Human error plays a major role in a majority of a idents. In an in depth

8

2.1. Traffi A ident Causation

Rear-End

28.4%

Crossing Paths

24.9%

Off Roadway

22.7%

Lane Change

9%

Animal

4%

Opposite Direction

2.6%

Backing

2.1%

Pedestrian

1.1%Pedalcyclist

0.8%

Other

4.4%

Figure 2.2: Distribution in terms of major rash types for all 6 394 000

poli e-reported motor vehi le rashes in the U.S. whi h resulted in

3 189 000 injured people and a total of 41 821 fatalities, [8℄. The �gure

is based on statisti s from the 2000 National Automotive Sampling System

(NASS)/General Estimates System (GES) rash database.

study of real a idents in the 1970s, [11℄, in luding on-s ene investigations,

it was on luded that human parti ipants were solely or partly to blame in

92.6% of the investigated a idents. The orresponding numbers for envi-

ronmental and vehi ular fa tors were 33.8% and 12.6% respe tively. Com-

mon human errors were e.g. ex essive speed, improper evasive a tion and

driver inattention or distra tion. Environmental fa tors were e.g. view ob-

stru tions and slippery road surfa es while vehi ular fa tors in luded brake

failures and inadequate tyre tread depth.

More re ently, in 2005, a Field Operational Test (FOT) known as the

100-Car Study, [12, 13℄, was ompleted. 100 ars were equipped with un-

obtrusive data olle tion instrumentation to olle t naturalisti data from

normal driving. The study rea�rms that drivers are often to blame for

a idents as nearly 80% of all rashes involved the driver looking away from

the forward roadway just prior to the ollision. Driver inattention or dis-

tra tion, e.g. using a mobile phone while driving, does not ne essarily lead

to an a ident but if oin iding with another unfortunate event, e.g. the

vehi le in front suddenly braking, the probability of an a ident in reases

signi� antly. Multiple a ident auses mean that there are multiple possible

preventive measures. As a idents are very diverse, preventing a majority of

9


a idents requires the deployment of a large number of preventive measures.

2.2 Vehi le Dynami s Control

Following advan es in ele troni s te hnology, mass produ tion of ABS sta-

rted on road vehi les in the 1970s but the innovation had been present in

the railway and aviation industries de ades before that. ABS monitors the

rotational speed of the wheels and automati ally redu e the brake for e if

the wheels ease to rotate, thus preventing brake lo k-up. This enables

steering of the vehi le while simultaneously braking hard.

In the 1990s, Ele troni Stability Control (ESC) was introdu ed to han-

dle problems with vehi le instability. ESC dete ts when the vehi le starts

to skid and ountera ts this by automati ally braking the wheels individ-

ually, as illustrated in Figure 2.3. A natural evolution of ESC is to also

prevent the vehi le from rolling over, as presented in [14℄. Roll Stability

Control (RSC) is mostly relevant for vehi les with high enter of gravity,

su h as Sport Utility Vehi les (SUVs) and tru ks, and was �rst introdu ed

in 2002, [15℄.

The interested reader is referred to e.g. [16,17℄, for more omprehensive

treatments of vehi le dynami s ontrol systems.

Figure 2.3: A vehi le drives onto an i e pat h in a urve. Without ESC

the vehi le be omes unstable and starts spinning. With ESC the left front

wheel is braked, thereby ountera ting the rotation, ensuring that stability

is maintained.

10

2.3. Driver Assistan e

2.3 Driver Assistan e

Re ent advan es in remote sensing te hnology have led to the introdu -

tion of several DA systems, see e.g [17, 18℄ for extensive overviews. One of

the �rst examples, laun hed in 1995, is an extension of the ruise ontrol

whi h automati ally maintains a onstant vehi le speed set by the driver.

Adaptive Cruise Control (ACC), thoroughly des ribed in [16,19℄, uses infor-

mation from a forward looking sensor, e.g. a radar, to maintain a onstant

distan e or time gap, set by the driver, to the vehi le in front of the host

vehi le, see Figure 2.4. ACC ontributes to safe driving by assuring that a

safe distan e is kept to the vehi le ahead. Also, ACC an redu e fuel on-

sumption and ongestion through smooth ontrol of the brakes and throttle,

thereby ontributing to a leaner environment.

Utilizing the same forward-looking sensor, Forward Collision Warning

(FCW) indi ates to the driver, as exempli�ed in Figure 2.5a, when im-

minent a tion is needed to avoid a ollision, e.g. when the vehi le ahead

suddenly brakes. If there is insu� ient time or if the driver fails to respond

to warnings, a Collision Avoidan e (CA) system an autonomously ontrol

the vehi le to avoid the impending ollision. A ommon a tion for CA sys-

tems is to automati ally apply the brakes in situations where a ollision

is imminent, so alled AEB, illustrated in Figure 2.5b. If the ollision is

unavoidable, AEB may still be triggered to redu e impa t speed, so alled

Collision Mitigation (CM).

There are also numerous DA systems whi h support the lateral ontrol

of the vehi le, as illustrated in Figure 2.6. If the vehi le rosses a lane

marking a Lane Departure Warning (LDW), [20℄, may be issued to the

driver. A lane guidan e system losely related to LDW is Lane Keeping

Assistan e (LKA), [16℄, where the driver is supported by a torque on the

steering wheel to stay in the urrent lane. In [7℄ the problem of road or

lane departures and vehi le stability are addressed in a ommon framework,

Figure 2.4: ACC automati ally maintains a driver set time gap to the vehi le

in front.

11


(a) (b)

Figure 2.5: (a) FCW displayed in a Head Up Display (HUD). The red light

displayed to the driver in the windshield is designed to resemble the appear-

an e of vehi le brake lights. (b) When the host vehi le enters the red zone,

an imminent ollision is dete ted and an autonomous brake intervention is

initiated.

thereby ombining and enhan ing the fun tionality of lane guidan e systems

and ESC. There are also systems that support the driver when performing

lane hange maneuvers. Lane Change Aid (LCA) systems, [21℄, monitor

adja ent lanes and inform the driver when an obsta le is present in the blind

spot of the rear view mirrors, see Figure 2.6b. In some situations there is

very little, if any, time to warn the driver of a potential hazard, making

it justi�ed for a CA system to ontrol the steering of the host vehi le to

avoid a idents. A system designed to avoid ollisions with on oming tra�

using steering interventions, referred to as Emergen y Lane Assist (ELA),

is presented in [22℄.

Information of host vehi le motion and road geometry an also be used

to assess the present state of the driver. If a driver is fatigued, distra ted

or even impaired by drugs, this will a�e t the driver's ability to maneuver

the vehi le smoothly in the urrent road lane. [23℄ presents a method for

dete ting inadequate driving behaviour, whi h an be used by systems to

e.g. inform the driver when about to fall asleep.

The underlying te hnology for DA systems is dis ussed in the follow-

ing subse tions. DA systems are me hatroni systems and onsist of three

basi layers, namely the per eption, de ision and a tion layers. The ar hi-

te ture for a DA system performing autonomous interventions is illustrated

in Figure 2.7.

12

2.3. Driver Assistan e

(a) (b)

Figure 2.6: (a) A lane guidan e system dete ts the lane markings and warns

the driver (LDW), or applies a steering wheel torque (LKA), when rossing

the lane boundary. (b) The olored zones visualize the blind spots, i.e. the

zones not visible to the driver through the rear view mirrors. LCA indi ates

that an obsta le is present in the blind spot by lighting a small lamp lose

to the rear view mirror.

2.3.1 Sensor Te hnology

A key enabler for DA systems is reliable remote sensing te hnology. In the

per eption layer, see Figure 2.7, sensors olle t observations from the envi-

ronment, driver and host vehi le. Depending on the requirements imposed

by the system, various te hnologies an be hosen to deliver an interpreta-

tion of the surrounding environment.

A frequently used sensor te hnology is omputer vision, whi h dete ts

and lassi�es obje ts in the environment using image data olle ted by am-

eras. Computer vision is the dominant te hnology to retrieve information

on the road geometry and the relative position of the host vehi le to the

road, whi h is done by dete ting the lane markings or the edge of the road.

A tive sensors su h as radar, laser or ultrasoni sensors transmit radio,

opti al or sound signals and evaluate obje t attributes by interpreting the

re�e ted response of the transmitted signal. Also, observations from digital

maps and sensors mounted on other vehi les or infrastru ture an be made

available to the safety system through a ommuni ation devi e.

In many appli ations, system requirements annot be ful�lled by a single

sensor. Sensor observations from multiple sensors are ombined, or fused,

to provide an enhan ed view of the environment. Also, obje ts observed

by sensors are tra ked over time to redu e the in�uen e of noise. General

frameworks for sensor data fusion and tra king are des ribed in [24, 25℄

while [26�29℄ des ribe work tailored to DA systems.

13


Sensor 1

Sensor 2

Sensor n

SensorFusion

&Tracking

ThreatAssessment

DecisionMaking

VehicleControl

DriverInteraction

Perception Decision Action

Figure 2.7: System ar hite ture for an a tive safety system designed to

intervene in ase a riti al situation arises. The per eption layer provides

information used for de ision making in the de ision layer. The de ision is

exe uted in the a tion layer via one or multiple a tuators, e.g. brake system

or driver information displays.

2.3.2 De ision-Making and Interventions

In the de ision layer, see Figure 2.7, input from the per eption layer is used

to de ide if and how to intervene. This de ision fun tion onsists of two

parts. The pro ess of onverting state estimations, e.g. obje t positions,

into measures des ribing whether or not the host vehi le is in a hazardous

situation, i.e. if surrounding road users and obje ts onstitute a threat of

ollision, is termed threat assessment. Based on the threat measures, a

de ision-making algorithm hooses what, if any, a tion should be taken by

the system.

The earlier, relative to the potential a ident, the system intervenes,

the more likely it is to prevent the a ident. Also, the earlier the system

intervenes, the more likely it is that the driver is well aware of the hazard

and thus perfe tly apable of preventing the a ident. If the latter is true

then the driver would onsider the intervention unne essary. Therefore, the

aim of the de ision fun tion is usually to intervene at the latest point in

time when the intervention type is still likely to su eed, where su ess is

de�ned as e.g. preventing or mitigating the onsequen es of an a ident.

A CA system aims to avoid all potential ollisions. For lane guidan e

systems, the aim is not as straightforward to de�ne sin e a lane departure

not ne essarily leads to a dangerous situation. Most LDW systems aim

14

2.4. Autonomous Driving

at issuing warnings ex lusively when lane departures are unintentional. In

situations when the driver intentionally deviates from the urrent lane, it is

assumed that the driver an manage the situation.

There is a range of possible a tions, or intervention types, whi h an be

applied when a hazardous situation is dete ted. If the situation is dete ted

early, the system, e.g. FCW or LDW, an warn the driver by for instan e

audible, visual or hapti feedba k. In ertain situations, there is no time

for the driver to rea t to the feedba k and perform a driving maneuver to

avoid the impending a ident. In those situations the system an, to avoid

the a ident, autonomously ontrol the brakes or the steering.

System interventions are sometimes per eived as intrusive by the driver.

The level of intrusiveness varies between intervention types where warnings

or information to the driver are generally less intrusive than autonomous

vehi le ontrol. The amplitude of the intervention also has an in�uen e as

e.g. a loud warning signal is often onsidered more intrusive than a subtle

warning signal. The possibility for the driver to override an intervention

also a�e ts the level of intrusiveness.

2.4 Autonomous Driving

Automotive safety systems whi h intervene autonomously to prevent a i-

dents are urrently ommer ially available from a large number of vehi le

manufa turers. The systems are evolving to handle more and more oper-

ating s enarios su h as interse tions and night-time driving, and this trend

is likely to ontinue, see Figure 2.8. An enabler for this evolution is the

availability of more a urate, a�ordable remote sensors.

The resear h ommunity has for quite some time fo used on the next ma-

jor step in automotive safety, namely Autonomous Driving systems. These

are systems whi h takes full responsibility for the driving task as opposed

to DA systems whi h still require the driver to monitor the system. In the

2007 DARPA Urban Challenge, [30℄, 35 teams formed from ollaborations

between industry and a ademia ompeted with driverless vehi les in an ur-

ban environment. A total of six self-driving vehi les ompleted the ourse

whi h in luded tasks su h as negotiating interse tions, parking and avoiding

vehi les stalled on the road.

In many ways AD systems are a natural evolution of DA systems and

a number of ompanies, vehi le manufa turers and others, have ommuni-

ated their aim to ommer ialize this te hnology. The potential bene�t of

AD systems is undoubtedly huge, not only in terms of safety, but also in

terms of redu ed fuel onsumption, redu ed ongestion and added driver

onvenien e.

15


(a) (b)

Figure 2.8: (a) Possible sensor setup for future vehi les: 360◦ �eld of view

with ameras and radars. (b) Autonomous vehi les and future driver assis-

tan e systems must handle more tra� s enarios, e.g. night onditions.

2.5 System E�e tiveness

In the last de ades, passive safety systems have made a major ontribution

to road tra� safety through innovations su h as the safety belt, rumple

zones and airbags, see Figure 2.9. Their e�e tiveness has been extensively

studied using a ident statisti s. In the U.S. during 2008, a ording to [31℄,

seat belts saved 13 250 lives, frontal airbags 2 546 and hild restraints 244.In [32℄, it is shown that passive safety improvements have ontributed to a

signi� ant de rease in injury severity between the 1970s and the 1990s, also

when ignoring e�e ts from seat belts and airbags.

The e�e tiveness of passive safety systems is assessed by governments

around the world. In Europe, EuroNCAP has sin e 1997 assessed ars, by

e.g. rash tests, in order to provide onsumers with an independent rating

of safety performan e. A tive safety systems su h as ESC are in luded

in this rating and in 2014 AEB will also be in luded, [33℄. These ratings

are important selling arguments for vehi le manufa turers and thus they

en ourage rapid development of new safety te hnology.

Vehi le dynami s ontrol systems have been widely deployed in the mar-

ket for many years, making it possible to assess their e�e tiveness in im-

proving road safety using a ident statisti s. In [34℄, multiple studies inves-

tigating the safety impa t of ABS are reviewed. A majority of the studies

indi ate that equipping vehi les with ABS signi� antly redu es the o ur-

ren e of a idents involving multiple vehi les. Some studies also indi ate

that ABS in reases the o urren e of run-o�-road a idents. Possible on-

tributing fa tors to this in rease in lude inappropriate use of ABS and driver

behaviour adaption, e.g. when the driver de reases driving safety margins

16

2.5. System Effe tiveness

(a)

(b)

Figure 2.9: (a) The airbag is an example of passive safety te hnology. (b)

Crash tests are used to assess passive safety e�e tiveness.

due to awareness of the positive safety e�e ts of ABS.

A ording to [35℄, use of ESC redu es fatal single-vehi le a idents in-

volving ars and Sport Utility Vehi les (SUVs) by 30-50% and 50-70% re-

spe tively. Considering that single-vehi le rashes stand for 60% of all fatal

rashes in the U.S., [31℄, the potential safety impa t of ESC is signi� ant.

Additionally, the redu tion of rollover a ident fatalities, related to the use

of ESC, is in [35℄ estimated to 70-90%, regardless of vehi le type.

DA systems have, if at all, been introdu ed to the market relatively re-

ently whi h explains why their e�e tiveness has not been studied to the

same extent. An overview on the subje t is given in [17℄ whi h on ludes

that the safety impa ts of DA systems are expe ted to be onsiderable. Due

to the la k of data, several approa hes have been proposed to predi t sys-

tem e�e tiveness su h as re onstru ting real-world a idents from a ident

databases and using simulations to determine if a given system ould prevent

these a idents. Using this method [36℄ predi ts that a newly introdu ed

CA system ould prevent up to 24% of pedestrian fatalities and [37℄ predi ts

that a similar system ould redu e driver fatalities in rear-end rashes by

up to 50%.

17


Driver

Process

EnvironmentVehicle

Active Safety

System

Figure 2.10: The driver monitors both the vehi le and the surrounding envi-

ronment to ontrol the vehi le. The a tive safety system monitors the om-

plete pro ess and ontrol the vehi le either dire tly, or indire tly through

driver intera tion, see Figure 2.7. The swit hes determine if the system is

exe uted in open- or losed-loop, see Se tion 3.2.

2.6 Veri� ation Challenges

This se tion introdu es terminology and dis usses the hallenges asso iated

with system veri� ation. Consider a pro ess, as illustrated by the top part

of Figure 2.10 onsisting of a vehi le, a driver, and an environment. The

environment has in general both stati and dynami ontent where stati

ontent is e.g. roads, trees and tra� signs and dynami ontent is e.g. road

users su h as ars, bi y les and pedestrians. The a tive safety system, in-

tera t with the pro ess a ording to Figure 2.10. The system monitor and

ontrol the pro ess to ensure that the host vehi le is operated safely.

The purpose of system veri� ation is to ensure that the system per-

forman e meets the system requirements. This must be addressed for the

omplete set of operating s enarios, de�ned by the variations in the pro-

ess, i.e. the variations in vehi le, driver and environment behaviour. The

set of operating s enarios is essentially unlimited in size as ombinations

of e.g. weather onditions, road user types, appearan e and motion pat-

terns are in�nite, see Figure 2.11a. De�ning the boundaries of this set is a

hallenge in itself sin e the system is mobile and travels in an environment

whi h is ompletely or partially unknown to the system a priori.

The system relies on real-time remote sensing of e.g. road users and road

geometry to make de isions on when and how to intervene. The sensing

performan e depends on variations in the environment, as illustrated by

Figure 2.11b. For instan e, a amera subje ted to dire t sunlight will exhibit

poor obje t dete tion performan e, mu h like the human eye.

The more intrusive an intervention type is, see Se tion 2.3.2, the less

18

2.6. Verifi ation Challenges

(a) (b)

Figure 2.11: Illustration of system veri� ation hallenges as seen by a vision

sensor. (a) One of the many possible omplex tra� situations. (b) The

sensor is partially blinded when exiting the tunnel.

likely the driver is to a ept an unne essary intervention. Consequently,

the a eptable rate of unne essary interventions is very low for systems per-

forming intrusive interventions. The large quantity of operating s enarios

makes veri� ation of this requirement, on a low rate of unne essary inter-

ventions, espe ially hallenging.

19

20

Chapter 3

Veri� ation Methods

The goal of system performan e evaluation, in the ontext of this thesis,

is to determine the performan e of an a tive safety system in a given set

of operating s enarios. If system performan e evaluation is used for sys-

tem veri� ation, the performan e estimate is ompared to a set of system

requirements, whi h spe ify the a eptable level of system performan e. A -

urate and e� ient methods for system performan e evaluation are needed

for several purposes, e.g. system veri� ation, system tuning or analysing

the system sensitivity to disturban es. For veri� ation purposes it is usu-

ally su� ient to derive or estimate a bound on performan e, to show that

the system requirements are ful�lled. As a onsequen e, some methods

fo us on performan e bounds while some fo us on performan e estimates.

This hapter provides an overview on veri� ation methods used in an a tive

safety ontext.

3.1 Performan e Metri s

In this se tion, performan e metri s des ribing the ability of the system to

make orre t de isions are presented. A ommonly used terminology for de-

s ribing the nature of in orre t de isions omes from statisti al hypothesis

testing, extensively overed in [38℄, and was �rst dis ussed in [39℄. A hy-

pothesis test is lassi�ed with regards to the test out ome, i.e. the de ision

on what hypothesis to a ept, and the true hypothesis, see Figure 3.1. The

default de ision, often a de ision not to perform an a tion, is in statisti al

hypothesis testing represented by the null hypothesis. A test out ome is

said to be negative if the null hypothesis is a epted and positive in the

opposite ase.

In an a tive safety ontext, a test would be e.g. to de ide whether or

not to initiate an autonomous brake intervention, the true hypothesis would

21

Chapter 3. Verifi ation Methods

Missed InterventionType II Error

False Negative

True NegativeIntervention is needed(Null hypothesis is

nottrue)

Intervention is needed(Null hypothesis is false)

System intervene(Null hypothesis is accepted)

does not System intervenes(Null hypothesis is rejected)

Unnecessary InterventionType I Error

False Positive

True Positive

Figure 3.1: Error types for a system de iding on whether or not to intervene.

represent the orre t de ision and the null hypothesis would represent the

de ision not to intervene.

Linked to this, there are two types of errors, ommonly referred to as

Type I and Type II errors. If the null hypothesis is true and is reje ted by

the test, the error is Type I or false positive. If instead the null hypothesis is

false and is a epted by the test, the error is Type II or false negative. False

positives and false negatives are in this thesis referred to as unne essary and

missed interventions respe tively, sin e these terms are more des riptive for

a tive safety appli ations.

3.2 Method Properties

Di�erent methods have di�erent properties and ea h property ontributes

to the overall strength or weakness of the method. Below, relevant method

properties are de�ned and dis ussed.

Coverage

Coverage is a measure used to des ribe the degree to whi h the set of oper-

ating s enarios is evaluated. A major bene�t of theoreti al methods is that

full overage is possible to attain.

By ondu ting experiments, i.e. tests, system performan e an be eval-

uated in a hosen set of s enarios. For a omplex pro ess, generally, the set

of operating s enarios an be des ribed by an unbounded number of param-

eters. As the number of operating s enarios grows exponentially with the

number of s enario parameters, this set is very large. This e�e t, known

as the urse of dimensionality, makes full overage of the set of operating

s enarios unrealisti .

Methods for sele ting a set of s enarios to evaluate are generally referred

22

3.2. Method Properties

to as experimental design or Design of Experiments (DoE), see e.g. [40℄ for

a wide treatment of the subje t. The s enario parameter spa e may for

instan e be overed by drawing random samples, or using a more systemati

approa h, the samples may be hosen su h that the overage is evenly spread

while minimizing the number of evaluated s enarios.

Online/O�ine

Systems are evaluated either online or o�ine, where these terms are used

a ording to the following de�nitions.

De�nition 1 A system is online when for ed to exe ute in real-time.

De�nition 2 A system is o�ine when not for ed to exe ute in real-time.

Online experiments evaluate if the system omply with real-time re-

quirements but have the obvious disadvantages of not being able to exe ute

slower, or faster, than real-time.

Open/Closed-Loop

An a tive safety system monitors a pro ess and use this information to

in�uen e said pro ess, as shown in Figure 2.10. In some experiments the

s enario is partially or ompletely �xed, meaning that the system has limited

or no in�uen e on the pro ess. As a onsequen e, the following de�nitions

are useful.

De�nition 3 A system is exe uted in losed-loop when the ontrol loop

between the system and the pro ess is losed.

De�nition 4 A system is exe uted in open-loop when the ontrol loop be-

tween the system and the pro ess is open.

Note that open-loop exe ution does not equal open-loop ontrol, whi h om-

monly refers to a ontrol system operating without feedba k. Open-loop

exe ution means that the system annot in�uen e the pro ess during exe-

ution. When evaluating the orre tness of de isions, it is in many ases

su� ient to exe ute the system in open-loop. This is valid when the system

does not perform any a tion prior to the de ision to intervene, and will

onsequently not in�uen e the pro ess prior to said de ision.

23


E� ien y

The ost in terms of time and money are measures of the method e� ien y.

Online methods are time onsuming as real-time exe ution is required and,

in general, methods involving real world experiments have higher �nan ial

ost than theoreti al analysis and omputer experiments. Also, the ost

asso iated with method development vary between di�erent methods.

Repeatability and Reprodu ibility

Repeatability and reprodu ibility are statisti al terms asso iated with a -

ura y. An experiment is repeatable if it an be performed on two di�erent

o asions with no substantial hange between measured quantities. Re-

peatability only requires this to be possible using the same personnel and

equipment. An experiment is also reprodu ible if it is repeatable using dif-

ferent personnel and equipment when performed on two di�erent o asions.

Repeatability and reprodu ibility ensures that the experiment results are

not signi� antly a�e ted by temporary fa tors.

Ground Truth Data

Ground truth data refers to information that is on�rmed in an a tual �eld

he k at a lo ation, as opposed to information a quired from a distan e.

In remote sensing, the term is ommonly used to des ribe information on-

sidered a urate, relative to information a quired from the remote sensing

system being evaluated. Ground truth data des ribes the true s enario,

e.g. how obje ts move in the s ene, and aids in evaluating sensor and on-

trol system performan e.

Model A ura y

The model a ura y is the ability of the model to generate output equivalent

to output from the real system. Model a ura y should not be onfused with

system a ura y whi h is the ability of the system to generate output with

small errors, e.g. a sensor delivering a urate measurements. If the input

to an a urate model is equivalent to the real operating onditions, the

output is realisti . Output olle ted from real systems in the real operating

environment is realisti per de�nition. A hieving high output realism from

models often indu es high ost, in the form of time, money or both.

24

3.3. Models

S enario Representativeness

For a s enario set to be representative, it must orre tly re�e t the set of op-

erating s enarios in terms of system performan e. Sampling s enarios using

the real system in the real operating environment is the most obvious way

to olle t data with high representativeness. When modeling or re reating

real s enarios, e.g. in omputer simulations or real world test environments,

limitations imposed by pro ess models or test equipment make the s enarios

less representative to a varying degree.

Pro ess Controllability

The ability to ontrol the pro ess during evaluation is referred to as pro-

ess ontrollability. La k of pro ess ontrollability is primarily an issue in

real world experiments, where ontrol of the pro ess related to for instan e

weather or multiple obje t dynami s is hallenging. Also, safety- riti al sit-

uations su h as ollisions and near- ollisions are di� ult to realize in real

world experiments where they are potentially hazardous for involved per-

sonnel and destru tive to the equipment used.

3.3 Models

Many veri� ation methods use mathemati al models to des ribe the a tive

safety system, the pro ess and their intera tion, see Figure 2.10. The om-

plexity of the models vary signi� antly and also depends on the interfa es

to other models, e.g. interfa es to per eption or a tion layer models might

require more or less omplexity in the orresponding pro ess models.

3.3.1 System Models

A tive safety systems onsist of three layers, see Figure 2.7. Commonly, the

de ision layer is an available software intera ting only with the per eption

and a tion layers. The latter two layers intera t physi ally with the pro ess,

e.g. the surrounding environment, and modeling of these layers are dis ussed

below.

Per eption Layer Models

As des ribed in Se tion 2.3, the per eption layer provides input data to the

de ision layer, based on sensor observations of a pro ess. In the per eption

layer, observations from one or several sensors are generally passed through

multiple layers of advan ed signal pro essing, fusing sensor observations into

25


estimated states su h as positions and velo ities of dete ted obje ts. Sensor

models des ribe how the pro ess is per eived by the sensors and an be

formulated on many di�erent abstra tion levels.

Low-level sensor models des ribe the transformation between the pro ess

and the unpro essed sensor observations whereas high-level sensor models

des ribes the transformation between the pro ess and the estimated states.

Modeling the physi s of remote sensing te hnologies su h as ameras, lidars

and radars, is a omplex task, espe ially when onsidering situations where

the sensor is observing a omplex environment, see e.g. [41℄ for an overview

of low-level radar models. This is why high-level empiri al models are often

used.

A ommon high-level approa h is to model state estimates, e.g. obje t

position or velo ity, as the true estimate in�uen ed by a noise model. If

noise is ignored, the models represent ideal or perfe t sensors, as used in

e.g. [42℄. A ommon noise model is additive Gaussian noise, as used in

e.g. [43�45℄ and Paper 3. High-level models are in many ases a major

simpli� ation of the sensor and in orporate very limited information on

how the sensor errors depend on the observed pro ess. Nevertheless, they

are useful when studying systems in limited s enario sets, systems with

very a urate sensors, or the aspe ts of system performan e not a�e ted by

sensor errors.

For a omputer vision system, a low-level sensor model des ribes how

a amera per eives the pro ess, i.e. generates a sequen e of images. Te h-

niques for generating images with omputers, known as rendering, are stud-

ied in omputer graphi s. Rendering imagery requires pro ess models whi h

des ribe e.g. the 3D stru ture of obje ts in the environment. Rendered

imagery based on 3D models, in ontrast to real imagery olle ted from

ameras, is denoted virtual imagery while rendered imagery where virtual

obje ts are superimposed on real imagery is denoted augmented imagery.

In [46℄, published in 1995, it is argued that the realism of virtual imagery

is su� ient for evaluation of mobile omputer vision systems. Sin e then

omputer graphi s has evolved rapidly, as an be observed in for instan e the

video gaming and movie industries. Nonetheless, photo-realism in virtual

imagery is not easily a hieved and an overview of the multidis iplinary

hallenges of rendering is found in [47℄. Paper 4 explores the possibility

of rendering augmented imagery for o�ine evaluation of omputer vision

systems.

Online evaluation methods require image rendering in real-time, mak-

ing it more hallenging to attain high realism in rendered images. The

tra� simulation environments des ribed in [48, 49℄ have software modules

available for rendering of virtual imagery in real-time.

26

3.3. Models

A tion Layer Models

Models of e.g. braking and steering systems are needed to des ribe how

the driver and vehi le are in�uen ed by system de isions. Des riptions

and models of automotive systems and omponents, in luding a tive safety

a tuators, are thoroughly des ribed in [50℄. Note that when the system is

exe uted in open-loop, modeling the a tion layer is unne essary.

3.3.2 Pro ess Models

Modeling of the pro ess, i.e. the driver, vehi le and surrounding tra� en-

vironment, is dis ussed in the following se tions.

Driver Models

For evaluation methods based on real or augmented data, the behaviour of

the driver is in orporated in the data. Therefore, only purely model-based

methods require a driver model to generate the driver input to the vehi le,

e.g. steering and braking, based on feedba k from the vehi le, environment

and a tive safety system. Driver modeling is a wide �eld of resear h and

models are often more or less appli ation spe i� . A olle tion of papers

treating driver models in the automotive domain from a variety of perspe -

tives is found in [51℄.

Vehi le Models

Vehi le motion models are needed to des ribe both the motion of the host

vehi le as well as vehi les in the surrounding environment. Vehi le motion

is studied within the �eld of vehi le dynami s whi h is the topi of several

books, e.g. [52℄.

Environment Models

When modeling a dynami tra� environment, ea h obje t in the environ-

ment, e.g. ars, roads and pedestrians, are des ribed by individual models.

Depending on the interfa e to the a tive safety system, e.g. sensor and a -

tuator models, the environment models need to in lude di�erent aspe ts. If

low-level sensor models are used, the level of detail of the environment mod-

els is usually higher ompared to when high-level sensor models are used.

If for instan e virtual imagery is generated by a sensor model, a omplete

3D stru ture of the environment is required.

Presently, there exist several simulations environments for simulating

tra� environments in luding a tive safety systems su h as PreS an, [48℄,

27


v-TRAFFIC, [49℄, or the Volvo Cars Tra� Simulator (VCTS), [27℄. These

softwares in lude models of driver, vehi le and environment.

3.4 Methods

This se tion des ribes di�erent types of analysis and veri� ation methods.

The methods evaluate real physi al systems, mathemati al models, or a

ombination thereof.

3.4.1 Real Driving

Online experiments using real vehi les are performed both in real tra�

and on dedi ated test tra ks. They are repeatable to some degree at test

tra ks but to a minor degree in real tra� . If the system is online, it an be

evaluated in losed-loop and sensor data often have the advantage of being

realisti .

Real Tra�

Real tra� experiments are primarily used to estimate the probability of an

unne essary intervention from a set of randomly sampled s enarios. Also,

experiments are ondu ted to estimate the probability of a missed interven-

tion, given that the tested system has relatively frequent and non-intrusive

interventions, whi h is valid for e.g. an LDW system.

Variations between di�erent vehi les and system omponents are hand-

led by using multiple vehi les and omponents in testing. For a randomly

sampled s enario set to be representative, the s enarios available for sam-

pling must also be representative. In [27, 53, 54℄, a Real World User Pro�le

(RWUP) is used to ensure that a representative s enario set is sampled, tak-

ing into a ount for instan e di�erent driving styles, weather and driving

environments.

As dis ussed in Se tion 2.6, the a eptable rate of unne essary interven-

tions for highly intrusive systems is very low, meaning that a large amount of

driving data needs to be olle ted to ensure that the requirement is ful�lled.

The obvious drawba k is that real tra� experiments are both expensive

and time onsuming. Also, ground truth data is hallenging to obtain sin e

the environment is un ontrolled.

Test Tra k

On test tra ks, spe i� types of s enarios are tested in a more ontrolled

setting. Compared to real tra� experiments, test tra k experiments of-

28

3.4. Methods

(a) (b) ( )

Figure 3.2: Non-destru tive tests in ollision and near- ollision s enarios

where (a) shows stationary pedestrian dummies of both adult and hild

size, (b) shows an in�atable moving obje t representing a moving vehi le

and ( ) shows an arti� ial obje t representing a moose.

fer a higher degree of pro ess ontrollability, repeatability and reprodu-

ibility. Motions of involved obje ts an be ontrolled to reate desired

s enarios. Also, ground truth an be obtained by e.g. positioning involved

tra� parti ipants and obje ts with an a urate positioning system su h

as Di�erential Global Positioning System (DGPS). Pro ess ontrollability

on test tra ks is better but not without limitations as for instan e weather,

e.g. snow or rain, and animals rossing the road are still di� ult to repro-

du e on demand.

When re reating ollision and near- ollision situations on test tra ks,

non-destru tive tests are preferred to ensure safety. Therefore, ollisions

are ondu ted between the host vehi le and low-mass obje ts su h as in�at-

able ars, see Figure 3.2. This reates limitations on s enarios possible to

re reate as even state-of-the-art in�atable ar or pedestrian systems annot

re reate all motions possible for real ars or pedestrians. It also degrades

the representativeness of the s enario sin e an in�atable obje t might not

be per eived by the sensors as would an equivalent real obje t.

For highly intrusive systems su h as AEB, the s enarios in whi h the sys-

tem should intervene are very rare, meaning that estimating the probability

of a missed intervention would require an unrealisti amount of driving data

from real tra� onditions. Consequently, the probability of a missed inter-

vention is often, e.g. [27, 53, 54℄, assessed by repli ating ollision situations

on test tra ks. Su h tests are also used to estimate the e�e tiveness of the

system, for instan e impa t speed redu tion.

For veri� ation purposes, s enarios in whi h an in orre t system de ision

is most likely, i.e. the worst ase s enarios, are often repli ated on test tra ks,

thus omplementing tests in real tra� . If the system behaves orre tly in

these worst ase s enarios it an be argued that less hallenging s enarios

29


are not likely to pose a problem. Paper 1 presents a theoreti al method for

identifying the worst ase s enarios for a CA system. Examples of s enarios

most likely to ause unne essary interventions are near- ollision situations,

e.g. evasive maneuvers where the time or distan e margins to a potential

ollision are small.

3.4.2 Closed-Loop Simulations

If the pro ess is mathemati ally modeled, the system behaviour an be sim-

ulated in losed-loop with omputer generated inputs. Model-In-the-Loop

(MIL) simulations use a system model while Software-In-the-Loop (SIL)

simulations use an a tual system implementation, whi h not ne essarily

is exe uted on the produ tion hardware. The border between MIL and

SIL is sometimes hard to de�ne but examples of one or the other is found

in [27, 43, 44, 55℄.

MIL/SIL o�ers many bene�ts over real driving when omparing for in-

stan e e� ien y and pro ess ontrollability. Experiments are repeatable and

reprodu ible and these are important properties when omparing di�erent

system on�gurations. MIL/SIL are o�ine methods, meaning there are no

real-time onstraints, making it possible to simulate s enarios with speeds

limited only by the omputational power available. In addition, systems

an be tested before deployment at early stages in development, without

the need of fun tioning hardware.

If system hardware omponents are available, their performan e an

be tested online with omputer generated inputs as Hardware-In-the-Loop

(HIL). The bene�t of HIL, ompared to MIL/SIL, is that the hardware

is also evaluated. The drawba k is the online property, onstraining HIL

simulations to real-time exe ution. In [56℄ a test fa ility where a omplete

vehi le is set up on a hassis dynamometer, with robot vehi les represent-

ing the surrounding environment, is des ribed and referred to as Vehi le

Hardware-In-the-Loop (VeHIL).

3.4.3 Data Replay

In data replay methods, real re orded data is used to evaluate the system

o�ine. If real data is used ex lusively, di�erent system software on�gura-

tions an be evaluated by omputer simulations without any loss in input

data realism, as done in [57℄, but often without the impe able ground truth

data a essible using in-the-loop methods. Limited ground truth, e.g. im-

proved estimates of obje t motion, an be obtained by o�ine pro essing

of real measurement data, as done in [43, 57℄. Data replay methods are

restri ted to open-loop, sin e the s enario is �xed by the olle ted data.

30

3.5. Method Comparison

Another option is to ombine real data with model-based methods thus

generating mixed or augmented data, for example by adding new obje ts or

errors in re orded sensor data. This is exempli�ed in [43℄ where three FCW

algorithms are simulated with input data onsisting of a urate lead vehi le

motion, obtained from real data, and noise, from a radar model. Augmented

data replay has the potential to pi k the best out of two worlds but also

risk pi king the worst. The modeling e�ort ompared to purely model-

based methods is limited and many of the advantages are partly preserved,

e.g. pro ess ontrollability, or ompletely preserved, e.g. repeatability and

la k of real-time onstraints. The downside is that simulation is limited

to open-loop, as some real data is used, and that the data realism is now

dependent on a model-based data augmentation method whi h then requires

validation.

In Paper 4, an augmented data replay framework is formulated, used

for omputer vision systems. This framework uses a low-level sensor model,

dis ussed in Se tion 3.3. If instead high-level sensor models are available,

Paper 3 presents e� ient data replay methods for de ision fun tion tuning

and sensitivity analysis with regards to input perturbations, whi h an be

applied to real, model-based or augmented data.

3.4.4 Theoreti al Methods

The ultimate goal of system veri� ation is to prove that the system meets

the system requirements. Methods for proving system properties, su h as

requirement omplian e, are known as formal methods, see [58℄ for an ex-

tensive survey.

If the a tive safety system and the set of operating s enarios are de-

s ribed mathemati ally it is sometimes possible to derive analyti al expres-

sions des ribing system performan e, as done in Paper 1. Generally, this is

only possible when making quite signi� ant simpli� ations.

For dynami al systems, guarantees of not entering an undesired system

state may be obtained by omputing the set of rea hable states. Paper 2

explores the use of rea hability analysis, [59℄, and viability theory, [60℄, to

formally verify a ollision avoidan e system.

3.5 Method Comparison

This se tion provides a brief omparison of the methods presented in Se -

tion 3.4 with regards to the properties dis ussed in Se tion 3.2. An overview

of the more or less dis rete properties is presented in Table 3.1. Figures 3.3

and 3.4 ompare pro ess ontrollability, sensor data realism and e� ien y

31


O�ine

Online

Real TrafficData Replay

Hardware-In-the-Loop

Test Track

Model/Software-In-the-Loop

Vehicle Hardware-In-the-Loop

Real Traffic

AugmentedData Replay

Process Controllability

TheoreticalAnalysis

Eff

icie

ncy

Figure 3.3: A qualitative sket h for the relation between e� ien y and

pro ess ontrollability for di�erent evaluation methods.

for di�erent methods. It should be noted that these properties are appli a-

tion dependent, meaning that the �gures should not be onsidered absolute

truths.

In Figure 3.3 it an be noted how the model-based methods are superior

Vehi le

System

Online

Closed- Ground

Repeatability Reprodu ibility

hardware loop truth

Theoreti al analysis x x x x

Real tra� x x x x

Test tra k x x x x x (x) (x)

Model/software-in-the-loop x x x x

Hardware-in-the-loop x x x x x x

Vehi le hardware-in-the-loop x x x x x x x

Real tra� data replay (x) (x) (x) x

Augmented data replay (x) (x) x x

Table 3.1: Overview of method properties for di�erent methods. The fa t

that data replay methods use vehi les and system hardware indire tly,

i.e. for initial data olle tion, is represented by a tentative "(x)". The

same notation indi ates that, on test tra ks, some aspe ts are repeatable

and reprodu ible while others are not. Ground truth for real data replay is

also tentatively marked as o�ine pro essing an o�er limited ground truth

data.

32

3.5. Method Comparison

TheoreticalAnalysis

Online

O�ineEff

icie

ncy

Sensor Data Realism

Model/Software-In-the-Loop

AugmentedData Replay

Real TrafficData Replay

Real TrafficTest Track

Vehicle Hardware-In-the-Loop

Hardware-In-the-Loop

Figure 3.4: A qualitative sket h for the relation between e� ien y and

sensor data realism for di�erent evaluation methods.

in terms of pro ess ontrollability and also in many ases are very e� ient,

largely due to the o�ine property. The major hallenge for the model-based

methods is related to the sensor model a ura y, as visualized in Figure 3.4.

For the purely model-based methods, e.g. losed-loop simulations, sensor

models generating realisti data are either unexisting or resour e demand-

ing. Methods using ex lusively real tra� data have, by de�nition, realisti

sensor data. Augmented data is relatively realisti but with the drawba k

that augmented data replay is limited to open-loop exe ution of the system,

as shown in Table 3.1.

The omparisons in Table 3.1, Figure 3.3 and Figure 3.4 learly show the

omplementary nature of the presented methods. Thus, veri� ation is often

arried out using a variety of methods, as exempli�ed in [27, 53, 54℄. Meth-

ods whi h require omplete vehi les or system hardware are onstrained

to use in the later stages of the development pro ess. Alternatively, they

may be employed with de reased predi tion a ura y using early hardware

prototypes.

33

34

Chapter 4

Summary of In luded Papers

This hapter provides a brief summary of the papers in luded in the thesis

and also des ribes the ontributions to ea h paper by the author of this

thesis. Full versions of the papers are in luded in Part II.

Paper 1

J. Nilsson, A. Ödblom and J. Fredriksson, Worst Case Analy-

sis of Automotive Collision Avoidan e Systems, submitted for


As dis ussed in Se tion 2.6, the set of tra� s enarios whi h generates the

input to an a tive safety de ision fun tion is very large. This paper theo-

reti ally identi�es s enarios with a high risk of in orre t system de isions,

i.e. the worst ase s enarios. The main hallenge with this approa h, as

dis ussed in Chapter 3, is to model system and s enarios in su h a way

that performan e an be des ribed analyti ally while still in luding the key

fa tors a�e ting performan e, e.g. sensor errors or obje t motion.

The key idea of this paper is to theoreti ally investigate the fundamental

limitations of a ollision avoidan e system, subje t to systemati measure-

ment errors and unexpe ted future obje t motion, in terms of early and

unne essary interventions. Spe i� ally, we in lude e�e ts of sensor and a -

tuator delays, and derive losed-form expressions for the worst ase perfor-

man e, with regards to longitudinal or lateral predi tion and measurement

errors. For a system example, numeri al results show how de ision timing

and robustness depend on s enario and system parameters. The method an

be used for system veri� ation, tuning or sensitivity analysis with regards

to s enario variations and sensor errors. Also, s enarios with inadequate

performan e an be identi�ed, thus improving existing test methods by di-

re ting testing and analysis e�orts towards relevant s enarios.

35

Chapter 4. Summary of In luded Papers

The thesis author was responsible for the problem formulation, deriva-

tion of the losed-form expressions, implementation and writing the paper.

Paper 2

J. Nilsson, J. Fredriksson and A. Ödblom, Veri� ation of Colli-

sion Avoidan e Systems using Rea hability Analysis, submitted

as invited paper to the 19th IFAC World Congress, Cape Town,

South Afri a, 2014.

The losed-form expressions for performan e derived in Paper 1 are very

useful from a veri� ation perspe tive but for many omplex a tive safety

de ision fun tions, they are not possible to derive. The alternative of eval-

uating state traje tories, as done in traditional simulations and real vehi le

tests, does not provide guarantees for system performan e for all possible

state traje tories.

To address these limitations, Paper 2 des ribes a novel set-based frame-

work for analyzing under what onditions the absen e of in orre t de isions

may be guaranteed for a given ollision avoidan e de ision fun tion. Rea h-

ability analysis and viability theory are used to ompute unsafe and safe

sets, i.e. sets where an ideal system should or should not intervene respe -

tively. In these sets, in orre t de isions for a given de ision fun tion are

identi�ed using optimization te hniques. By separating the dynami s of

the input spa e from the de ision fun tion, non-linear and ad-ho de ision

fun tions are e� iently handled in the proposed framework.

The method is demonstrated on a ollision avoidan e system example

and, given the models used and absen e of measurements errors, we show

that the system does not make in orre t de isions. Furthermore, we des ribe

and demonstrate how to evaluate the robustness to measurement errors,

using the proposed framework.

The thesis author was responsible for the problem formulation, develop-

ment of the proposed methods, implementation and writing the paper.

Paper 3

J. Nilsson and M. Ali, Sensitivity Analysis and Tuning for A tive

Safety Systems, in Pro eedings of the 13th International IEEE

Conferen e on Intelligent Transportation Systems, 2010, pages

161-167, Madeira Island, Portugal.

Papers 1 and 2 are full overage methods, i.e. are on erned with veri�-

ation of the omplete s enario parameter spa e. Full overage methods

36

are desirable but set limitations on the omplexity of the involved math-

emati al models. In ontrast, Paper 3 onsiders veri� ation given that a

representative experimental data set is available.

The design and tuning of an a tive safety de ision fun tion, e.g. how

thresholds are pla ed, will de ide how sensitive the system performan e is

to input errors. Investigating the interplay between input errors, de ision

fun tion and system performan e gives rise to three relevant questions:

i. Given a de ision fun tion and input errors, what is the system perfor-

man e?

ii. Given a de ision fun tion and system performan e requirements, what

are the input requirements?

iii. Given input errors and system performan e requirements, how should

the de ision fun tion be tuned?

This paper proposes a framework for open-loop analysis of de ision fun -

tions, with regards to the above mentioned questions. By introdu ing a ro-

bustness measure, des ribing the robustness to input errors for the de ision

fun tion, e� ient o�ine methods are formulated. The robustness measure

is independent of the input errors, meaning that it needs to be estimated

only on e for ea h de ision fun tion and data set. This allows for e� ient

evaluation of the system performan e as ombinations of de ision fun tion

and input errors an be pro essed without evaluating the de ision fun tion

output for ea h ombination. The framework is applied to data olle ted

in an experimental setting. Also, it is demonstrated how it an be used for

setting input requirements and tuning the de ision fun tion.

The formulation of the presented framework and writing the paper were

jointly ondu ted by both authors of the paper. The author of this thesis is

responsible for the demonstration of the framework while the se ond author

is responsible for the olle tion of experimental data and development of

the de ision fun tion example.

Paper 4

J. Nilsson, A. Ödblom, J. Fredriksson, A. Zafar and F. Ahmed,

Performan e Evaluation Method for Mobile Computer Vision

Systems using Augmented Reality, in Pro eedings of the IEEE

Virtual Reality Conferen e, 2010, pages 19-22, Waltham, Mas-

sa husetts, USA.

37

Chapter 4. Summary of In luded Papers

The methods for analyzing de ision fun tions in Papers 1-3, all rely on

a urate modeling of sensor errors. In Paper 4, a novel framework using

augmented imagery is proposed for determining sensor errors of omputer

vision systems, whi h are widely used in a tive safety systems. The proposed

framework exploits the possibility to add virtual agents into a real data

sequen e olle ted in an unknown environment, thus making it possible to

e� iently reate augmented data sequen es, in luding ground truth, to be

used for performan e evaluation. Varying the ontent in the data sequen e

by adding di�erent virtual agents is straightforward, making the proposed

framework very �exible.

The method has been implemented and tested on a pedestrian dete -

tion system used for ollision avoidan e. Preliminary results show that the

method has the potential to repla e and omplement physi al testing, for

instan e by reating ollision s enarios, whi h are di� ult to test in reality.

The formulation of the novel framework was jointly ondu ted by the

�rst two authors of the paper. The author of this thesis was also responsible

for writing the paper and supervising the ase study implementation done

by authors four and �ve.

Paper 5

J. Nilsson, J. Fredriksson and A. Ödblom, Reliable Vehi le Pose

Estimation using Vision and Single-Tra k Model, submitted for


The method in Paper 4 relies on an a urate 3D re onstru tion of the am-

era motion in six Degrees of Freedom (6-DoF). Extensive use of this method

requires this to be done without adding additional expensive sensors to the

vehi le. The ore idea of Paper 5 is to use a single-tra k vehi le model in a

lo al bundle adjustment framework to improve the pose estimates obtained

from a standard vehi le sensor setup, i.e. a forward looking mono ular am-

era, wheel speed, yaw rate and steering wheel angle sensors. This means

pose estimates are optimized not only with regards to observed image fea-

tures, but also with respe t to a single-tra k vehi le model and standard

in-vehi le sensors.

The des ribed method has been tested experimentally on hallenging

data sets at both low and high vehi le speeds as well as on a data set with

moving obje ts. The vehi le motion model in ombination with in-vehi le

sensors exhibit good a ura y in estimating planar vehi le motion. Results

show that this property is preserved when ombining these information

sour es with vision. Furthermore, the a ura y obtained from vision-only in

38

dire tion estimation is improved, primarily in situations where the mat hed

visual features are few.

The thesis author was responsible for the problem formulation, devel-

opment of algorithms, implementation, experimental validation and writing

the paper.

Paper 6

J. Nilsson, P. Andersson, I. Gu and J. Fredriksson, Augmented

Training Data for Pedestrian Dete tion, submitted to the 22nd

International Conferen e on Pattern Re ognition, Sto kholm,

Sweden, 2014.

Ma hine learning te hniques are widely used in omputer vision to train

obje t lassi�ers. In many appli ations, e.g. pedestrian dete tion, the dom-

inating approa h in literature is to use supervised learning, e.g. Support

Ve tor Ma hines (SVM), to train a lassi�er using labelled data. This la-

belled data is hosen su h that it represents the environment where the

lassi�er will be used. Thus, for a mobile system operating in a omplex

and un ontrolled environment, e.g. a ar, the training data set must on-

tain a great amount of variation. Colle ting and manually labelling large

amounts of data is an expensive and time onsuming pro ess.

In Paper 6, we propose to repla e or omplement real data with aug-

mented data, using the method presented in Paper 4. Augmented data an

be automati ally labelled while still exhibiting a real, and onsequently real-

isti , ba kground. The proposed solution is evaluated by training pedestrian

lassi�ers using one of the gold-standard methods in pedestrian lassi� a-

tion, spe i� ally a linear SVM and the Histogram of Oriented Gradients

(HOG), [61℄. Experimental validation is performed on real data sets and

the results are ompared to performan e obtained using real training data.

The thesis author was responsible for the problem formulation and writ-

ing the paper. The design of experiments was ondu ted jointly by the

author of this thesis and the se ond author of the paper. Note that the

development and implementation of algorithms were primarily the respon-

sibility of the se ond author, and not the author of this thesis.

39

40

Chapter 5

Con luding Remarks

This hapter states the most important ontributions and provides re om-

mendations for future resear h.

5.1 Contributions

System veri� ation of an automotive safety system must assess the orre t-

ness of system de isions in a vast array of tra� s enarios. These de isions

are based on remote sensing of the surrounding environment and onse-

quently, in luding sensors in the analysis and veri� ation methods is ru ial.

Computational methods have the potential to signi� antly improve the ver-

i� ation pro ess in terms of e.g. e� ien y and overage. This thesis fo us

on omputational methods for both de ision fun tion analysis, in luding

the dependen e on sensor errors, and methods for determining these sensor

errors.

Related to de ision fun tion analysis and veri� ation, the main ontri-

butions of this thesis are:

• Derivation of losed-form expressions for the worst ase de ision tim-

ing, in the presen e of predi tion and measurement errors, for a ol-

lision avoidan e system example. Also, losed-form expressions are

derived for robust avoidan e s enarios, i.e. s enarios whi h are guar-

anteed not to exhibit an unne essary intervention. These results are

presented in Paper 1.

• A novel set-based framework for analyzing under what onditions the

absen e of in orre t de isions may be guaranteed for a given a tive

safety de ision fun tion. In ontrast to evaluating state traje tories,

rea hability analysis and viability theory are used to ompute unsafe

and safe sets, in whi h absen e of in orre t de isions and robustness to

41

Chapter 5. Con luding Remarks

sensor errors may be guaranteed using optimization te hniques. This

framework is presented in Paper 2 and forms a generalization of the

work shown in Paper 1.

• A framework for a tive safety de ision fun tion analysis using re orded

or simulated data. E� ient methods for system performan e evalua-

tion are derived and these an be used to analyze the de ision fun tion

sensitivity to input errors, or for de ision fun tion tuning. This frame-

work is presented in Paper 3.

Related to performan e evaluation of omputer vision systems, the main

ontributions of this thesis are:

• A novel performan e evaluation approa h using augmented imagery

for evaluation of mobile omputer vision systems. Performan e is

evaluated in ollision and near- ollision s enarios, safely and non-

destru tively, while still using a real image ba kground from re orded

data. This on ept is presented in Paper 4 and the use of augmented

data is extended from performan e evaluation to training of a pedes-

trian lassi�er in Paper 6.

• An approa h for 6-DoF vehi le pose estimation using a single vehi le-

based standard amera. Visual features are omplemented by stan-

dard in-vehi le sensors and a single tra k vehi le model in a bundle

adjustment framework. The method has been validated experimen-

tally in hallenging situations at both low and high vehi le speeds.

This method is presented in Paper 5 and is an important module

needed for the framework introdu ed in Paper 4.

5.2 Dire tions of Future Resear h

There is a great need for more e� ient veri� ation methods to handle the

hallenges asso iated with future automotive safety systems. The work

presented in this thesis has inspired multiple ideas on this topi .

Sensor error models

To make full use of the theoreti al methods for performan e estimation,

presented in Papers 1-3, a urate sensor error models are needed. This

requires a quiring and pro essing large amounts of sensor data, with asso-

iated ground truth, but also proper hoi es of model stru tures. The pre-

sented framework for sensor evaluation using augmented data may prove to

be a valuable resour e.

42

5.2. Dire tions of Future Resear h

Extending rea hability methods

The dynami al models used in Paper 2 are linear and low-dimensional,

handling only a single moving obje t. Applying existing methods for rea h-

abality analysis of more omplex systems is an interesting approa h. This

ould enable the analysis of the same problem with more omplex vehi le

dynami s models and/or multiple obje ts.

Augmenting other sensors

The augmentation framework in Paper 4 has been applied primarily on

image data. Many safety systems fuse information from di�erent sensor

te hnologies, e.g. radar, laser. Thus, a natural extension would be to ex-

tend the on ept to in lude also other sensor types. This requires in-depth

knowledge of the sensor te hnology to be added and also a urate and de-

tailed modeling of the spe i� sensor used.

43

44

Referen es

[1℄ �Global status report on road safety 2013: supporting a de ade of a -

tion,� World Health Organization (WHO), Geneva, Switzerland, 2013.

[2℄ G. Ja obs, A. Aeron-Thomas, and A. Astrop, �Estimating global road

fatalities,� Transport Resear h Laboratory, Global Road Safety Part-

nership (GRSP), TRL445, 2000.

[3℄ �IRTAD Annual Report 2009,� International Tra� Safety Data &

Analysis Group (IRTAD), International Transport Forum, 2010.

[4℄ (2013, De .) OECD Statisti s. OECD. [Online℄. Available:

http://stats.oe d.org/

[5℄ F. Winston and R. Menon, �Tra� Mortality in Germany Before, Dur-

ing, and After Reuni� ation,� Annual Pro eedings / Asso iation for the

Advan ement of Automotive Medi ine, pp. 239�250, 1999.

[6℄ �Towards Zero: Ambitious Road Safety Targets and the Safe System

Approa h,� OECD, International Transport Forum, OECD Publishing,

2008.

[7℄ M. Ali, �De ision Making and Control for Automotive Safety,� PhD

Thesis, No 3413, ISSN 0346-718X, Chalmers University of Te hnology,

Göteborg, Sweden, 2012.

[8℄ W. Najm, B. Sen, J. Smith, and B. Campbell, �Analysis of Light Vehi-

le Crashes and Pre-Crash S enarios Based on the 2000 General Esti-

mates System,� U.S. Department of Transportation, National Highway

Tra� Safety Administration, John A. Volpe National Transportation

Systems Center Cambridge, DOT HS 809 573, 2003.

[9℄ �Tra� Safety Fa ts 2011 - A Compilation of Motor Vehi le Crash

Data from the Fatality Analysis Reporting System and the General

Estimates System,� National Highway Tra� Safety Administration,

National Center for Statisti s and Analysis, U.S. Department of Trans-

portation, DOT HS 811 754, Washington, DC, USA, 2012.

45

Referen es

[10℄ �Vägtra�kskador 2012 (Road tra� injuries 2012),� Tra�kanalys,

Statistik 2013:9, Sto kholm, Sweden, 2013.

[11℄ J. Treat, N. Tumbas, S. M Donald, D. Shinar, R. Hume, R. Mayer,

R. Stansifer, and N. Castellan, �Tri-level Study of the Causes of Tra�

A idents: Exe utive Summary,� U.S. Department of Transportation,

National Highway Tra� Safety Administration (Contra t No. DOT

HS 034-3-535), DOT HS 805 099, Washington, DC, 1979.

[12℄ V. L. Neale, T. A. Dingus, S. G. Klauer, J. Sudweeks, and M. Good-

man, �An overview of the 100- ar naturalisti study and �ndings,� in

Pro eedings of the 19th International Te hni al Conferen e on the En-

han ed Safety of Vehi les, 2005, pp. paper 05�0400.

[13℄ T. A. Dingus, S. Klauer, V. L. Neale, A. Petersen, S. E. Lee, J. Sud-

weeks, M. A. Perez, J. Hankey, D. Ramsey, S. Gupta, C. Bu her, Z. R.

Doerzaph, J. Jermeland, and R. Knipling, �The 100-Car naturalisti

driving study: Phase II - Results of the 100-Car �eld experiment,�

U.S. Department of Transportation, National Highway Tra� Safety

Administration, DOT HS 810 593, Washington, DC, USA, 2006.

[14℄ L. Palkovi s, A. Semsey, and E. Gerum, �Roll-Over Prevention System

for Commer ial Vehi les - Additional Sensorless Fun tion of the Ele -

troni Brake System,� Vehi le System Dynami s, vol. 32, no. 4-5, pp.

285�297, Nov. 1999.

[15℄ J. Lu, D. Messih, and A. Salib, �Roll Rate Based Stability Control-The

Roll Stability Control System,� in Pro eedings of the 20th Enhan ed

Safety of Vehi les Conferen e (ESV), paper 07-0136, Lyon, Fran e,

2007.

[16℄ R. Rajamani, Vehi le Dynami s and Control. Springer, 2006.

[17℄ M. Bayly, B. Fildes, M. Regan, and K. Young, �Review of rash ef-

fe tiveness of Intelligent Transport Systems,� TRACE Proje t, No.

027763, Deliverable D4.1.1 - D6.2, 2007.

[18℄ R. Bishop, Intelligent Vehi le Te hnology and Trends. Norwood, MA,

USA: Arte h House, 2005.

[19℄ R. K. Jurgen, Adaptive Cruise Control. SAE International, 2006.

[20℄ D. LeBlan , G. Johnson, P. Venhovens, G. Gerber, R. DeSonia,

R. Ervin, A. Ulsoy, and T. Pilutti, �CAPC: A Road-Departure Pre-

vention System,� IEEE Control Systems Magazine, vol. 16, no. 6, pp.

43�60, 1996.

46

Referen es

[21℄ M. Ruder, W. Enkelmann, and R. Garnitz, �Highway Lane Change

Assistant,� in IEEE Intelligent Vehi le Symposium. IEEE, 2002, pp.

240�244.

[22℄ A. Eidehall, J. Pohl, F. Gustafsson, and J. Ekmark, �Toward Autono-

mous Collision Avoidan e by Steering,� IEEE Transa tions on Intelli-

gent Transportation Systems, vol. 8, no. 1, pp. 84�94, Mar. 2007.

[23℄ W. Birk, M. Brännström, and D. Levin, �Method for determining a

measure for evaluating the behaviour of a driver of a vehi le,� Patent,

EP1674375, European Patent O� e, 2006.

[24℄ S. S. Bla kman and R. Popoli, Design and analysis of modern tra king

systems. Norwood, MA: Arte h House, 1999.

[25℄ B. Risti , S. Arulampalam, and N. Gordon, Beyond the Kalman �lter :

parti le �lters for tra king appli ations. Boston, Mass.: Arte h House,

2004.

[26℄ M. Darms and H. Winner, �A Modular System Ar hite ture for Sensor

Data Pro essing of ADAS Appli ations,� in IEEE Intelligent Vehi les

Symposium. IEEE, 2005, pp. 729�734.

[27℄ E. Coelingh, H. Lind, W. Birk, and D. Wetterberg, �Collision Warning

with Auto Brake,� in FISITAWorld Congress, Yokohama, Japan, 2006.

[28℄ A. Poly hronopoulos, M. Tsogas, A. Amditis, and L. Andreone, �Sensor

Fusion for Predi ting Vehi les' Path for Collision Avoidan e Systems,�

IEEE Transa tions on Intelligent Transportation Systems, vol. 8, no. 3,

pp. 549�562, Sep. 2007.

[29℄ F. Bengtsson and L. Danielsson, �A design ar hite ture for sensor data

fusion systems with appli ation to automotive safety,� in Intelligent

Transportation Systems World Congress, New York, USA, 2008.

[30℄ (2014, Jan.) DARPA Urban Challenge. Defense Advan ed

Resear h Proje ts Agen y (DARPA). [Online℄. Available:

http://ar hive.darpa.mil/grand hallenge/index.asp

[31℄ �Tra� Safety Fa ts 2008 - A Compilation of Motor Vehi le Crash

Data from the Fatality Analysis Reporting System and the General

Estimates System,� National Highway Tra� Safety Administration,

National Center for Statisti s and Analysis, U.S. Department of Trans-

portation, DOT HS 811 170, Washington, DC, USA, 2009.

47

Referen es

[32℄ M. Ri hter, H.-C. Pape, D. Otte, and C. Krettek, �Improvements in

passive ar safety led to de reased injury severity - a omparison be-

tween the 1970s and 1990s.� Injury, vol. 36, no. 4, pp. 484�8, Apr.

2005.

[33℄ (2014, Jan.) The European New Car Assessment Programme

(EuroNCAP). [Online℄. Available: http://www.euron ap. om/

[34℄ D. Burton, A. Delaney, S. Newstead, D. Logan, and B. Fildes, �Ef-

fe tiveness of ABS and vehi le stability ontrol systems,� Report No.

04/01, Royal Automobile Club Of Vi toria (RACV), 2004.

[35℄ S. A. Ferguson, �The e�e tiveness of ele troni stability ontrol in re-

du ing real-world rashes: a literature review.� Tra� injury preven-

tion, vol. 8, no. 4, pp. 329�38, De . 2007.

[36℄ M. Lindman, A. Ödblom, E. Bergvall, A. Eidehall, B. Svanberg, and

T. Lukaszewi z, �Bene�t estimation model for pedestrian auto brake

fun tionality,� in Pro eedings of the ESAR Conferen e, no. 300961,

2010.

[37℄ K. Kusano and H. Gabler, �Safety bene�ts of forward ollision warning,

brake assist, and autonomous braking systems in rear-end ollisions,�

IEEE Transa tions on Intelligent Transportation Systems, vol. 13,

no. 4, pp. 1546�1555, De . 2012.

[38℄ E. L. Lehmann, Testing statisti al hypotheses, 2nd ed. Pa i� Grove,

Calif.: Wadsworth & Brooks/Cole Advan ed Books & Software, 1991.

[39℄ J. Neyman and E. S. Pearson, �On the Use and Interpretation of

Certain Test Criteria for Purposes of Statisti al Inferen e: Part I,�

Biometrika, vol. 20A, no. 1/2, pp. 175�240, 1928.

[40℄ R. O. Kuehl, Design of Experiments: Statisti al Prin iples of Resear h

Design and Analysis, 2nd ed. Pa i� Grove : Duxbury-Thomson

Learning, 2000.

[41℄ L. Danielsson, �Tra king and radar sensor modelling for automotive

safety systems,� PhD Thesis, No 3064, ISSN 0346-718X, Chalmers Uni-

versity of Te hnology, Göteborg, Sweden, 2010.

[42℄ J. Hillenbrand, A. M. Spieker, and K. Kros hel, �A Multilevel Collision

Mitigation Approa h - Its Situation Assessment, De ision Making, and

Performan e Tradeo�s,� IEEE Transa tions on Intelligent Transporta-

tion Systems, vol. 7, no. 4, pp. 528�540, De . 2006.

48

Referen es

[43℄ P. Zheng and M. M Donald, �The e�e t of sensor errors on the perfor-

man e of ollision warning systems,� in IEEE Intelligent Transportation

Systems Conferen e, vol. 1. Pis ataway, NJ, USA: IEEE, 2003, pp.

469�474.

[44℄ L. Yang, J. Yang, E. Feron, and V. Kulkarni, �Development of a

performan e-based approa h for a rear-end ollision warning and avoid-

an e system for automobiles,� in IEEE Intelligent Vehi les Symposium,

2003, pp. 316�321.

[45℄ O. Gietelink, B. De S hutter, and M. Verhaegen, �Probabilisti valida-

tion of advan ed driver assistan e systems,� in Pro eedings of the 16th

IFAC World Congress, vol. 19, Prague, Cze h Republi , 2005.

[46℄ W. Burger, M. Barth, and W. Sturzlinger, �Immersive Simulation for

Computer Vision,� in joint 19th AGM and 1st SDRV workshop Visual

Modules. Maribor, Slovenia: Oldenbourg Press, 1995, pp. 160�168.

[47℄ A. S. Glassner, Prin iples of Digital Image Synthesis, 1st ed. Morgan

Kaufmann, 1995.

[48℄ (2014, Jan.) PreS an. TASS International. [Online℄. Available:

https://www.tassinternational. om/pres an

[49℄ (2014, Jan.) v-TRAFFIC. VIRES Simulationste hnologie GmbH.

[Online℄. Available: http://www.vires. om/Produ ts_ToolChain.htm

[50℄ Bos h Automotive Handbook, 7th ed. Robert Bos h GmbH, 2007.

[51℄ P. C. Ca iabue, Ed., Modelling Driver Behaviour in Automotive En-

vironments. Springer, 2007.

[52℄ H. Pa ejka, Tyre and Vehi le Dynami s, 2nd ed. Elsevier Ltd, 2006.

[53℄ R. Kiefer, D. Leblan e, M. Palmer, J. Salinger, R. Deering, and

M. Shulman, �Development and Validation of Fun tional De�nitions

and Evaluation Pro edures for Collision Warning/Avoidan e System,�

National Highway Tra� Safety Administration, DOT HS 808 964,

Washington, DC, USA, 1999.

[54℄ M. Distner, M. Bengtsson, T. Broberg, and L. Jakobsson, �City Safety

- A system addessing rear-end ollisions at low speeds,� in Pro eedings

of the 21st International Te hni al Conferen e on the Enhan ed Safety

of Vehi les (ESV), Stuttgart, Germany, 2009.

49

Referen es

[55℄ J. Hillenbrand and K. Kros hel, �A Study on the Performan e of Un o-

operative Collision Mitigation Systems at Interse tion-like Tra� Sit-

uations,� in IEEE Conferen e on Cyberneti s and Intelligent Systems.

IEEE, Jun. 2006, pp. 1�6.

[56℄ O. Gietelink, J. Ploeg, B. De S hutter, and M. Verhaegen, �Develop-

ment of advan ed driver assistan e systems with vehi le hardware-in-

the-loop simulations,� Vehi le System Dynami s, vol. 44, no. 7, pp.

569�590, Jul. 2006.

[57℄ K. Lee and H. Peng, �Evaluation of automotive forward ollision warn-

ing and ollision avoidan e algorithms,� Vehi le System Dynami s,

vol. 43, no. 10, pp. 735�751, O t. 2005.

[58℄ J. Wood o k, P. G. Larsen, J. Bi arregui, and J. Fitzgerald, �Formal

methods: Pra ti e and Experien e,� ACM Computing Surveys, vol. 41,

no. 4, pp. 1�36, O t. 2009.

[59℄ I. Mit hell, �Comparing forward and ba kward rea hability as tools

for safety analysis,� in 10th International Workshop, Hybrid systems:

omputation and ontrol. Pisa, Italy: Springer Berlin Heidelberg, 2007,

pp. 428�443.

[60℄ J.-P. Aubin, A. M. Bayen, and P. Saint-Pierre, Viability Theory - New

Dire tions, 2nd ed. Berlin, Heidelberg: Springer Berlin Heidelberg,

2011.

[61℄ N. Dalal, �Finding people in images and videos,� PhD Thesis, Institut

National Polyte hnique de Grenoble / INRIA Rh�ne-Alpes, Grenoble,

2006.

50

Documents

Thesis - publications.lib.chalmers.sepublications.lib.chalmers.se/records/fulltext/193048/193048.pdf · training ob ject classi ers. ords: Keyw e, Automotiv e ctiv A, y Safet Semi-Autonomous