www.pegasusprojekt.de
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
PEGASUS Method for Testing HIGHLY Automated Driving Functions
PEGASUS enables the connection to existing tools.
How are existing SiL-Tools integrated into
the toolchain and the test field?
Simulation Proving
Ground
Real
World Test
Test coverage >> 10,000 simulation runs per scenario, especially critical scenarios
Effort for qualification test equipment /
tools / methods
High effort for simulation model parametrization und validation
(driving dynamics, sensor, actuators, traffic agents) as well as
overall verification of the simulation chain with real driving data.
Effort for test setup / preparation Low, but depending on the degree of test automation
Effort for test execution Low, "only" computing power necessary
Standardization methodology / tools
Methodology given by Pegasus, but no, or only conditional,
standardization of manufacturer-specific simulation models and
tools
Current limits Ego vehicle, environment and sensors not yet fully modeled in the
model
Goal: Test of all scenarios from the database and their variations to
identify collision-relevant scenario characteristics for the automated
system:
- Number of test cases: „BIG“ >> ~ 10,000
Input: Scenarios and their parameters (including distributions), Pass-
criteria / criticality metrics, original ECU code
Output: Scenarios assessed against pass criteria and, if applicable,
their probability of failure (in the case of accidents)
B
O
O
T
H
2
3
B
O
O
T
H
2
7
The objective of PEGASUS Testing was to
develop the PEGASUS method regarding
completeness, correctness and consistency.
The module SIMULATION is an enabling
element / tool.
www.pegasusprojekt.de
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
PEGASUS enables the connection to existing tools. How are existing SiL-Tools integrated into the toolchain and the test field?
Motivation:
• Using a common toolchain to execute tests
• Common interfaces inside the toolchain enables an interchangeability of tools
• Using in PEGASUS developed Sensor- Models and Stochastics- Module
Common Toolchain
Test Automation
Test-Case Generation
Stochastics-Module
Test Control
Scenario-
database
Simulationstool
Environment Model
Sensor
Models
Overall
Result
Result
Evaluation
OSI or OEM-
Connection
Transpiler OpenDrive-
Generator
Parametrization
AD
Function
Vehicle Model
Logical Scenario
Logical „Simplified
Road“
+ Parameter description
file
+ Evaluation criteria
OSI or OEM-
Connection
Test Results in
mat or hdf5
Concr.
OpenScenario
Concr. OpenDrive
www.pegasusprojekt.de
Visualization of Results
Simulation Run
Min
TT
C (
Obje
ctive F
unction)
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
Test-Case Description
Pass-Criterium: no accident, i.e.
time-to-collision > 0 s
Varied Parameters:
Video
Simulation-Tool DB
1 logical scenario
+ parameters
+ test criteria
Functions + Models
Sim-
Results:
JSON +
hdf5 „1000“ concrete
scenarios
Results Evaluation (local)
OSI
Sensor Models
Overall-Results
Test Automation
Stochastics-Module
BMW - Realization
Critical and Uncritical Scene
⇒ 𝑇𝑇𝐶min= 0.83 s
Critical Scene:
L4_Ego_Speed_Initial 31.5 m/s
L4_Obj_Position_Initial 150.3 m
L4_Obj_Speed_Final 14.9 m/s
L4_Obj_Speed_Initial 30.3 m/s
Convergence Behavior of Optimizer
Ego
Parameter Lower
Bound
Upper
Bound
L4_Ego_Speed_Initial 20 m/s 36 m/s
L4_Obj_Position_Initial 120 m 160 m
L4_Obj_Speed_Final 12 m/s 33 m/s
L4_Obj_Speed_Initial 12 m/s 33 m/s
L1_Road_Length 3000 m 4000 m
www.pegasusprojekt.de
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
-Realization
Simulation-Tool DB
1 logical scenario
+ parameters
+ test criteria
Functions + Models
Sim-
Results:
JSON +
mat or
hdf5 „10000“ concrete
scenarios
Results Evaluation (local)
OEM
Sensor Models
Overall-Results
Test Automation
Stochastics-Module
Test-Case Description
Cut out vehicle standing
vehicle
Pass-Criterium:
No accident:
Distance >0 m EGO
Result:
Varied Parameters :
TTC (max: green, min: red)
50
57.5
65
72.5
80
0.6 0.7 0.8 0.9 1.0
𝜇
𝑣 [
km
/h]
1 2 Uncritical case Relevant case
Video 1 Video 2
1
2
Parameters of uncritical case:
𝑚 = 1600 kg; 𝑠 = 5 s; 𝑑𝐴 = 20 m; 𝑣 = 50 km/h;
𝑇𝑇𝐶min = 1.892 s
Parameters of relevant case:
𝑚 = 2200 kg; 𝑠 = 2 s; 𝑑𝐴 = 0 m; 𝑣 = 80 km/h;
𝑇𝑇𝐶min = 0.676 s
Parameter Lower
Bound
Upper
Bound
Mass EGO 𝑚 1600 kg 2200 kg
Cut-out duration 𝑠 2 s 5 s
Cut-out waypoint 𝑑𝐴 0 m 20 m
Velocity 𝑣 50 km/h 80 km/h
Friction value 𝜇 0.6 1.0
Simulations No
accident accident
„relevant“
(TTC < 1,0 s)
3125 2759 366 1725
www.pegasusprojekt.de
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
-Realization
Simulation-Tool DB
1 logical scenario
+ parameters
+ test criteria
Functions + Models
Sim-
Results:
JSON +
mat or
hdf5
Testnachsteuerung
„10000“ concrete
scenarios
Results Evaluation (local)
OEM
Sensor Models
Overall-Results
Test Automation
Stochastics-Module
2 Relevant Case
Time [s]
TT
C [
s]
𝑣 [
km
/h],
𝑎𝑉
𝑒𝑟
𝑧 [
dm
/s²]
Detail plot relevant case
Proving ground Simulation
Matching proving ground/simulation:
• Situation passed
uncritical Case
www.pegasusprojekt.de
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
-Realization
Simulation-Tool DB
1 logical scenario
+ parameters
+ test criteria
Functions + Models
Sim-
Results:
JSON +
mat or
hdf5 „10000“ concrete
scenarios
Results Evaluation (local)
OEM
Sensor Models
Overall-Results
Test Automation
Stochastics-Module
Quality criterion regarding the probability of failure
• Standard deviation of failure probability
• Confidence interval
Simulations
7x10-6
Pro
ba
bili
ty o
f fa
ilure
<10% Sta
nd
ard
de
via
tio
n
Stepwise approach of the failure
region until a specific safety limit
has been reached
TTC < 1.0: pf = 0.558906
TTC < 0.5: pf = 0.210416
TTC < 0.4: pf = 0.014846
TTC < 0.3: pf = 0.000061
TTC < 0.2: pf = 0.000024
TTC < 0.1: pf = 0.000014
TTC < 0.0: pf = 0.000007
Determination of the probability of failure by using exemplary data and distributions
𝜇
𝜇
𝜇
𝑣 [km/h]
𝑣 [km/h] 𝑣 [km/h]
Characterization of the parameter
space
Distribution of cumulated
probability
Distribution of cumulated
probability of violating criteria
probability probability TTC_Vcoll
0.6 0.7
0.8
0.9 1.0
0.6
0.7
0.8
0.9
1.0 0.6
0.7
0.8 0.9 1.0
50 57.5
65 72.5
80
50 57.5 65 72.5 80
50 57.5
65 72.5
80
0.25
0.15
0.05
0.000006
0.000004
0.000002
2.0
-4
-2
0
www.pegasusprojekt.de
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
-Realization
Simulation-Tool DB
1 logical scenario
+ parameters
+ test criteria
Functions + Models
Sim-
Results:
JSON +
hdf5
Testnachsteuerung
„10000“ concrete
scenarios
Results Evaluation (local)
OEM
Sensor Models
Overall-Results
Test Automation
Stochastics-Module
Visualization of Results
Examples of Critical and Uncritical Cases
Pass-Criterium: no collision, i.e. RSS < 1
2
Ego SCT
1
Ego Lane Change
Varied Parameters Lower
Bound
Upper
Bound Dep.
Distance to Ego d1 -50 m 50 m -
Distance to Ego d2 -50 m 50 m d1 > d2
Distance to Ego dSCT 100 m 200 m -
Brake Trigger Dist. dSctTrig. 40 m 100 m -
Brake Accel. aSCT 1 m/s² 10 m/s² -
Optimization Generations Simulations
Maximize: Fitness(RSS) 30 1500
Critical Case d1 = 30 m d2 = 1 m Fit 𝑅𝑆𝑆 = 100
Uncritical Case d1 = 30 m d2 = -30 m Fit 𝑅𝑆𝑆 = 0
www.pegasusprojekt.de
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
-Realization
Simulation-Tool DB
1 logical scenario
+ parameters
+ test criteria
Functions + Models
Simulation-
data „10000“ concrete
scenarios
Results Evaluation (local)
OSI/
OEM
Continental Camera Model
Overall-Results
Test Automation
Stochastics-Module
Optimization towards a critical scenario
Pass-Criterium: no accident, i.e.
time-to-brake > 0 s
Varied Parameters:
Cut-in from right
Parameters Lower
Bound
Upper
Bound
Object position (initial) 0 m 260 m
Object speed (initial) 5 m/s 48 m/s
Object speed (final) 4 m/s 50 m/s
Ego speed (initial) 18 m/s 36 m/s
cut-in duration 𝑡 2 s 9,4 s
Ego
In several iterations, the particle swarm optimizer
aims to optimize the parameters of the scenario to
generate a critical one.
Visualization of Results
www.pegasusprojekt.de
TESTING AND SAFEGUARDING – Booth No. 21
SIMULATION –
SOFTWARE-IN-THE-LOOP
-Realization
Simulation-Tool DB
1 logical scenario
+ parameters
+ test criteria
Functions + Models
Simulation-
data „10000“ concrete
scenarios
Results Evaluation (local)
OSI/
OEM
Continental Camera Model
Overall-Results
Test Automation
Stochastics-Module
Visualization of Results