© TESIS DYNAware GmbH, www.tesis-dynaware.com 1

GPU-supported real-time sensor modelling

for ADAS development in Simulink

F. Chucholowski, C. Gnandt, C. Hepperle

Stuttgart

2

ADAS and AD

Challenge

Uncounted numbers of different traffic situations

Rising complexity of the systems

Rising complexity of road test setup

Solution

Virtual driving tests MiL / SiL / HiL

as a complement of real road tests

Benefits

Reproducibility

Safety

Frontloading

3

Virtual test drives for ADAS and AD

Situation

Object detection and classification

Information from sensor data fusion

Realistic enough simulation especially

regarding environmental model

Real-time capability

Requirements

Complete complex and configurable virtual world

Closed-loop of

virtual world

vehicle under test

controller functions

4

Sensors

Sensors are crucial for environment perception

of ADAS and AD algorithms

Virtual world must be recognized by all sensor types

For different applications different sensor modelling approaches are

suitable (e.g. stochastic, deterministic, ideal, measurement based)

Consider real-time capability for sensor modelling

Environment Sensors

electromagnetic optic acoustic

Car-2-X Radar Video Lidar Laser Infrared Ultrasonic

5

Main Idea - DYNAanimation

Use solutions from computer gaming industry

Use gaming graphics engine for visualization of a virtual 3D world

Use high-performance GPU computation for sensor modeling and other

suitable calculations

Use open gaming development environment as an asset for specific

application customization

Connect virtual world to overall simulation system

(e.g. MATLAB / Simulink)

Solution: Unity

Established within gaming industry

State-of-the-art computer graphics

Complete SDK + unity ecosystem, open system

Support of standard interfaces and technologies

6

Typical elements of the virtual 3D world

TerrainRoad

Weather Conditions

LightTraffic Signs

Traffic members

Buildings

VuT

Controller Information

7

Intelligent Objects

Powerful possibility to add content

Objects with own control / physics

Full scripting access

Full configuration

GPU-based calculation

8

Ambient conditions

Light

Fog

Rain

Snow

9

Display Objects

Powerful possibility to add content

Information display

Custom product visualization

10

Sensor modelling with unity

Full Unity support

Full scripting access, e.g.

Custom processing

11

Camera based sensors

Adjustable field of view (sensor size and focal length)

Adjustable resolution

Pixel shader to modifiy image

HDR RGB images

Color filter

Lens distortion

12

Time-of-Flight based sensors

LIDAR, RADAR

Configurable source (transmitter)

Propagation characteristics via light cookies

Damping

Reflection behaviour via materials

Intensity information

Depth information inherently available

Occlusion taken into account

Object IDs

13

Simulink s-functions

Bidirectional

Distributed computing and visualization

Real-time capability

Simulink Connection

Simulation Control, Motion Signals, Sensor Control

Time Stamp, Sensor Signals, Objects Interaction

Controller Development

Plant model

Animation of virtual world

Sensor simulation

14

LKAS with image processing (1)

Motivation

Lane Keeping Assist System (LKAS) test bench

with control unit and camera

Limitations regarding

e.g. optical effects

Solution

Closed loop SiL

controller tests

with direct image processing

15

LKAS with image processing (2)

TMW LKAS demo

DYNAanimation

Camera Sensor

Transmit Image to Simulink

Simulink

DYNA4 vehicle dynamics

Camera image receiver

Lane detection algorithm

Controller demands fed back

to vehicle dynamics model

Varying weather conditions

with and without rain

16

Parallel Parking with Ultrasonic Sensors (1)

Application: Parking Assist

Control algorithm and plant model

simulated in Simulink

Ultrasonic sensors simulated in 3D

animation

0.2 0.4 0.6

0.8 1

30

210

60

90270

300

150

330

180

0

horizontal

vertikal

horizontal

vertical

normalized

intensity

Sensor

48 kHz sound, 20 Hz sampling

Field of view from 0.2 to 2 m,

width +/- 60°, height +/- 30°

Dynamic membrane stimulus

and attenuation

0.25 m over ground

17

Parallel Parking with Ultrasonic Sensors (2)

Light sourceIntensity

image capture

Depthimage capture

Intensity-depth-

histogram

18

X [m]

Y [m]

Normalized

measured

intensity

Contour

of parked

vehicle

Curb stone

Simulation accuracy (Front protection scenario)

19

Conclusion

Simulink very suitable for vehicle dynamics and controller algorithms

Complex 3D environment set up in 3D animation

Source for user’s display

Powerful and flexible sensor modelling

Effects, such as reflection intensity or occlusion, handled automatically

Perfect complement to Simulink