1 1
Utilization of Simulink Verification and
Validation (V&V) and Simulink Design
Verifier (SDV) for HVAC Controls Software
Arun Chakrapani Rao, Rupesh Kakade, Mohan Murugesan
HVAC Controls Algo And Readiness, Electronic Controls and Software Engineering,
General Motors Technical Centre India Pvt. Ltd., Bengaluru, India
2 2
Outline
Readiness Testing and Core Algorithm work overview
HVAC Production-oriented testing (ECU, model)
What is structural coverage? Why use it?
What are model coverage metrics?
Overview of work done and results
Recommendations for incremental improvements
Potential for Automatic Test case Generation
Potential for Property Proving
Current challenges and some proposed workflows
3 3
Regulates the air temperature, flow rate and moisture
throughout the vehicle interior (by considering the effects of
ambient temperature, sun load, and heat transfer mechanisms)
in real-time
Challenges overcome using Model-Based Designs in Development
Unit level and integrated software verified early
Same software deployed to many different vehicles by simply calibrating
parameters such as vehicle dimensions
Same s/w also deployed to multiple controllers with varying hardware and
software architecture (Non-standard or standard ones like AUTOSAR)
Integration of legacy software and the model-based software possible for
vehicles nearing production
Parallel development of several components possible
Production code auto-generated, compiled and targeted efficiently and
accurately
HVAC Control Software
4 4
Aero Shutter Control
Combinational logic for on/off control of magnetically driven set
of flaps which close front end airflow paths to enhance vehicle
aerodynamics
Cabin Air Recirculation Control
Physics-based design to ensure minimal compressor work while
maintaining thermal comfort of the occupants
Repeated calculations (physical properties) implemented by
creating and using our own library blocks
Functional verification using approximate plant model for closed-
loop simulation
Standard test inputs derived from requirements and vehicle like
scenarios (vehicle test data)
HVAC Control Software – Example Components
5 5
Test cases mainly guided by Requirements
Both Manual and Automated Testing
Peer Reviews
Simulation Model Testing
CPP Unit Testing
Regression, Delta Change, Acceptance Testing (Hardware Bench/HIL)
Current Testing in Production
6 6
Simulation Model Testing
Performed at the unit level
Closed-loop simulation of the control system with approximate plant model
Detailed functional verification based on requirements, internal standards and over several vehicle
like scenarios
Performed using standard test inputs developed once
CPP Unit Testing
Simulation model I/Os are automatically translated using a MATLAB M-script
Verifies interface between the automatically generated code from the model and the wrapper
interface code and the buried conversion mathematics
Performs acceptance check for example, requirements, rounding errors etc. with the use of CPP
asserts
Plant models for closed-loop simulation
Simulation and early verification possible
Core Algo Development and testing
of various
HVAC component models
Core Algorithm Modeling Group
7 7
Regression Test
Detailed Component level verification
Performed once on a Model Year Software
Performed using automated test scripts on dSPACE HIL
Delta Change Verification
Verifies the specific delta change on every release
Manual / automated test scripts
Acceptance Test
Verifies the system level functionalities on every release
Performed using automated test scripts on dSPACE HIL
Readiness
Testing of
HVAC components
at the
integrated ECU
Readiness Group
8 8
Shift towards early model-based V&V
9 9
I1 O
The output shall be set to 100 times the sensor input.
I2
O
If sensor input is valid, the output shall be 100 times,
else a fail safe value of 180 should be output.
Choices of input values affect the calculations done downstream
Overall coverage gets influenced by such choices!
Sample inputs: -15, 10, 45
Boundary values: -5, -4.9, 40, 40.1
Valid and invalid cases also
I1
-5 ≤I1≤ 40
Structural Coverage
10 10
i1 i2
i20
o1 o2
o8
.
. .
Tested enough?
Irrespective of the test design techniques, in real-life scenario,
model coverage assessment becomes necessary and crucial!
more inputs
more outputs
more conditions
complex conditions
more simulation paths
Req1 Req2 . . ReqN
Real-life Requirements:
Numerous ;
In Natural Language
Real-life Code or Model
Structural Coverage
11 11
Find out gaps in requirements-based test cases
Identify gaps in requirements
Identify unreachable parts of the model (or code)
Identify unintended functionality
ISO/FDIS 26262-6:2010(E)
Why Structural Coverage?
12 12
Relevant Mathworks toolbox:
Simulink Verification and Validation toolbox (V&V toolbox)
Structural Coverage Assessment
Principle Practice for Production
13 13
Condition Coverage
Analyzes blocks that output logical combinations of their inputs
Logical Operator blocks, Stateflow transitions
2 AND blocks;
2*2, 3*2
c1
c2
c3
c4
V1
V2
V3
Model Coverage Metrics – Condition Coverage
14 14
Cal value was T in all test cases
c5
c6
k1
k2
k3
V4
V5
Model Coverage Metrics – Condition Coverage
15 15
No True for one of the AND conditions
=> making it T will cover 2 more conditions (for the AND, OR together)
k4
k5
k6
k7
k8
k9
k10
k11
V6
V7
V8
V9
V10
V11
Model Coverage Metrics – Condition Coverage
16 16
Decision Coverage
Analyzes model elements that represent decision points
Switch block, Stateflow states
V12
V13
V14
V15
Model Coverage Metrics
17 17
MCDC
Independence of logical block inputs and transition conditions
INIT -> NORMAL
/* T1_2 */
[(Mode1 != C1) &&...
(Mode2 != C2) && ...
(Mode2 != C3) && ...
( (V1_Err > 0) || ...
f1() || ...
!f2() )]
CC 75% (3/4)
T F
C1 V1_Err < K1_Min
C2 K1_Max > V2
DC 100% (2/2)
MCDC 50% (1/2 conditions reveresed the outcome)
C1 V1_Err < K1_Min
C2 K1_Max > V2
Out C1 && C2 T Out F Out
C1 TT Fx
C2 TT (TF)
FUNC=f1
FUNC=F; FUNC=T;
V1_Err < K1_Min && ..K1_Max > V2
Stateflow
Graphical
Function
with a condition of
the form C1 && C2
Model Coverage Metrics
18 18
INIT -> NORMAL
/* T1_2 */
[(Mode1 != C1) &&...
(Mode2 != C2) && ...
(Mode2 != C3) && ...
( (V1_Err > 0) || ...
f1() || ...
!f2() )]
CC 75% (3/4)
T F
C1 V1_Err < K1_Min
C2 K1_Max > V2
DC 100% (2/2)
MCDC 50% (1/2 conditions reveresed the outcome)
C1 V1_Err < K1_Min
C2 K1_Max > V2
Out C1 && C2 T Out F Out
C1 TT Fx
C2 TT (TF)
V1=f1
V1=F; V1=T;
V1_Err < K1_Min && ..K1_Max > V2
Date : 27-Jun-11
Test Cases C12 Test Cases - adapted from HIL test cases
Component : C12 Test Cases - adapted from HIL test cases
Software V1, V1.1 (new)
Requirements
Document :
HVAC Control - C12
COVERAGE ANALYSIS REPORTProject name : HVAC
Tester Name : Arun Rao
0
10
20
30
40
50
60
70
80
90
100
C1
2 S1 S11
C1
S11
1
S11
11
S11
12
S11
13
S11
14
S11
15
S11
15
1
S11
15
2
S11
16
S11
16
1
S11
16
2
S11
2
S11
21
S11
22
S11
3 S2 S21
S22
S23
S3 S31
S32
S33
Condition Coverage
CC
Internal tool for test automation
Excel sheet textual description of steps
MATLAB M Scripts for
automation around the
utilization of the Simulink
V&V toolbox for
structural coverage
assessment
Recommendations to improve test cases
Srl. No. Recommendation Expected effect
1 Set V1 > K1 function f1 will get 100% CC (See sheet f1)
2 Set V2_MinMxAirSetPt > K2
function f2 gets 100% CC (See sheet Other Graphical Funcs
50%)
3 Set V3_MaxMxAirSetPt > K3-C1
function f3 gets 100% CC (See sheet Other Graphical Funcs
50%)
4 V4 >= K4 function f4 will get 100% CC (See sheet f4)
6 Set V5 to 9, 12, 20 and 28 Distribution modes D5, D7, D8 and D12 will be reached
1 Modify speed values in Test 6 Sub Test 9 Covers Transition TRANSxyz
2
Change Validity value V5 to True from False in
Test 7 SubTest 2 Achieves the goals for this test case
3
Look into cal. and/or validity values for Test 7
SubTests 3 to 10 Reaches various substates of STATEabc
1
Correction needed for test cases Test 3 SubTest
1: K1 is being set to 100000 but it's max. is
defined as 15000 in the spec.
1
K2 has to be set to 0 for some test cases so that
states transitions such as from STATE_S1 to
STATE_COOL_DOWN, STATE_COOL_DOWN to
STATE_NORMAL, STATE_NORMAL to STATE_INIT
become possible. Additional state coverage
Overview of automation done around V&V toolbox
19 19
Date : 27-Jun-11
Test Cases C12 Test Cases - adapted from HIL test cases
Component : C12 Test Cases - adapted from HIL test cases
Software V1, V1.1 (new)
Requirements
Document :
HVAC Control - C12
COVERAGE ANALYSIS REPORTProject name : HVAC
Tester Name : Arun Rao
0
10
20
30
40
50
60
70
80
90
100
C1
2 S1 S11
C1
S11
1
S11
11
S11
12
S11
13
S11
14
S11
15
S11
15
1
S11
15
2
S11
16
S11
16
1
S11
16
2
S11
2
S11
21
S11
22
S11
3 S2 S21
S22
S23
S3 S31
S32
S33
Condition Coverage
CC
Low coverage here!
Report – Overview sheet
20 20
Srl. No. Recommendation Expected effect
1 Set V1 > K1 function f1 will get 100% CC (See sheet f1)
2 Set V2_MinMxAirSetPt > K2
function f2 gets 100% CC (See sheet Other Graphical Funcs
50%)
3 Set V3_MaxMxAirSetPt > K3-C1
function f3 gets 100% CC (See sheet Other Graphical Funcs
50%)
4 V4 >= K4 function f4 will get 100% CC (See sheet f4)
6 Set V5 to 9, 12, 20 and 28 Distribution modes D5, D7, D8 and D12 will be reached
1 Modify speed values in Test 6 Sub Test 9 Covers Transition TRANSxyz
2
Change Validity value V5 to True from False in
Test 7 SubTest 2 Achieves the goals for this test case
3
Look into cal. and/or validity values for Test 7
SubTests 3 to 10 Reaches various substates of STATEabc
1
Correction needed for test cases Test 3 SubTest
1: K1 is being set to 100000 but it's max. is
defined as 15000 in the spec.
1
K2 has to be set to 0 for some test cases so that
states transitions such as from STATE_S1 to
STATE_COOL_DOWN, STATE_COOL_DOWN to
STATE_NORMAL, STATE_NORMAL to STATE_INIT
become possible. Additional state coverage
Sample recommendations for C12, C8, C2, C1
Recommendations
21 21
Srl. No. Component Ver
Total test
cases CC
Cyclomatic
Complexity
Total
Conditions
in the
Model
Conditions
Covered
by Test
Cases
1 C1 v2 15 94 7 54 51
2 C2 v1 25 83 37 36 30
3 C3 v2 3 83 49 196 162
4 C4 v2 32 66 55 412 270
5 C5 v2 6 65 57 54 35
6 C6 v2 10 76 94 130 99
7 C7 v2 30 86 100 404 346
8 C8 v1 60 57 101 116 66
9 C9 v2 28 48 130 164 78
10 C10 v1 21 76 139 372 283
11 C11 v2 55 65 211 674 437
12 C12 v1 47 83 234 316 262
13 C13 v2 14 70 275 858 604
14 C14 v2 55 68 302 302 204
15 C15 v2 165 51 322 528 268
16 C16 v1 53 61 353 758 460
Coverage for various components
22 22
Coverage for various components
23 23
Original test cases created for the hardware bench/HIL
Extra effort to recreate test cases; capture intention of
the tester
Solution for the future: Model-level test cases to be
updated/created/maintained for Readiness testing
Utilization of the results requires some extra effort and
time from component owners
Ideally suited for independent V&V activities to assist
Production work and teams initially
Some learnings – Simulink V&V toolbox
24 24
Some components might have a very good coverage
already
> 80% Condition Coverage
Small models/low complexity: C1, C2, C3
Test cases have evolved well over time: C7, C12
Some components have lower coverage
Only around (50%-60%)
Larger models/higher complexity
Much large number of test cases also haven’t helped; so, gaps
are important
Irrespective of the above, structural coverage assessment is necessary!
Improvements can only happen after assessment!
Some key take-always
25 25
SDV – Automatic Test Generation (ATG)
The toolbox can generate test cases automatically as per user-
defined coverage requirements
SDV – Property Proving (PP)
A technique to check if the model satisfies critical
requirements without writing numerous test cases
Simulink Design Verifier (SDV) toolbox
26 26
Use Simulink Design Verifier for Automatic Test case Generation!!
SDV - ATG
27 27
Use Simulink Design Verifier ATG capability to improve test cases further
SDV - ATG
28 28
ATG test cases to supplement existing test cases
First assess coverage of existing test cases
Identify gaps to increase coverage via self-designed test
cases if desired
Use SDV ATG for even further improvements
Existing models
May have unsupported constructs; Use automatic stubbing
May encounter some scalability issues
Use ATG for selective models/subsystems
Where complexity is involved
To find out if any parts of the model are unreachable
Some points to note
29 29
Relevant Mathworks toolbox:
Simulink Design Verifier (SDV)
Design Verification
Principle Practice for Production
30 30
Aero Shutter is
never closed if
the speed is less than 50 kmph.
Always, if the Aero Shutter is closed,
it implies that
the coolant temperature is less than some defined maximum (92 degC).
Once ON, heater coolant pump should remain ON
for at least 30s
even if
the request becomes FALSE in the meantime.
Some Example Properties for Proving
31 31
Indicate some workflows
for V&V and SDV toolboxes
through short demos
Demos
32 32
Structural coverage assessment using the V&V toolbox
important to improve on test cases
Standards recommend it - not just for critical applications
Workflows could be tailored and adopted to suit particular
production environments
SDV toolbox capabilities could be used to improve test
cases via ATG for uncovered objectives
In addition, Property Proving feature of the SDV toolbox
complements traditional testing approaches to increase
overall confidence
Final Conclusions