Need To Automate Test And Integration Beyond Current Limits?

Jakob Engblom, Product Line Manager, System Simulation, Wind River

Graham Morphew, Sr. Director of Product Management, System Simulation, Wind River

Need To Automate Test And Integration Beyond

Current Limits? Use Simulation

Moderator:

Brandon Lewis, OpenSystems Media

Speakers:

Agenda

Housekeeping

Presentation

Questions and Answers

Wrap-up

3 © 2015 Wind River. All Rights Reserved.

Need To Automate Test and Integration Beyond Current Limits? Use Simulation

3

Maximize Velocity – Webinar Series

Agile, Continuous Practices and DevOps

Graham Morphew & Jakob Engblom

Wind River


Embedded Software Trends

Software driving majority of value and innovation

IoT driving future differentiation, value & service

Customer expectations have changed to demand new features more often and faster

Pushing software organizations to change and be more agile and efficient

ChangeChange

Change


Faster Delivery Of Software To Customers

Agile Continuous Practices Cross Disciplined Teams DevOps

New development practices and organization styles getting established for embedded software development

Key Challenges:

• Access

• Collaboration

• Automation

Automation is key because manual processes don’t scale.

Covered in previous webinar

Covered in upcoming webinar


Developer changes or adds code

Build and test: single unit

Build and test: multiple units

Build and test: entire system

Code OK

Immediate

Coffee break or lunch

Daily

Continuous Practices

Quick feedback to developers

Find different classes of errors at each level

Avoid big-bang integration & associated problems

Avoid regressions by rerunning old tests

= higher quality and shorter lead times

(Typical time frames)

Needs to be Automated!


The Hardware Test Lab

bus, network, …

Data generator or world model, running on hardware

Target provisioning and control

JTAG, serial, Ethernet, flash programmer, …

System under test

Test manager Build server

Lab test rig – critical resource

Automation based on coordinating

several separate pieces of hardware


The Challenge of Automation in Hardware

Hardware test lab is difficult to build, scale and maintain

How many lab setups can you afford?

Reconfiguration is manual, even when tests are automatic

Automation is often brittle

Impact• Large equipment expenses

• Unable to scale

• Automation of failure


Simulation system – all just software

The Simulation Test Lab

Simulated bus, network, …

Data generator or world model, running as software only

Target provisioning and control

System under test - simulated

Test manager Build server

Simulator tools

Model of the world

Use the existing test management system

Automation based on software control, not hardware control!


Simulation Makes Automation a Reality

Automate any operation on the target system

Test in the context of any system or subsystem

Parallelize, scale & reconfigure test suites in any way

Use any server to run any test – flexibility*

Send the context of test failure to developers to solve*

Value using simulation for automation• Make embedded test automation possible & cost effective

• Do more, effective testing and get higher quality releases

• Deploy faster and more frequently


Parallelization and Performance

Simulation provides automation and access, which enables more tests to be run in parallel

Test 1 Test 2 Test 4Test 3 Test 5 simulation

Test 1

Test 2

Test 4

Test 3

Test 5

Overnight tests take a few minutes

Weekly tests become daily tests

Test more often = less chance for bugs to creep back into the software – higher quality

1/5 the time


Test management server

Test run system (such as Jenkins)

Test generation and management

Test server 1

Simics

board

Networkboard

board

Testing agent/slave node

Parallel Testing and Automation Setup

Physics/Electronics/World

Test server n

Simics

board

Network

board

board

Testing agent/slave node

Physics/Electronics/World

board

Test system launches simulated test labs

Each test can be of a different configuration

Sends out test cases & collects results


Simulation

Expanding Test Coverage: Variation

Simulation

Board A

Simulation

Board A

Network

Board B

Network

Board A

Board A

Board B

Board C

Simulation

Board B

Program 1

Program 2

Program 3

Program 4

generates

generates

generates

System setups are just software

Change setup program to get a new configuration

Infinite supply of each type of system

Randomly generate setups for “didn’t think of that” tests

System creation & setup is automatable

Input file

World / Physics Simulation


Expanding Test Coverage: FaultsFault injection is a natural match for simulation

Test fault detection, handling, recovery, reporting, …

POST, diagnostic software, firmware, applications, system, …

For example:– Corrupt and drop network packets

– Feed bad data from the environment

– Crash processors

– Change memory contents

– Modify system configuration on the fly

Simulation makes faults:– Repeatable

– Non-destructive

– Scriptable and automatable!


Automatically capture test failures

Analyze using record and replay of tests

Run and debug in reverse

When Things go Wrong

More on Collaboration with simulation in our next webinar.


Key Take-Aways Continuous integration and testing requires automation

Automation is made easier by using simulation

Simulation provides the ability to automate any test

Simulation allows testing and automation of what cannot be tested with hardware

Audience Q & A

Jakob Engblom,

Product Line Manager,

System Simulation,

Wind River

Graham Morphew,

Sr. Director of Product Management,

System Simulation,

Wind River

Thanks for joining us

Event archive available at:

http://ecast.opensystemsmedia.com/

E-mail us at: [email protected]

http://ecast.opensystemsmedia.com/

mailto:[email protected]

Technology

Need To Automate Test And Integration Beyond Current Limits?