Test Architects at Siemens - Hochschule BremerhavenTest Architect (TeA), 2016 Test Architect Requirements Engineering for project / product managers, 2010 ... governance Incremental

Test Architects at SiemensPeter Zimmerer

Siemens Corporate TechnologyUnrestricted © Siemens AG 2019 Bremerhaven, GermanyFebruary 21–22, 2019

https://fg-tav.gi.de/veranstaltung/43-tav/https://informatik.hs-bremerhaven.de/seuh2019/

Unrestricted © Siemens AG 2019February 2019Page 2 Corporate TechnologyPeter Zimmerer

Agenda

Why do we need a Test Architect?

What is a Test Architect?

What are the responsibilities and tasks of a Test Architect?

How can a Test Architect provide value and create impact on the business?Examples on strategies, tactics, and practices


Systems we build today at Siemens for our customersVision 2020+New company organization will go live on April 1, 2019

Gas and Power

Smart Infrastructure

Digital Industries

Operating Companies Strategic Companies

Service Companies

Financial Services Global Business Services Real Estate Services

Mobility


Siemens Core Learning Programs and beyond

Software Architect (SWA), 2009System Architect (SyA), 2012

R&D Manager, 2012

Senior Software Architect (SSWA), 2007

Softw

are

Arch

itect

Syst

em

Arch

itect

Seni

or

Softw

are

Arch

itect

Functional Trainings

Test Architect (TeA), 2016 Test

A

rchi

tect

Requirements Engineeringfor project / product managers, 2010


Key parts of the core learning programs

Fostering best practice sharingand experience exchange

Architects’network

Assuring practical experiencefor critical projects Certification

Setting standards forsoftware and system development

Qualification(courses andon-the-job training) So

ftwar

eA

rchi

tect


Test Architect – Some external references (1)

Page, A., Johnston, K. and Rollison, Bj. 2008. How We Test at Microsoft, Chapter 2. 2008

Alan Page (Microsoft), What is a Test Architect?, 2008https://blogs.msdn.microsoft.com/alanpa/2008/05/01/what-is-a-test-architect/

John Morrison (Oracle), Test Architect, Oct 8, 2007https://blogs.oracle.com/johnmorrison/entry/test_architect

James Brewer et.al. (Raytheon), The Value of the Test Architect, March 2012http://www.dtic.mil/ndia/2012TEST/13942_Brewer.pdf

ICST InSTA workshops (International Workshop on Software Test Architecture)http://www.aster.or.jp/workshops/insta2015/

UML Testing Profile (UTP) Version 1.2. 2013. Chapter 8, “Test Architecture”, pp. 19-24http://utp.omg.org/, http://www.omg.org/spec/UTP/

Some of the links are expired!


Test Architect – Some external references (2)

ISTQB Glossary (http://www.istqb.org/)

Test architect

(1) A person who provides guidance and strategic direction for a test organization and for its relationship with other disciplines.

(2) A person who defines the way testing is structured for a given system, including topics such as test tools and test data management.

That’s ok … but there is more …


Driving triumviratefor testing & quality engineering

Test Manager

Process quality,certification and audits,regulations and norms

across lifecycles and versions

Budget, people,plans, logistics

for the test organization

Quality goals

Productivity

Process

Stra

tegy

Quality Manager


Driving triumviratefor testing & quality engineering

Test Manager Test Architect

Process quality,certification and audits,regulations and norms

across lifecycles and versions

Budget, people,plans, logistics

for the test organization

Cost efficient, accelerated, innovative, sustainable

testing solutions

for test methodology and test infrastructure

Quality goals

Productivity

Process

Stra

tegy

Quality Manager


As a Test ArchitectYou have one Role – but you are wearing two Hats

Test Expertfor the system under test (SUT)

• Design the test approach• Apply innovative test technologies• Drive the quality of the SUT

Software / System Architectfor the test system

• Design and realize the test architecture• Apply innovative software technologies• Drive the quality of the test system

This is the architect's job!


A Test Architect needs a broad set of competencies

tricttteededddddee ©©©©©©©©©©© SSSSSSSiieieieiememememensnsnsns AAAAAGGGGG 20202020201919191919February 20191 CorporaPeter Zimmerer

BusinessUnderstanding

Testingand Quality

RequirementsEngineering

Architecture andDevelopment

Social Skills& Leadership

Business modelunderstanding

Global developmentStandards, Regulations, Legal issues

Quality requirements

Product & Systemrequirements

Arch.& Design

Social skillsProject management

Development processes

TeA Test ArchitectTeM Test Manager

Lifecycle issuesDomain

knowledge

Leadership

Ecosystems,Partnering, Suppliers

Architecture Quality

Test automation

Integration & Deployment

Test & Quality management

L

Advanced

Expert

Basic

Basic knows aboutAdvanced can applyExpert drives, coaches and is able to

improve in own domain & scope

Test strategy

Lean & Agile testing

Testing Governance & Guidance

Test architecture

Test process improvement

Test methods & technologies

Design for Testability


Test Architect – Role profileResponsibilities and tasks

UnUUnUUnUUnrerereresssststttttttttttttttttttttttsssssssstssssssssss ririrriririiiririiiiiiiiririiiiirirrrriiirrrriiiirriiiirrrriiiiririrriirictcctctcttttcttctttcccttctctttccttctcctctctctctcccccttccccccccccccccttcct dedededededeeddeddddedddddededdeddddddedddddedededeeddedddddddddeeeedddedddedddedddedededdeeedddeeeedddddeeeeeeeeeddedddedddeded ©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©©© SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSieieiieieieieeieiieiieiiieieiiieiieiiieieeeeiieeeeeeiiieeeeeeeeeeeeiememememmemmmemememeememmememmmmemmmmmeeememeemememmmmmemmmeeeemmmmmemmmemmeeeeeeeemmmmmmmmmmmeeemeememmmememennsnsnnnsnsnnsssnssnsssnsnnnnnnnnsnssnnnnnnnssnsnsnnnsnssssssnsnnnsnnnnssssnsnnnnnsssssssnsssssns AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG 0202020020202002020222020202220202000020202222222020200200020220222200002220202022202002020220000202020200000222222000000222200000022200000201111911111111111111111111111111111111111919

Test Architect – Role profileResponsibilities and tasks


Test Architect – Role profileResponsibilities and tasks (extract)

• Analyze the current testing process and practices and drive improvements• Evaluate and select technologies (e.g. by prototyping)• Identify and manage technical product risks• Perform make-or-buy decisions

• Ensure testability of requirements engineering artifacts• Define testability requirements• Foster the use of test-driven approaches in requirements engineering

• Manage the internal quality of the whole testing environment• Provide comprehensive documentation of the testing infrastructure's architecture


Testing & QualityBusinessUnderstanding


Architectureand Design Maintenance

BusinessUnderstanding

Maintenance


Test Architect – Role profileResponsibilities and tasks (extract)

• Ensure that testability requirements are incorporated into product design• Provide testing infrastructure requirements• Identify and manage technical risks for the testing environment• Drive integration of testing infrastructure with the SUT• Guide/Coach system/software architects in testing related topics• Review architecture artifacts• Define and implement a justified, effective test strategy for the project, aligned with the business and quality

goals• Measure, control and continuously optimize coverage and degree of test automation• Define and maintain an effective and efficient test architecture in close cooperation

with system integration and test managers• Govern the design and improvement of test automation framework and test suites• Ensure quality of the test code

Architectureand Design

Testing & QualityBusinessUnderstanding


Architectureand Design Maintenance

Testing & Quality


Test Architect Expert Training (TeA) Topics

Business Understanding (~15%) Testing & Quality (~40%)

Requirements Engineering (~10%) Social Skills (~20%)

• Product Lifecycle Models (PLM)• Lifecycle models and processes• Portfolio management• Risk & Uncertainty• Estimation and prioritization

• Communication with stakeholders about testing and quality

• Situational leadership model• Leadership and motivation• Conflict management

• Good requirements• Reviewing requirements• Configuration and change request

management

Architecture (~15%)• Systematic architecting• Design strategies & tactics• Design patterns• Architectural views and documentation• Architecture quality• Architecture reviews• Refactoring

• Value of testing• Test strategies, risk-based testing• Design for testability• Test management• Agile testing• Test design techniques• Test automation• Test environment, test architecture• Test documentation• Internal quality• Test Architect revisited

Cross-Cutting• Role of the Test Architect• Dealing with quality attributes

(non-functional requirements)• Agile / Lean development


Test Architect Expert Training (TeA) Schedule

Preparationphase

~2 months

CertificationWorkshop 1Live as a TeA

4.5 days

Workshop 2Work as a TeA

3.5 days

Workshop 3Survive as a TeA

4 days

Projectphase 1

~2 months

Projectphase 2

~2 months

Selection of candidates(self perception)

Experiencephase

>6 weeks

In parallel setup and maintenance of networking bycentrally organized events and exchange platforms


Example: Domain Model for TestingMetamodel of the UML Testing Profile (UTP)

http://www.omg.org/spec/UTP/http://utp.omg.org/http://utp.zen-tools.com/

In software engineering, a domain model is a conceptual model of the domainthat incorporates both behavior and data.


Test architecture

Test architecture refers to the high level structures of a test system that is used/required to test a system, the discipline of creating such structures, and the documentation of these structures. These structures are needed to reason about the test system. Each structure comprises test elements, relations among them, and properties of both elements and relations … (adapted from Wikipedia)Test architecture is the architecture of the test system

Two kinds of scope: test system architecture and test suite architecture• Test system architecture is for test system, system/software to be tested (SUT), platform where SUT is

executed, generator of test cases etc. (see UTP)• Test suite architecture is for test suite inside, which mainly consists of test levels, test types, test

conditions, test cycles, groups of test cases, etc.

A good test architecture and design for testability is a precondition/enabler/driver for effective and cost-efficient test automation that is easy to maintain and evolveA test architecture is needed to design and manage the quality attributes (NFRs) of a test system

http://www.aster.or.jp/workshops/insta2015//i t 2015/


Test Architect & “Architecture”

Test Architect

Product / Systemarchitecture Test architecture

TestDefineDesign

Test infrastructurearchitecture

contains

ReviewCoach


Systematic architecting

Performuser requirements

elicitation

Createdomain model

Model dynamicsof scenarios

Determine scope& boundaries

Create first conceptual draft

Structure thebaseline

architecture

Introducedeployment

views

Defineprinciples &guidelines

Plan and realizeincrements

Product / System / Service

SWA / SyA


Systematic architectingLinks between Test Architect (TeA) and Software / System Architect (SWA / SyA)

Performuser requirements

elicitation

Createdomain model

Model dynamicsof scenarios

Determine scope& boundaries

Create first conceptual draft

Structure thebaseline

architecture

Introducedeployment

views

Defineprinciples &guidelines

Plan and realizeincrements

Domain model Product / System / Service

SWA / SyA

IdentifyASR

“Companion” domain model

Conceptual draft of test system

Model dynamics of test system

Scope & boundaries of test system

Refine architecture of

test system

Define test deployment architecture

Testing principles

governance

Incremental evolution

Define qualities

Dynamics

Boundaries

Increments,

Integration TestsReview both

architectures

SynchronizeTest

Sys

tem

–Te

A

Test strategy


Parallel Architecture for Component Testing (PACT)

ApplicabilityThis approach is useful at class and cluster scope for unit and integration testing of an application class hierarchy

IntentImplement a driver as a class hierarchy that is symmetric to the hierarchy of the class under test. Each driver class sends a test message to an object of the class under test

John D. McGregorClemson University, South Carolina


ISTQB Generic test automation architecture (gTAA)IIIISSSSSSSSSSSSSSSSSSSSSSSSSSSTTTTTQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQBBB

http://www.istqb.org/


TTCN-3 test architecture

System Under Test (SUT)

System Adapter Platform Adapter

TTCN-3 Executable

Test Management

External Codecs Test Logging

Test System User

TTCN-3 ControlInterface (TCI)

TTCN-3 RuntimeInterface (TRI)

Provided by the test environment

Compiled TTCN-3module

System-dependent

Standardized interface

Standardized interface

Test Control Component Handler

TTCN-3 Test System

http://www.ttcn-3.org/


Testing strategies in a microservice architecture

Toby Clemson, ThoughtWorks: Testing Strategies in a Microservice Architecturehttps://martinfowler.com/articles/microservice-testing/


Utility tree and scenario descriptions

A Utility Tree• describes the most important quality attributes• details each quality attribute in relevant facets• relates scenarios to quality attributes or their facets and thus allows to prioritize them

Scenarios are used to• represent stakeholders’ interests• understand and clarify quality attribute requirementsUtility

Reliability

...

Scalability

PerformanceThroughput

Signal delay

Target: The track allows a frequency of at least 12 trainsper hour in normal operation

(H,M)

(L,H) Outstanding: The track allows a frequency of at least 15 trains per

hour in normal operation

Train length

Numberof seats

Trains can be configured with 2, 4 or 6 wagons.

(H,L)

(L,L) 30% of the seats shall be removable in the service station.

Safety

Source of stimulus

StimulusArtifact

Environment

Response

Responsemeasure

Source of stimulus

StimulusArtifact

Environment

Response

Responsemeasure


Design alternatives for a quality attributeDesign strategies and tactics

Performance of IT Systems

• Increase computation efficiency

• Reduce computational overhead

• Manage event rate• Control frequency of

events / raw data

Resourcedemand

• Introduce concurrency

• Maintain multiple copies of data/services

• Adapt HW resources(processor, memory, network)

Resourcemanagement

• Scheduling policy• Distribution• Localization

Resourcearbitration

The Quality in focus

What could be done

How could it be done

Design patterns for implementing the tactics• Caching• Lazy Evaluation• Coordinator• Half Sync / Half Async• Command Processor








Quality Attributesof the test system (test code)

Scope of Dealing with Quality Attributes

Quality Attributesof the SUT (production code)








Internal Qualityof the test system (test code)

Scope of Internal Quality

Internal Qualityof the SUT (production code)


Some numbers

• Started in 2016

• Currently in RUN 6 (typically 2 RUNs per year)

• Overall ca. 90 participants in the trainings

• >30 Certified Siemens Test Architects (TeA)

Invented – Defined – Designed – Implemented – Piloted – Established – Grow


Summary Test Architect

A new key role Test Architect on eye level with software / system architects

Diverse responsibilities and tasks that require proper skills, competencies, capabilities

A new career path for testers

A good Test Architect provides value and creates impact on the business

Test Architect



Peter ZimmererPrincipal Key Expert EngineerSiemens AGCorporate Technology Research and Development for Digitalization and AutomationSoftware & Systems InnovationsCT RDA SSI

Otto-Hahn-Ring 681739 Muenchen

Phone: +49 (89) 636 633509Fax: +49 (89) 636 45450Mobile: +49 (172) 4328121

E-mail: [email protected]

Internetsiemens.com/innovation siemens.com/corporate-technology

Documents

Test Architects at Siemens - Hochschule BremerhavenTest Architect (TeA), 2016 Test Architect Requirements Engineering for project / product managers, 2010 ... governance Incremental