Automation Guild - The Forces of Destiny

RETRO EDITION

AUTOMATION

© 2017 - ALL RIGHTS RESERVED.


About me

Jonathon is a strategic thought leader and distinguished technology evangelist. He specialises in emerging technologies, innovation and automation, and has more than 15 years of international commercial experience within global organisations, and is currently Director of Digital Assurance at CA R&D based in Oxford in the UK.

Jonathon combines his practical experience and leadership with insights into real-world applications of the core principles and methodologies underpinning DevOps, Digital Assurance and Enterprise Digital (Smart Cities, IoT and AI). As a result, he is frequently in demand as a speaker at international conferences such as EuroSTAR, Gartner, HPE Discover, Oracle Digital Forum, STAREast, STARWest, and Unicom. Jonathon is the author of several award winning books on test automation, as well as numerous podcasts, training courses and webinars.


Roles Awards Thought Leadership Colleagues

Me At a Glance

http://www.amazon.com/Experiences-Test-Automation-Studies-Software/dp/0321754069

https://www.packtpub.com/application-development/advanced-qtp-12-test-engineers-cookbook

© 2017 - ALL RIGHTS RESERVED.Source - https://t.co/tvNZTSkGYr

© 2017 - ALL RIGHTS RESERVED.© 2017 - ALL RIGHTS RESERVED.


Automation Trends 2016 – DevOps

EuroSTAR - TestOps in the Cloud – The Force Awakens(http://www.slideshare.net/Jonathon_Wright/eurostar-testops-in-the-cloud)

SoftTest (Ireland) - Digital DevOps (http://www.slideshare.net/Jonathon_Wright/softtest-ireland-digital-engineering-evolution-devops)


Automation Trends 2016 – Methylmercury.

BCS SIGIST – Test Automation in Hybrid Cloud(http://www.slideshare.net/Jonathon_Wright/bcs-sigist-test-automation-in-the-hybrid-cloud)

BCS SIGIST – Testing as a Service - Models(http://www.slideshare.net/Jonathon_Wright/bcs-sigist-trackjonathonwrighttestingasaservicemodels)


Automation Trends 2016 – Java as a Service

STAREast – Testing in the Digital Transformation Age(http://www.slideshare.net/Jonathon_Wright/stareast-testing-in-the-digital-transformation-age-54958268)

YOW! (Brisbane) – DooM – How to guide(http://www.slideshare.net/Jonathon_Wright/yow-nights-in-brisbane-doom-how-to-survive-the-digital-mothership)


Automation Trends 2016 – MicroContainerization

WinOps (Melbourne) – Azure Stack – Docker & MicroContainerization(http://www.slideshare.net/Jonathon_Wright/winops-workshop-tfs-2015-update-3-windows-2016-server-azure-stack)

Oracle – Enterprise of Things (EoT)(http://www.slideshare.net/Jonathon_Wright/oracle-digital-evolution-enterprise-of-things)


Automation Trends 2016 – Quantum Teleportation

STARWest – Think you can just test that API think again!(http://www.slideshare.net/Jonathon_Wright/starwest-think-you-can-just-test-that-api-think-again)

BCS SIGIST – Testing as a Service - Models(http://www.slideshare.net/Jonathon_Wright/bcs-sigist-test-automation-in-the-hybrid-cloud)


Automation Trends 2016 – Autonomics

Vivid (Sydney) Festivals of Ideas – Digital Assurance(http://www.slideshare.net/Jonathon_Wright/digital-mothership-digitalfirestormcom-digital-assurance-kjr-digital)

Vivid (Sydney) Festivals of Ideas – Digital(http://www.slideshare.net/Jonathon_Wright/vivid-sydney-festivals-of-ideas-beyond-the-buzzwords-kjr)



Tool assist?

Automation

Execution

Autonomics


Are you ready for ‘Test Automation’?

Automation Maturity Model index (AMMi®)Assessment to provide a current automation maturity level and the target maturity level:

“Sometimes half the battle just starting companies on their

automation journey by taking them one step closer to becoming

ready for automation.”

Level 1 – Accelerating – Automated Test Lifecycle Methodology (ATLM)

Level 2 – Managed – Processes are planned, performed, and controlled

Level 3 – Sustaining – Automation process defined and validated against international standards (ISO-29119)

Level 4 – Quantified – Predictability of automation process performance

Level 5 – Optimizing – Automation process variation and statistical predictability

Source – (http://testautomationpatterns.wikispaces.com/AMMI)

The AMMi model is useful in determining the maturity provided by automation


Product Technology

Product Roadmap

Test Process Test Environments

Test Automation

Tools

Test Automation effectiveness

Test Automation

Planning

Test Automation

Development

Test Automation Maintenance

Organization Structure

Automation Maturity Model (AMMi)


AMMi Assessment


First Day Automation: Test Automation Experiences

* Dorothy Graham, ’Experiences of Test Automation’, http://www.dorothygraham.co.uk/automationExperiences, ISBN-10: 0321754069, 9th Jan 2012


Automation Patterns

https://testautomationpatterns.wikispaces.com/

Which ‘Automation Tool’ (10 Years Ago)

“One of my previous roles at large investment banks. Their landscape consisted

of over 300+ solutions under test (SUT), representing 2,700+ applications under

test (AUT), across 50+ global locations with over 300 automation practitioners.”

http://www.jetbrains.com/teamcity/

http://www.jetbrains.com/dotcover/

https://github.com/oneclick/rubyinstaller/wiki/Development-Kit

http://pm.versionone.com/

http://www8.hp.com/us/en/software-solutions/software.html?compURI=1172957

http://www.codeproject.com/Articles/289028/White-An-UI-Automation-tool-for-windows-applicatio

http://www.testautomationfx.com/

http://rubygems.org/gems/cucumber

http://www.greenpeppersoftware.com/confluence/display/GPW/Home/

http://jbehave.org/

http://www.thoughtworks-studios.com/twist-agile-testing

http://www.nunit.org/index.php?p=home

http://www.facilita.co.uk/

http://www.sikuli.org/download.html

http://www.testplant.com/dlds/eggplant-functional/

http://docs.seleniumhq.org/

http://www.concordion.org/Tutorial.html

http://storytestiq.solutionsiq.com/wiki/Getting_Started

http://www.tricentis.com/en/

http://www-01.ibm.com/software/rational/welcome/greenhat/

http://msbuildextensionpack.codeplex.com/


https://www.automation.org.uk/downloads/cv.pdf

Which ‘Automation Tool’ (Today)


How can we move ‘Test Automation’ to the Cloud?

“Existing technologies of virtualisation, business process modelling, cloud based

test automation solutions and rapid and easy internet access allow for the

development of approaches that allow companies to order ‘Testing as a Service’

and pay only for what they use (PAYU).”

platform, technology, client, browser, version, language, test type agnostic

Executing Cloud Tests

containerization

PaaS / IaaS

natural language validated against content sensitive validation

Creating Cloud Tests service-oriented architectureBorn in the Cloud

Test Automation as a Service - Introduction

Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)

Test Automation as a Service - Summary

Test Automation in the Real World?


Test Automation in the Cloud: Case Study

http://partners.telerik.com/docs/default-source/Case-Studies/telerik-test-automation-in-the-cloud_automation-development-servicesBB95938E9FB5.pdf



https://dzone.com/articles/the-death-star-an-ambiguous-requirements-issue

Blogs #ExcuseFreeTesting – Death Star Requirements

https://dzone.com/articles/the-death-star-an-ambiguous-requirements-issue

Team - Confederacy of Independent Systems

Enterprise Architect – Galen Erso

Sprint 1 - Top-secret Imperial weapons

project

Sprint 2 – Raids on Jedha, Eadu and Scarif

Sprint 3 – Technical Data on Tantive IV

Sprint 4 – Battle of Toprawa

Sprint 5 – Raid on Polis Massa

Sprint 6 – Moff Kalast and the Bothan Spynet

Co

llab

ora

tio

n

“The Death Star

blueprints are not in

the main computer”

- Stormtrooper

Blueprint – Death Star

Data Storage – R2-D2

Sprint 7 – Delivery blueprints to Bail Organa

Sprint 8 – Delivery blueprints to Jan Dodonna

ARA - Release Management – General Jan Dodonna Showcases “Thermal Exhaust Port”

Pattern –Fleet of 22 X-wing, 8 Y-wing, 2 R-22 Spearhead Starfighters

Deliv

ery

Patt

ern

sR

eci

pes

Recipe - DestoryDeathStar1.0 - Continuous, Build and Delivery

Define

Design

Develop

(Plan)

Deploy

Blueprint – Stardust

Data Storage – HDD

(Scarif – Data Centre)

Digital AssuranceDigital across the Galaxy


Digital AssuranceDigital across the Galaxy

* AxisAgile.com.au, http://www.axisagile.com.au/resources/scrumtrooper-images/


Digital AssurancePocketbook – Official Launch

NEW exclusive Pocketbook authored by Paul Gerrard, eminent Test-Transformation consultant, thought leader, teacher and international conference speaker.

Order your FREE copy today from http://bit.ly/digital-assurance


Digital AssurancePocketbook – World Tour

TORONTOLOS ANGELESSILCON VALLEY


© 2017 - ALL RIGHTS RESERVED.Product Test Engineer

Gen Zero

(Linear)AUTSUT

Budget in pounds

MainframeLocations:2Teams:5Tool(s): 1

License

x1

History - Three Decades - Mid-Late 90s


TestDirector 1.511995

1995

1998 TestDirector 5

2000

Quality Centre 112010

HP Next Generation ALM (NGA)

- HP Application Lifecycle Management

- HP Agile Manager2016 - TODAY

2014

2004

2010

21 years of ALM (1995 to 2016)

History – Application Lifecycle



History – OneSync

© 2017 - ALL RIGHTS RESERVED.Product Test Engineer

1st Gen

(Modular)AUTSUT

Locations:5Teams:10 Client ServerTool(s): 3

Licenses

10+

http://www.google.co.in/patents/US7526498

Budget in thousands

Automation Framework ArchitectCore Hours 1am to 10am (EST)

Graduate TestersCore Hours 9am to 5pm (EST)

Automation EngineersCore Hours 9am to 5pm (GMT)

History - Three Decades - Late 90s


Automation - Retro Edition (1994 to 2016)

Joe Colantonio - First Conference Graduate Automation Role

First IoT Device

Head of Automation

Y2K

Progression Computers

First Laptop (286SX)

First Automation Framework (1999)


Automation - My Mentor (David Dubovsky)



History - Automation Pyramid

Automated tests can run fast and frequently, which is cost-effective for software products with a long maintenance life

Automated tests run faster and efficiently,due to reused modules within different tests

Automated regression tests ensure the continuous system stability and functionality after changes to the solution under test

Frequent repetition of the same or similar test cases is laborious and time consuming to perform manually

Optimization of Speed, Efficiency, Quality and the Decrease of Costs

UI

API

Unit


© 2017 - ALL RIGHTS RESERVED.Automation Consultant

2nd Gen

(Data)AUTSUT

Locations:5Teams:15 Client ServerTool(s): 5

Budget ¼ million

Licenses

x25

Automation Consultants (UK)Core Hours 9am to 5pm (GMT)

Development (2 week sprints)Core Hours 9am to 5pm (GMT+1)

History - Three Decades - Early 00s

History - FrAgile (UI)

Shift from physical (UI)

Identity Services

Connectivity (HTTPS)

DOM(HTML5)

BIG Data(Database)

External Services

Workflows(Logic)

• OCR / Image• Object Models• XPATH• Regular

Expressions• Descriptive

Programming• Fuzzy Logic• Social Intelligence• Artificial

Intelligence

* Jonathon Wright or Wilson Marr, Hybrid Pattern, https://en.wikipedia.org/wiki/Hybrid_testing






Performance Testing - JMeter

Stuart Moncrieff, http://www.myloadtest.com/tools/

© 2017 - ALL RIGHTS RESERVED.Automation Framework Architect

3rd Gen

(Keyword)SUT

Locations:20+Teams:100+Tool(s): 15+

Budget in millions

Web

AUT


Automation Consultants (NY)Core Hours 9am to 5pm (EST)

Automation Consultants 9am to 5pm (UTC+8)

Development Capability (US)Core Hours 9am to 5pm

Development Capability (EUR)Core Hours 9am to 5pm

Licenses

100+



History - Automation Pyramid (API)

Focus shift from physical (UI) to logical (API) / messaging layer / database layer

SOA(API)

AJAX

CORBA

AMF

HTTP

SOAP

REST

Database(s)

ODBC

Ole DB

ADO .NET

JDBC

Java

Native

Messaging

ESB

AMQP

MSMQ

MQ

JMS

Tibco





WebServices - XML / JSON


© 2017 - ALL RIGHTS RESERVED.Global Head of Automation

4th Gen

(Hybrid)

Locations:25+Teams:50+Tool(s): 20+

Budget in multi millions

AUTSUT

Cloud




Development Capability (EUR)Core Hours 9am to 5pm

Licenses

PAYU

History - Three Decades - Early 10s


Application

Services

Data

Model

Model

Test Data

Services (SOA)

Application

Overlaying all layers

Legacy Distributed and Mainframe - Layered approach

Continuous Deployment

Supporting Future Test Automation

WCF

AJAX1%GWT0%WPF1%

Infragistics0%Flex1%

Web 2.0 (ExtJS, GWT)1%

FIX2%

SWIFT3%

Java/Java Swing6%

SAP0%

XML Web Services including SOAP4%

JMS4%

IBM MQ4%

TIBCO RV1%

UNIX Shell5%

Oracle Forms1%Peoplesoft0%

Oracle UIX1%

PL/SQL/Database6%

Java API5%

C++ API1%

.Net API2%

Informatica0%MF COBAL0%

TCP/IP2%

File comparison5%

Database <-> File comparison2%

Database <-> Database comparison2%

Jasper2%

Kondor+ 0%DBCheque0%SSR1%E30%WEBLOGIC0%Documentum0%oracle GoldenGate0%db-Cris1%DB Fastcollect0%Adobe lifecycle0%iBatis1%Smart Client1%

JSP1%

SPRING1%HIBERNATE1%

SOLARIS1%

Oracle 10G1%JSTL0%MF COBOL0%

AIX1%

Pega 6.10%Jboss0%Linux1%Net 3.51%

4.0 WPF1%RMDS1%INFRAGISTICS1%JDBC1%RMI1%dbAnalytic0%COHERENCE CACHE0%CORBA API0%SunOS0%STARR HUI0%HP/UX0%TOMCAT0%APACHE0%

JMS1%

HADOOP0%ICMP0%PERL0%XML0%VB SCRIPTING0%TIBCORV0%FRAPI0%

J2EE1%

Tibco messaging1%

REST0%

MSSQL 20050%Sybase0%SQL 2014IIS0%Hessian0%

STRUTS 20%

SITE MESH0%

LOG4J0%

EHCACHE0%

Other36%

Sample – 2,738 applications under test (AUTs)

History – CI and B(UI)ilds



© 2017 - ALL RIGHTS RESERVED.Global Head of Automation

5th Gen

(TaaaS)

Locations:100sTeams:1k+Tool(s): 50+

Budget 78 million

Cloud

SUT AUT



wA

gile

Team

s (A

sia)

Co

re H

ou

rs 9

am t

o 5

pm

(U

TC+

8)

wAgile Teams (US)Core Hours 9am to 5pm (EST)

wAgile Teams (EUR)Core Hours 9am to 5pm (CET)




Automation

Applications

Environment

Release

Services

Data

Model

UX Design/Mock-ups

Model

Test Data

Microservices

Release Management

Environment Management

Applications Code

Automation

Continuous Delivery - Layered approach

Continuous Delivery

CRMWPF4%

JAVA16%

Forms

Win32

JS2%Web2.0

ExtJSHTML5

ASP

GWT4% VB6

MFC

SAP

Qt

UNIXSILVERLIGHT

2%SWIFT/FIX

FLEX5%

PEOPLESOFTC++2%C#2%J2EE

.NETAIR

jQuery2%

Other34%

History – Continuous Delivery

41%

33%

6%

20%Automation (UI)

No UI

Unsuitable

Unsupported

Sample – 200 solutions under test (SUTs)

https://youtu.be/ql8ANg9p-ao



Blogs #ExcuseFreeTesting – Legacy is your Legacy!

https://dzone.com/articles/legacy-is-our-legacy-1

https://dzone.com/articles/legacy-is-our-legacy-1



Source: Gartner (March 2015)

DevOps = Automation


Design QA/TESTDEV PRODUCTIONPRE-PROD

Specification

CI/Build Functional testing UAT Integration

testing Performance engineering

Deploy to pre-prod

Code commit SCM

Design Spec

Requirements

Business Users

TESTData

TESTStub

TESTData

User BA

Product Manager

CustomerUsers

CustomerExperience

CustomerExperience

CustomerExperience

CustomerExperience

Design

Business Value

Business Value

Business Value

Improvement

Innovation

Intelligence

Assessment

Insight

Learning

Maintenance

Operations

Delivery

Testing

Support

Cu

sto

me

r

Value-Driven

Value Insight

Learn Learn

Monitor Monitor TESTStub

REALUsers

TESTData

REALData

TESTStub

USERCase

Bu

sin

ess

DevOps: Shift-X

idea


TestOps BUILD then TEST vs. DevOps BUILD then Deploy?

De-coupling

Solutions (SOA)

Continuous

Build, Integration

& Delivery

Good Coding

PracticesConfigurability &

Maintainability

Ops Hub

Infra as Code

Extend QA

(Ops)

Agile

Engineering

(Shift Left)

Agile

Engineering

(Shift Right)

APMBlueprints Recipes PatternsARABlueprints Recipes PatternsArD SV

TestOps (Test-Dev) with Automated Testing DevOps (Dev-Test) with Automated Deployment

Ops

Monitoring

(Testing)

Monitoring

(Ops)

Modelling

(Testing)

Modelling

(Ops)

Dev Test


QualityGate™ Development

6-10 secs – Check-in / Analyse / Commit (GIT)

0-6 secs – Minor UX change (Dev to TestOps) 10-40 secs – Predictive Testing (ArD)

QualityGate™ Map

Confidence Level 74%

40-50 secs – Predictive Deploy (Automic)

50-60 secs – Predictive Release (RA)

6% chance to break build

66% Quality Gate Map (confidence rate)

88% Quality Gate Map (confidence rate)


QualityGate™ Map

Confidence Level 69%

40-50 secs – Predictive Testing (ArD)1-5 secs – Smart Analysis (MVC)

5-10 secs – Smart Build (RA) 30-40 secs – Smart Testing (VS 2017)

10-20 secs – Smart Asset Cube (TAaaS.net)

QualityGate™ Testing


All media links:Domino’s Pizza orders Hitachi Consulting to advise on DevOps pushHitachi to boost Domino’s Pizza delivery with DevOps strategyHitachi Consulting and Domino’s Pizza collaborate for a slice of DevOps actionDomino’s signs up to Hitachi Consulting’s DevOps expertiseDomino’s continues to deliver with a DevOps strategy

Jonathon Wright said: "With DevOps, there is a strong focus on continuous testing and continuous delivery. This means there is more emphasis on testing much earlier in the solution delivery lifecycle. "Domino's Pizza has embraced continuous integration and testing to help the software delivery team rapidly build, test, and deliver secure, high-quality applications."

DevOps: in the wild



“The physical nature of wearables

technology makes it vital to testing using

real devices…. helps to mitigate

syncing/connectivity issues.”

Wearables – First Day Automation



Release Automation

Infrastructure Provisioning

Configuration Management

Application Platforms / PaaS

vSphere

Version Control / Artifact

Repository

Planning & Requirements

Change Management

Test Automation

DatabasesCI/Build Automation

DevOps: Release Automation



Digital Experiences (DX) Book Release 2017

‘Why?’ are our Digital Experience so uniquely different? could it be related to our various levels of Digital Proficiency*?:

Digital Natives - Comfortable in engaging all Digital Channels (i.e. generation x, y & z). Digital Naïve – Belief that they are Digital Natives based on ownership of technology (i.e. millennials / Xbox generation). Digital Designer – Build Digital Technology around their lifestyles and living space. Digital Savvy – Self-taught and aware of the Digital Capabilities of technology. Digital Citizen – Someone born into Digital Lifestyle and consumes without questions. Digital Immigrates – Crossed the chasm into the Digital World, forced into engagement with Digital Channels. Digital Voyeurs – Recognized the shift to digital but not ready to be active yet themselves. Digital Holdouts – Resisting the shift to digital, ignoring the impact. Digital Disengaged – Understand Digital Technology but avoiding engagement on purpose, often for privacy reason. Digital Dangerous – Extremely overconfident in their Digital Capabilities to the point they can cause disruption or damage.

So ‘Why?’ do we develop a single purpose endpoint (i.e. UI) to support all Digital Personas?When even the same object instance (i.e. UI-matrix) viewed by the same categorisation of

Digital Persona could be interpreted completely differently.

* Disrupting Digital, R Wang, 2016, Constellation Research

“The typical mobile user touches his or her phone 2,617 time every day, over an average of 145mins duration.”


Wearables - Crowd testing

Testing in the Wild

Microsoft© HQ Seattle


Internet of Things – Testing Smart Cities

* Oracle Digital Forum - Digital Evolution: Enterprise of Things http://social-innovation.hitachi.com/en/copenhagens-big-energy-plans/


Internet of Everything: Sensors

* http://www.fastcompany.com/3051578/elasticity/hitachi-says-it-can-predict-crimes-before-they-happen




Microconterization

88mb

1gb

16mb

128mb

Windows Nano Server stateless and stateful Service Fabric Microservices supports massively scalable self-healing platforms. Unikernels reduces the footprint of Microservices in Microcontainers with Docker running MirageOS or Alpine Linux.


Windows Server 2016 TP4 (19/11/2015) supports Windows Docker Enginenative Windows Containers accessible through the Docker client Kitematic.

Pull Docker images from the Windows Docker Engine using PowerShell v5 (DCS).

Docker on Windows



forces-destiny.com


CO

GN

ITIV

E A

DA

PTI

VE

AD

OP

TIO

N

Manual

Continuous

EnterpriseDigital

Lean Engineering

Autonomics

BUSINESS VALUE

LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5

?

Optimized Organization

for DesignOps

Continuous Insight & Learning

Value StreamMapping

Consistency & Collaboration

Heroes & Heavy Lifting

DIG

ITA

L M

ATU

RIT

Y

• Insight-Driven (Predictive / Prescriptive)• NoOps organized (IoT-Ops/CX-Ops/Sec-Ops)• Self-healing end-to-end autonomic orchestration• Microcontainerization & Microservices enabled• Open Innovation & Connected Intelligence• Quantum Teleportation (Shift X)

Where are you?

Minimum Viable Quality

• Silos, manual handovers, waterfall process• One release/year• Monolithic apps• Long term project/resource planning• Error prone dev/test/release processes

Minimum Viable Product

Minimum Viable Delivery

Organization Change

User Story Mapping InsightBeliefs, Values,

CultureValue Behaviors Heuristics

Digital Automation


DIGITAL ASSURANCE

VALUE-DRIVEN DELIVERY

Cognitive Adaptive Adoption (CA-A)

1Cognitive Adaptive Learning

2Cognitive Adaptive Testing

3Cognitive Adaptive

Technology

4Cognitive AdaptiveDelivery


Continuous Adoption

CognitiveAdaptive Adoption

Continuous Testing

Continuous Improvement

Continuous Delivery

Continuous Intelligence

Continuous Learning

Continuous Innovation

idea

DEVELOPMENT OPERATIONS

‘Shift-Right’ Digital Ecology

‘Shift-Up’ Digital Assurance feat. Connected Intelligence

‘Shift-Left’ Digital Evolution feat. DesignOps

CUSTOMER EXPERIENCE TESTING

DesignDevTestOpsAgility across all 4 increases speed, reliability and efficiency

CognitiveAdaptive Testing

CognitiveAdaptive

Improvement

Cognitive Adaptive Delivery

CognitiveAdaptive

Intelligence

CognitiveAdaptive Learning

CognitiveAdaptive

Innovation

DESIGN

‘Shift-Down’ Digital Archaeology


Proactive over Reactive

Governance over

Ownership

Maturity over Mastery

Predictive over Continuous

Evolution over Revolution

Engineering over Agility

Intelligence over Things


© 2017 - ALL RIGHTS RESERVED.© 2016 CA. ALL RIGHTS RESERVED.

The Digital ArchaeologyArtificial Intelligence & Machine Learning

BRINGS TRUE ARTIFICIAL INTELLIGENCE (AI) TO LIFECYCLE VIRTUALIZATION Virtualize services without requiring any knowledge or

decoding of the service protocols

Applies a genome sequence alignment algorithm, discovers byte-level patterns in message protocols

Now virtualize a much wider range of protocols without requiring a new DPH

HIGH ACCURACY: 99.6 – 100%

Increased speed and accuracy with Entropy Weighting + Message Clustering

The more data a service observes the data, more intelligent it becomes

Perfect for performance testing where we deal with tons of data



‘Shift Right’ Becoming an Insight-Driven Business

The Digital Ecology

REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO

AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS

CONTINUOUS DELIVERY MAINFRAME SECURITY

Release management dashboard API analyticsReal-time app performance


* CA’s Analytics Reference Architecture

Data Collectors

Dat

a A

ggre

gato

rAgents

…

Probes

Machine/Log File Data

Insight-Driven

Existing Platforms

Custom Platforms

New Platforms

Partner Platforms

Analytics Engine

Dat

a In

ject

ion

AP

I

Dat

a A

cces

s A

PI

Service

Layer

Dat

a Ex

trac

tio

nData Processing (Speed)

Data Processing (Batch)

Data Bus

Powered by Connected Intelligence technologies

Cognitive Adaptive Insight (iCA)


Powered by Connected Intelligence technologies

Connected Intelligence

Release Management

Environment Management

ApplicationManagement

Infrastructure Management

RepositoryManagement

Data Management

Test Data LearnProbe

Cognitive Adaptive Insight (iCA)

CA App Experience Analytics


T H E D I G I TA L EC O LO G Y

RUNDEPLOYTESTBUILDPLAN

INTAKE RequirementsUser StoriesRelease Plan

MODEL Import User Stories to Automatically Create, Visualize and Optimize Tests

Initiate Functional, Non-Functional and Security as Code

FUNCTIONAL Subset/Mask Test Data Create/Reserve Test Data Test Automation Library Ensure Mobile Experience

INSIGHT-DRIVEN

CONFIG/DEPLOY Provision Entire Stack Confirm Configurations Approve Changes Successfully Deploy Internal or External Cloud

INTEGRATIONRemove Constraints with Virtual Services Mobile, Web, App Server,

Middleware, Backend, MF 3rd Party Systems / API’s

CODEDevelop and Commit Code, Scan Code, Version Control, Continuous Integration. Complete Build and Initiate Release

AUTOMATION AUTOMATION AUTOMATION

Speed/Time-to-Market Quality/Availability

AUTOMATION

Cost/Financial Mix Risk/Compliance

Data Automation

CustomerExperience

NON-FUNCTIONAL Performance Engineering Simulate Backend Load Test Outlier Conditions Ensure Mobile Experience Security / Penetration

Common Goals “Big Picture” Issues

DESIGN SHIFT-LEFT SHIFT-RIGHT OPS

Node.Probe

Node.TestTDDBDD

ATDD

BPMNNimbus

Visio

Node.Data

Node.Learn

MEASURE/FEEDBACK Customer Experience Business Service View Application View Infrastructure View Dynamic Capacity Feedback loop


Cognitive Adaptive Adoption


“The best aspects of these proven test approaches

demonstrate how they have evolved over the past

decade, and this echoes some of the changes towards

more lean and agile business methodologies. They are in

a constant state of evolution – just as the underpinning

technology evolves over time.”

My section in the ‘Experiences in Test Automation’ book

(Dorothy Graham & Mark Fewster) 2012.

Final Thoughts – The future of automation?


VR / AR – Real World Testing


VR / AR


“Digital engineering breaks down the silos across the

entire organisation – allows everyone to get involved in

evolving digital capability: connected intelligence maps

the value steam mapping of business challenges and

goals. This is a new way of IT delivery that allows for

digital initiatives to be deployed at pace across the

organisation but in a (relatively) stable and

industrialized state – a willingness to ‘fail forward’.”

My chapter in Engineering the New Reality /

Evolving the New Reality, 23rd October 2015

.


Continuous Testing 101 Webcast Series

Conversations with… Erik Simmons

Software & Systems Professional nuCognitive

Episodes:Solution Thinking & Value-Driven Delivery

Ray ArellFounder and Principal Coach

nuCognitive

catechnologies.podbean.com


IdeasBeliefs, Values, Culture

Heuristics

Behaviors

Value

AgileLean

Solution ThinkingWebcast Series - Continuous Testing 101

Copyright © 2016 nuCognitive LLC. All Rights Reserved.

Methods and Skills

Ray Arell


Erik Simmons

Copyright © 2016 nuCognitive LLC. All Rights Reserved.

Value-Driven DeliveryWebcast Series - Continuous Testing 101



Bonus Material


Test Automation in the Cloud: Case Study

http://partners.telerik.com/docs/default-source/Case-Studies/telerik-test-automation-in-the-cloud_automation-development-servicesBB95938E9FB5.pdf

Test Automation in the Real World?


BPM Export (Visio)

http://office.microsoft.com/en-gb/visio/visio-professional-2013-free-trial-flowchart-software-FX103791368.aspx


Business Process Modelling

http://www8.hp.com/uk/en/software-solutions/software.html?compURI=1215990


Business Process Workflow

http://msdn.microsoft.com/en-us/library/ms752059.aspx


Business Process Scenarios


Business Process Testing

“verbs and nouns in the test specific language are written in natural language which using context sensitive validation.”

Logi

n.P

roce

ssAction Object Event

Enter Username Text

Enter Password Text

Press Login Button

Domain Specific Language

Act

ion

s

Ob

ject

s

Even

ts

Press | Login | Button

Nouns

Verbs

Test Specific Language (TSL)

“The test editor is designed to be low waste, it will only offer nouns and verbs based on discovered artefacts not programming objects.”


Business Process Logic

http://www.altova.com/download/xmlspy.html


Business Process Data

ALTOVA XML Spy 2013 (R2 SP1)

http://www.altova.com/download/xmlspy.html

© 2017 - ALL RIGHTS RESERVED.www.eurostarconferences.com

Exploratory Testing


Dynamic Data Adapters - Capture


Dynamic Data Adapters – Action(s) Log


Dynamic Data Adapters – IntelliTrace


Dynamic Data Adapters – Issue(s)

Dynamic Data Adapters – Traffic Proxy

http://www.fiddler2.com/


Lifecycle Virtualization - Interfaces


Component Layer – Unit Test(s)

http://www.nunit.org/index.php?p=home


Reporting - Test Execution


Reporting - Burndown

TFS Workbench

Monitor velocity & capacity


Business Process Modelling


Continuous Deployment

http://jenkins-ci.org/

http://www.jetbrains.com/teamcity/


Test Agent(s) – Telerik Art of Testing Framework

http://www.telerik.com/products/free-testing-framework/overview.aspx


Performance Testing


Testing as a Service - Models

*‘Testing as Service’ Models’ discussion paper published, 26th October 2014, https://leanpub.com/taas