Upload
jonathon-wright
View
286
Download
2
Embed Size (px)
Citation preview
1 © 2017 CA. ALL RIGHTS RESERVED.
Cognitive Adaptive Testing based on analytics driven quality
Jonathon WrightDirector (R&D) – Automation, DevOps & Digital [email protected] | linkedin.com/in/automation | slideshare.com/jonathon_wright
© 2017 CA. ALL RIGHTS RESERVED.
2 © 2017 CA. ALL RIGHTS RESERVED.
Disclaimer
Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current information and resource allocations as of June 1st and is subject to change or withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.
Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-available basis. The information in this presentation is not deemed to be incorporated into any contract.
Copyright © 2017 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies
THIS PRESENTATION IS FOR YOUR INFORMATIONAL PURPOSES ONLY. CA assumes no responsibility for the accuracy or completeness of the information. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENT “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. In no event will CA be liable for any loss or damage, direct or indirect, in connection with this presentation, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised in advance of the possibility of such damages.
3 © 2017 CA. ALL RIGHTS RESERVED.
Abstract
In the digital economy, everyone’s in the software business, and the landscape is constantly changing. The traditional approach to providing business value through lengthy plan–build–test–deploy cycles no longer delivers value in an increasingly competitive, consumer–centric marketplace. Merely compressing these cycles by adopting iterative or agile methods doesn’t address the innovation, speed–to–market, quality and resilience challenges organisations face.
Shifting to a more dynamic, adaptive and continuous model calls for new tools and practices that accelerate communication, collaboration, integration, measurement and automation. Whether labelled this as digital, shift–left or DevOps, understanding the detail of this journey is an essential part to becoming digital–testing ready.
Join Jonathon as he explores the fundamentals of continuous adaptive testing that will enable the continuous delivery pipeline, and to provide instant, smart, cognitive adaptive technology that will power the next generation of insight and value–driven delivery capable platforms. Revolutionising how you test in the digital future will drive smart business decisions and end costly, reactive rework and errors.
Jonathon Wright
CA
Director (R&D) –DevOps, Automation & Digital Assurance
5 © 2017 CA. ALL RIGHTS RESERVED.
The success of our technologies largely depend
on the fact that, while we were speculating about the
possibility of ultra-intelligence, we increasingly
enveloped the world in so many devices, sensors,
application and data that it became an IT-friendly
environment, where technologies could replace us
without having any understanding, intentions,
interpretations, emotional states, semantic skills,
consciousness, self-awareness or flexible intelligence.
Memory as in algorithms and immense datasets now
outperforms intelligence. MegaTech, Technology in 2050, The Economist, Daniel Franklin, 2015
“
”
THE NEED FOR SPEED
94%of executives face increased pressure to release appsmore quickly
“
Journey to the Digital Testing
© 2017 CA. ALL RIGHTS RESERVED.5
Based on 2014 Vanson Bourne study commissioned by CA
6 © 2017 CA. ALL RIGHTS RESERVED.
Digital Testing, Quality & Assurance
DIGITAL TESTING, QUALITY & ASSURANCE
http://bit.ly/digital-assurance http://a.co/jlxMHWp
7 © 2017 CA. ALL RIGHTS RESERVED.
IdeasBeliefs, Values, Culture
Heuristics
Behaviors
Value
AgileLean
Continuous Testing 101Webcast Series – Value-Driven Delivery – Episode I
Methods and Skills
http://catechnologies.podbean.comRay Arell
8 © 2017 CA. ALL RIGHTS RESERVED.
QUALITY
2/3of business leaders say the future of their business depends on the quality of their software2
VELOCITY
94%of executives face increased pressure to release apps more quickly1
1. 2014 Vanson Bourne study commissioned by CA2. “Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015 3. “DevOps and the Cost of Downtime” – IDC 2014
“Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015
© 2017 CA. ALL RIGHTS RESERVED.8
COST
25%of a single application’s development and operations costs is wasteful3
While Ensuring a Superior Digital Experience!
Digital TestingDelivering Quality at Speed
SHIFT RIGHT
10 © 2017 CA. ALL RIGHTS RESERVED.
Digital Testing
TESTING IN THE DIGITAL WORLD
COGNITIVE ADAPTIVE ADOPTION (DESIGNOPS)
COGNITIVE ADAPTIVE DELIVERY (VALUE-DRIVEN)
COGNITIVE ADAPTIVE LEARNING (ANALYTICS-DRIVEN)
COGNITIVE ADAPTIVE TECHNOLOGY (DIAGNOSIS ENGINE)
WHERE NEXT?
1
2
3
4
5
6
11 © 2016 CA. ALL RIGHTS RESERVED.11 © 2017 CA. ALL RIGHTS RESERVED.
CA Portfolio Management
Unify long term strategy, investment and portfolio planning.
CA Project Management
Collaboratively plan, prioritize and track work across the enterprise.
Develop & Test
Synchronize efforts to dramatically speed app development & increase quality
Manage & Monitor
Make a great customer experience a competitive advantage
Release & Deploy
Control the release process, to continuously advance application quality, improve the customer experience and reduce costs.
ContinuousDelivery
Minimum Viable Experience
Minimum Viable Quality
Value-Driven Delivery
BUSINESS INITIATIVE
Complexity-Informed Organizational Change
BUSINESS VALUE
MinimumViableProduct
CUSTOMER
THINK
MAKE
Digital TestingComplexity-Informed Organizational Change
12 © 2017 CA. ALL RIGHTS RESERVED.
DesignOpsCreate and deliver Ideas to Outcomes
Pervasive Security
Redefine how work is planned, managed and
executed
Agile Management
Provide a flawless app experience optimized for
performance
Agile Operations
Accelerate and streamline development, testing and
release
Continuous Delivery
Continuous Development Continuous Adaptive Testing Continuous Release
Digital TestingValue-Driven Delivery (Shift-Left & Shift-Right)
13 © 2017 CA. ALL RIGHTS RESERVED.
DIGITAL ASSURANCE & TESTING
VALUE-DRIVEN DELIVERY
1Cognitive Adaptive Delivery
2Cognitive Adaptive Learning
4Cognitive Adaptive
Technology
3Cognitive AdaptiveTesting
Digital TestingCognitive Adaptive Adoption (CA-A)
14 © 2017 CA. ALL RIGHTS RESERVED.
Automate manual steps and scripts—eliminating repetitive, error-prone tasks
Design reusable deployment workflows
Integrate best-in-class tools dev through production
Gain visibility and transparency
Close feedback loops throughout the pipeline
Efficiently release multi-team, cross-app, composite releases
Define and enforce continuous delivery best practices
OPTIMIZED
AUTOMATED
CONTINUOUS
Monitor, measure and analyze release processes
Maximize quality as the velocity and volume grows
Improve business execution
Measure and improve team productivity and maturity
3
21
Digital TestingNavigate to Continuous Adaptive Delivery (CA-D)
15 © 2017 CA. ALL RIGHTS RESERVED.
The ability to reliably release high-quality solutions at any time
COGNITIVE ADAPTIVE TESTING
Accelerate Deliveries
Reduce Errors
Manage Complexity
Increase Visibility
Drive Collaboration
Continually Improve
Digital TestingContinuous Adaptive Testing (CAT)
16 © 2017 CA. ALL RIGHTS RESERVED.
Digital TestingWhat is analytics driven quality?
Model ProgressiveRefinement
Analytics-Driven
17 © 2017 CA. ALL RIGHTS RESERVED.
> 50%large organizations globally will compete using advanced analytics & proprietary algorithms
40%enterprises' net-new analytics investment will go to predictive / prescriptive
75%technology-oriented business intelligence competency centers will evolve to focus on information value generation
Source: Gartner
Digital TestingIntelligent Analytics is the New Battleground
18 © 2017 CA. ALL RIGHTS RESERVED.
SOI
Visualization
(Big) Data
Analytics
Inference (Machine
Learning & AI)
Intelligent Process
Automation
Digital Technology Integrations
Digital TestingSystems of Intelligence
© 2017 CA. ALL RIGHTS RESERVED.18
Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
19 © 2017 CA. ALL RIGHTS RESERVED.
Portfolio backlog prioritization and
value analysis
Improved scheduling and
capacity
Validated requirements in
backlog
Improved estimates
Predict failures based on code
commits
Dynamic code promotion
criteria
Predict defects/failures
Optimized Tests
Scenario identification
based on trends
DX Scores
Anomaly detection
PLAN DEFINE DEVELOP TEST RELEASE OPERATEPhase
Examples of Cognitive Adaptive
Intelligence
Examples of Outcomes
Investment decision analytics
Code quality
analytics
TestOptimization
& Defectprediction
A/B Testing
& Canary
Log & DX Analytics
DX-basedrequirements
Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
Digital Testing
Cognitive Adaptive Intelligence (CAi)
Systems of Intelligence
20 © 2017 CA. ALL RIGHTS RESERVED.
AUTOMATION MAINTENANCE
STATIC RULES BIG (un-structured) DATA
IOT, IOE
CX / DX SECURITY
STRUCTUREDAUTOMATION
Digital TestingKey Challenges Posed by Adaptive Digital Systems
Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
21 © 2017 CA. ALL RIGHTS RESERVED.
Failure detection and prevention
Scripted automation
Limited process automation based on static rules
Individual metrics based on structured data in lifecycle silo
Limited insight and perspectives
Reactive: deals with “Past”
Failure prediction
Automate the automation
Intelligent process automation based on dynamic inference
Analytics based on large scale structured and unstructured data across the lifecycle
Predictive: from “What did happen?” to “What would happen?
TRADITIONAL AUTOMATION
COGNITIVEAUTOMATION
Digital TestingIntelligent Automation Characteristics
Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
22 © 2017 CA. ALL RIGHTS RESERVED.
AUTOMATED & SELF-HEALING AUTOMATION
DYNAMIC/ADAPTIVE RULES
BIG DATA ANALYTICS
IOT-Ops
DX-Ops SEC-Ops
INTELIGENT / ADAPTIVE / COGNITIVE AUTOMATION
+ +
+
+
+
+
+
Digital TestingChallenges addressed by Cognitive Adaptive Testing (CAT)
Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
23 © 2017 CA. ALL RIGHTS RESERVED.
REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO
AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS
CONTINUOUS DELIVERY MAINFRAME SECURITY
23 © 2017 CA. ALL RIGHTS RESERVED.
API analyticsRelease management dashboard Access incident responseReal-time app performance
Digital TestingContinuous Adaptive Delivery (CA-D)
CA Adaptive Data Store
24 © 2017 CA. ALL RIGHTS RESERVED.
Best of Breed Open Source Technologies
(Hadoop, Spark, Kafka, Elasticsearch)
Industry Proven Architecture
(Lambda & Micro Services)
Single Code Line, Multiple Deployment Options
(Cloud, On Prem, Mainframe)
Encapsulated Complex Technology through Simple
Consumable Interfaces
(RESTful APIs)
CA Analytics
Digital TestingContinuous Adaptive Intelligence (CA-I)
Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
25 © 2017 CA. ALL RIGHTS RESERVED.
DATA COLLECTORS
Dat
a A
ggre
gato
rAgents
…
Probes
Machine/Log File Data
DATA-DRIVEN APPS
Existing CA Apps
Custom-build Apps
New CA Apps
Partner Apps
ANALYTICS ENGINE
Dat
a In
ject
ion
AP
I
Dat
a A
cce
ss A
PI
Service Layer
Dat
a Ex
trac
tio
n
Data Processing (Speed)
Data Processing (Batch)
Data Bus
Inspired by “Analytics at CA – What’s Cooking? Project Jarvis” Wednesday, November 16: 4:30 PM - 05:15 PMCA Accelerator Zone
Digital TestingCA Reference Architecture – Project Jarvis
26 © 2017 CA. ALL RIGHTS RESERVED.
Digital Testing – Diagnosis Analytics Engine
© 2017 CA. ALL RIGHTS RESERVED.
27 © 2017 CA. ALL RIGHTS RESERVED.
Emergent embedded AIfor neural networks
Digital TechnologyCognitive Adaptive Intelligence (iCA)
Singularity Ultra-Intelligence (2030)
“Let an ultra-intelligent machine be defined as a machine that far surpass all the intellectual activates of any person however clever” 1960s
28 © 2017 CA. ALL RIGHTS RESERVED.
Mode 0 - Descriptive
Cognitive Adaptive Testing (CAT):
1. “How long will it take to test these new features?” --- How long historically (test cases / execution time)2. “What is the associated risk rating of the new platform?” --- Failure rate (cyclomatic complexity > 13)3. “How much technical debt do we have?” 4. “What is the defect density and prediction for this solution?”5. “What is the current conference rating of the solution?”6. “What is the value of this capability?”
How will Intelligent Analytics help?
Digital TestingExample use cases for Intelligent Analytics
29 © 2017 CA. ALL RIGHTS RESERVED.
What were the past defect trends?
What were the root cause of the past defects?
What defects are we likely to find?
What preventive measures should we take?
Have we overlooked anything for defect prevention?
Predictive analytics
Descriptive Analytics
Diagnostic Analytics
Prescriptive Analytics
Deep Learning
Diagnosis Analytics EngineExamples of Analytics Use case
Cognitive Adaptive Intelligence (CAi)
Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
30 © 2017 CA. ALL RIGHTS RESERVED.
How do you pin point the failures within the release
pipeline?
Model-Driven Design of Tests from Use Cases 1
2
3
Execute ‘Automation as Code’ from the Model
Pinpoint Failure and Diagnose Issue from Results
“Easy, soon as you check in the new features ready for test, the automation will run then the results are analysed automatically and if we have any failures then it identified them and raise them as bugs!
Diagnosis Analytics EngineTest in DevOps (TiD)
31 © 2017 CA. ALL RIGHTS RESERVED.
Mode 2 - Predictive
Cognitive Adaptive Testing (CAT):
1. “Can you create me all the tests I need to fully test this system?”2. “How do I successfully test this new feature?”3. “How much testing has been done already?”4. “Have you detected any anomalies within the system?”5. “How much testing will be enough?”
How will Intelligent Analytics help?
Digital TestingExample use cases for Intelligent Analytics
32 © 2017 CA. ALL RIGHTS RESERVED.
Use Case A – Diagnosis Analytics Engine
Cognitive Learning
Diagnosis Analytics Engine.
Pinpoint Failure Analysis.
Actionable Insight.
Machine Deep Learning.
Always On Test (SaaS)
Test Automation as a Service.
Root Cause Analysis (RCA).
Support for Azure & AWS..
On Prem
Dynamically generated Virtual Automation Environment Provisioning.
Dynamically generated Tests based on Cognitive Adaptive Testing.
0 1 2
49 © 2017 CA. ALL RIGHTS RESERVED.
VALUE-DRIVEN DELIVERY
DIGITAL ASSURANCE & TESTING
1Cognitive Adaptive Delivery
2Cognitive Adaptive Learning
4Cognitive Adaptive
Technology
3Cognitive AdaptiveTesting
Cognitive Adaptive Technology (CA-t)Cognitive Adaptive Adoption (CA-A)
50 © 2017 CA. ALL RIGHTS RESERVED.© 2016 CA. ALL RIGHTS RESERVED.© 2017 CA. ALL RIGHTS RESERVED.
BRINGS TRUE ARTIFICIAL INTELLIGENCE (AI) TO LIFECYCLE VIRTUALIZATION Virtualize services without requiring any knowledge or
decoding of the service protocols
Applies a genome sequence alignment algorithm, discovers byte-level patterns in message protocols
Now virtualize a much wider range of protocols without requiring a new DPH
HIGH ACCURACY: 99.6 – 100%
Increased speed and accuracy with Entropy Weighting + Message Clustering
The more data a service observes the data, more intelligent it becomes
Perfect for performance testing where we deal with tons of data
Cognitive Adaptive Learning (CAI)Current Generation CA Testing™
51 © 2017 CA. ALL RIGHTS RESERVED.
Cognitive Adaptive Technology (CA-T)Next Generation of Digital Experiences (DX)
1. Digital Mobile, Chapter 2, Digital Experiences (DX), April 2017, Jonathon Wright
53 © 2017 CA. ALL RIGHTS RESERVED.
RBC Maintenance time reduced from 7.5 hours to 2 minutesCGI-Fiko: Estimated 30% effort saved on managing dataRogers: 320% over-testing reducedAMEX : 16% to 100% coverage; 5 hour test creation time reduced to 50 minutes
Customer Benefit Anecdotes Key PersonasBAs, QA Engineer, Testers, Test Automation Engineers, Site Reliability Engineers, SDET & Test in DevOps
64% of Defect Cost originate in reqs phase
Automate the automationmaintenance of test assets following change
95% reduction in time
to create test assets
- BPM Tools- AC/Jira/ALM/TFS- Testing FrameworksHyderabad Business School GITAM University Quality
Flaws: Issues and Challenges in Software Development 2012
Estimations derived from analysis of benchmark data which is a composite from multiple sources.
CA Agile Requirements Designer
CA Product modules:
CA Agile Requirements Designer
+ Automation Builder+ Data Builder+ Automator (Runtime)
Use Cases Collection:• Generate Model-Based Design
Users Stories & Requirements (ARD)• Automate Test Design / Optimize
Test Cases and Coverage• Generate Automation from Model• Link your tests with test data (TDM)• Quickly react to change
ROI Value Proposition Integration PointsChallenge
54 © 2017 CA. ALL RIGHTS RESERVED.
• Strategic initiative to drive innovation velocity and accelerate developer productivity.
• CA has earned the position of “trusted advisor”, helping to form their “OSI Model for Continuous Delivery”.
• Leveraging ARD, TDM, SV, and RA in combination.
GENERAL MOTORS Key PersonasTest Data Engineer, QA Manager, Quality Engineer, Testers, Business Analysts, Software Engineers, Release Managers
Real World - Value-Driven Delivery (VDD)
Product Feature Areas• CA Agile Requirements Designer• CA Test Data Manager• CA Release Automation• CA Service Virtualization• CA Application Test• CA Blazemeter
Use Cases Collection• Model-Driven Automation• Environment Provisioning• Functional / Integration• Virtualize Constraints• API Testing• Performance Testing• Digital Experiences
TransformTesting organizations toOptimize efficiency
React To changingRequirements faster thanEver before.
GenerateTest case automationInstead of writing themmanually
Optimize environment and data availability through automated delivery
Why CA: Don’t Take Our Word For it
CA is Positioned as a Leader in the Gartner MQ for ARA … Forrester Wave SV Leader
Faster time to deploy releases*
20x
Reduced testing costs & improved efficiency*
25%
Reduction in production errors*
98%
Differentiation Customer Success
5 of the top 5Telecom Companies
4 of the top 5 Retailers
4 of the top 5 Banks
Market Validation
Nominated in the “Best Use of Technology in a Project” category for
work with the London Metal Exchange
*Based on Global 500*Forrest TEI Research
57 © 2017 CA. ALL RIGHTS RESERVED.
Blogs #ExcuseFreeTesting – Legacy is your Legacy!
https://dzone.com/articles/legacy-is-our-legacy-1