us.sogeti.com

COGNITIVE QA: LEVERAGE AI AND ANALYTICS FOR GREATER SPEED AND QUALITY

COGNITIVE QA: LEVERAGE AI AND ANALYTICS FOR GREATER SPEED AND QUALITY

us.sogeti.com

ARTIFICIAL INTELLIGENCE vs. COGNITIVE COMPUTING

2

Build a system that

can generally perform

any intellectual task –

so called Strong AI

Build a system that

perform can an

intellectual task in

smaller domain – so

called Weak AI

Ability to learn and build

models, so that a system

perform activities like

prediction within specific

domains

Build a system that can

learn and interact in

natural way with humans

Emerging of Deep

Learning with neural

networks with many

layers, new topologies

and learning methods

Artificial Intelligence

Machine Learning

Cognitive Computing

1950 1960 1970 1980 1990 2000 2010

us.sogeti.com

Structured and active Unstructured and dark

Data that’s coming

• Customer records• Transactional systems

• Predictive models

• Institutional expertise• Operational systems

• News• Events

• Social media

• Weather• Geospatial information

• Internet of Things (IoT)• Sensory data

• Images

• Video

Data outside your firewallData you possess ++

CONNECTING DISPARATES TYPES OF DATA WITHIN AND OUTSIDE YOUR WALLS CREATES NEW OPPORTUNITIES FOR UNEXPECTED INSIGHTS

us.sogeti.com

WITH WATSON YOU CAN HARNESS THE POWER OF AUGMENTED INTELLIGENCE

Your data + the world’s data

Watson can ingest, cleanse, and comprehend multiple data types. Build off your domain expertise and the largest base of industry offerings in the market.

Ingests + transforms

Apply a powerful suite of Watson services and APIs to transform your data into valuable insights. Watson is optimized for robust AI workloads and lives on IBM Cloud.

Insights + outcomes

Watson extracts meaning to provide deep insights and produces powerful outcomes. Train your AI on what’s important to your industry gain knowledge and make the most informed, reasoned decisions.

StructuredUnstructured

Transform

Insights

us.sogeti.com

IBM WATSON IS THE AL PLATFORM FOR BUSINESS

Watson is the AI platform for business. With Watson

you can discover hidden insights, engage in new ways, and decide

with greater confidence.

us.sogeti.com

BY APPLYING COGNITVE CAPABILITIES TO YOUR DATA, YOU WILL ENHANCE DIGITAL INTELLIGENCE EXPONENTIALLY

REASON

They can reason, grasp underlying

concepts, form

hypotheses, and infer and extract ideas.

UNDERSTAND

Cognitive systems understand

imagery, language

and other unstructured data

like humans do.

LEARN

With each datapoint, interaction

and outcome, they

develop and sharpen their expertise, so they

never stop learning.

INTERACT

With abilities to see, talk and hear,

cognitive systems

interact with humans in a natural way.

us.sogeti.com7

WHAT CUSTOMERS ARE SAYING

“Intelligent test automation and smart analytics will become essential

to support testing as they enable smart decision making, fast

validation and automatic adaptation of test suites.”

World Quality Report 2017-18 findings suggest

How to ensure sufficient

quality without slowing down

on speed?

Which test cases are

relevant and which ones

are not?

How to improve testing with

user behavior patterns?

How to keep up with increasing complexity of

intelligent applications?

us.sogeti.com

KEY QA CHALLENGES TODAY

• Validating the increasing complexity of the application landscape is becoming more and more difficult

• Keeping pace with the demand for increasing speed to market is challenging

• Test sets are either missing up to 30% of relevant tests and/or containing many irrelevant test scenarios

Increasing speed and complexity

• Automation is currently under-exploited in QA & Testing and as a result there is too much reliance on manual activities

• Traditional automation approaches have delivered only limited benefits

• Smart applications demand intelligent automation, integrated in continuous delivery

Insufficient Automation

• Test environments and test data are the most cited challenges for core enterprise applications as well as more fluid IT applications

• Smart test platforms are required to deal with the growing challenges of test environments, data and virtualization

Test environments and test data

• Usage patterns and user feedback are not exploited for QA purposes

• Customer experience validation is a challenge

• Social media and other external data sources are seldom leveraged for QA purposes

Client focus inside out

us.sogeti.com9

COGNITIVE QA VALUE

• Leveraging self-learning and analytical technologies for

• Predictive QA Dashboards

• Smart Analytics for QA

• Intelligent QA Automation and

• Cognitive QA Platforms

• Enabling smart quality decision making on factual project data, actual usage patterns and user feedbacks

• Delivering quality with speed in a complex connected world at optimized cost.

COGNITIVE QA

INTELLIGENT QA

AUTOMATION

COGNITIVE QA

PLATFORMS

PREDICTIVE QA

DASHBOARDS

SMART ANALYTICS

FOR QA

With Cognitive QA organizations will be able to achieve

an accelerated and optimized quality by using an

intelligent approach to QA>

us.sogeti.com1010

DEVELOP A VISION AND ROADMAP

Predictive QA Dashboards

• X-application transparency

• Near real-time insight

• What-if prediction

Smart Analyticsfor QA

• Efficiency through focused Testing

• Testing aligned with actual usage patterns

• Improved test coverage

• Improved risk management

Intelligent QA Automation

• Maximized RoI on test automation

• Focused test automation aligned with actual usage patterns

• Accelerated, touch-less testing.

Cognitive QA Platforms

• Instant validation.

• Optimized test coverage with self adapting test suites.

• Minimized idle time and cost with self-aware and adaptive environment provisioning.

• Optimized risk management with self-learning capabilities.

Accelerated, intelligent QA

leveraging AI and Analytics.• Modular.• Innovative.• Business oriented.

Future Proof

QA

Guidance to the journey towards

quality with speed.

us.sogeti.com

PREDICTIVE QA DASHBOARDS

11

Leverage all relevant data from Dev / Test / Ops repositories for:• 360o Visibility – get comprehensive and transparent views on application quality on management and operational level.

• Tailor-made Insights – customized for your specific demands to allow better control and steering of quality across all your projects.

• Near Real-Time Updates - refreshed continuously to enable timely and improved decision making.

• Zoom In – drill down in details to do root cause analysis of real issues and analysis on expected outcomes.

• Alerts – avoid surprises with pro-active notifications to minimize the risk of serious errors.

• What-if Analysis – optimize test strategy in line with available resources and coverage requirements.

• Predictive Analysis – leverage historical usage patterns by applying proven algorithms for defect prediction and test coverage optimization.

us.sogeti.com

Implemented step-by-step to solve your Business Challenges

•Automated Test Case Selection and Prioritization – gain efficiencies through focused testing effort.

• Predictive Test Environment Configuration – minimize idle time with optimized environment provisioning.

• Test Automation Prioritization – maximize ROI of test automation effort.

• Test Coverage Optimization – make best possible choices based on ‘what if’ analysis.

• Test Execution Analysis – align test with actual usage patterns.

• Auto Generation of Test Scripts – create optimized test suites while reducing preparation effort.

SMART ANALYTICS FOR QA

12

Proven Analytics Solution Elements delivering Real Business Benefits

> 40 tried and tested rules

• Rules of Thumb

• Common Sense

• Best Practices

• Intuitive Judgments

• Positive / Negative Patterns

Applying our IP

Kick-StartKick-Start Proof of ValueProof of Value DeploymentDeployment

• Collecting basic information using Cognitive QA Pre-Assessment questionnaire.

• Checking availability, accessibility and quality of data.

• Performing Solution Build Workshop to define the scope of POV.

• Collecting basic information using Cognitive QA Pre-Assessment questionnaire.

• Checking availability, accessibility and quality of data.

• Performing Solution Build Workshop to define the scope of POV.

Conducting Proof of Value with following steps:• Verify analytics readiness.• Data collection and preparation.• Modeling Cognitive QA rules.• Validation and reporting.• Define the deployment approach.

Conducting Proof of Value with following steps:• Verify analytics readiness.• Data collection and preparation.• Modeling Cognitive QA rules.• Validation and reporting.• Define the deployment approach.

Operationalizing rules implemented during PoV. Major activities involved:• Implement connectors to pull live

data.• Implement orchestration engine to

automate workflows.• Implement portal (responsive web).

Operationalizing rules implemented during PoV. Major activities involved:• Implement connectors to pull live

data.• Implement orchestration engine to

automate workflows.• Implement portal (responsive web).

us.sogeti.com

Source ctrl /Build Mgmt System

Project Mgmt System

GrouperGrouper

Data RulesData Rules

Software Lifecycle Artifact

Quality

Software Lifecycle Artifact

Quality

Non-structured

Data

structured

Data

Test & Dev

Repositories

Self Learning

Recommendation Engine

Self Learning

Recommendation Engine

TestOps

Rules

TestOps

Rules

Test Results

Optimized Test list based

on real time data

Test Engine

Feedback

Test Mgmt System

Req. Mgmt system

Code Quality Parameters

Operations

Repositories

RTM Systems

Usage patterns

Support engr call notes

App. Logs and traces

structured

Data

Non-structured

Data Defect Mgmt System

SOLUTION CONCEPT OF SMART ANALYTICS

us.sogeti.com

TOOLS AND TECHNOLOGIES

14

• Automatic Regression Test Prioritization and Execution

‘PoV in a box’-assets• Pre-assessment questionnaire• Solution Build workshop template• Assessment checklists• Cognitive QA rules

• Report template• Customizable Dashboard with Standard Metrics• Proof of Value references

IN-HOUSE ASSETS

INDUSTRY TECHNOLOGIES

us.sogeti.com

CASE STUDY

15

INDUSTRIAL AUTOMATION OEM CUSTOMER

• Dynamic Regression Test Selection • PoV duration 10 weeks• 6 Cognitive QA rules implemented • The Total number of Test Cases were 22,289 ,

and regression identified for the current release was 8,773.

• We recommended 2,288 regression Test Cases.

• Dynamic Regression Test Selection • PoV duration 8 weeks • 6 Cognitive QA rules implemented• The Total number of Test Cases were 13, 721 • We recommended regression Test Cases of

1, 664.

• Selection of Configurations and Tests that matter

• PoV duration 10 weeks• 8 Cognitive QA rules implemented• We recommended: 325 high priority & 571

low priority test suites of 896 total test suites.

• We recommended : 44 high priority & 141 low priority configurations among 192 Configurations.

• ~73% effort saving, shorter release cycle

• Team gets more time to automate

• The high risk test cases are executed in the

beginning of the test cycle, early detection of

regression defects.

• Fixing the identified tool and process gaps will

prepare for better “Analytics Readiness”.

• ~73% effort saving, shorter release cycle

• Team gets more time to automate

• The high risk test cases are executed in the

beginning of the test cycle, early detection of

regression defects.

• Fixing the identified tool and process gaps will

prepare for better “Analytics Readiness”.

KEY BENEFITS

• Improved Adaptive Regression Test Selection

strategy factoring multiple inputs.

• More quality builds for special purpose testing

like system testing, performance testing.

• Recommended test list is being automated and

made part of regular automated test

executions.

• Fixing the identified tool and process gaps will

prepare for better “Analytics Readiness”.

• Improved Adaptive Regression Test Selection

strategy factoring multiple inputs.

• More quality builds for special purpose testing

like system testing, performance testing.

• Recommended test list is being automated and

made part of regular automated test

executions.

• Fixing the identified tool and process gaps will

prepare for better “Analytics Readiness”.

• Helped identify 63% test cases that can be

deprioritized.

• Derived critical tests and configurations with

high probability of risk.

• Systematic way of System under Test

configuration planning .

• Better utilization of the hardware/ Cost etc.

• Helped identify 63% test cases that can be

deprioritized.

• Derived critical tests and configurations with

high probability of risk.

• Systematic way of System under Test

configuration planning .

• Better utilization of the hardware/ Cost etc.

KEY BENEFITS

MEDICAL IMAGING SOLUTION PROVIDER

KEY BENEFITS

us.sogeti.com16

THANK YOU

us.sogeti.com17

BACKUP

us.sogeti.com

THERE ARE LOTS OF USE CASES

Improve coverage!

Optimize and align test sets based on real application usage

Test Coverage Optimization

Release Readiness

Predict and advice marginal value of additional testing

Automated impact Analysis

Quality Dashboard

Cost of a bug, Test efficiency, Release Dashboard

Predictive quality metrics

Auto-Heal

Throw away the Bad/ obsolete test cases (future proof testware )

Isolate Script Failures

Throw away the bad code fixes in CI pipeline

Automated Test Selection & Prioritization

Predictive test environment configuration

What should we automate?

Tests that matter!

Resource utilization

Auto Generation of Test Scripts

Predict risk level of product components / services / systems (incl. UX)

Predictive capacity planning

us.sogeti.com

RULES SELECTION