AI and Machine Learning for Testers

Jason Arbon, CEO @Appdiff

Ai for Test Automation

2

Relevant Context

Testing Neural Net Ranker

Personalized Web Search and Chrome Test Automation

AI for Mobile Test Automation

Ai for Test Automation

Agenda

AI For Testing

Testing AI

Future

Appdiff Presentation4

The Real Data Scientists

Joanne TsengData Scientist @appdiff.com

Appdiff mission:

Transform app

development with

automation & insightsFrancis Iannacci

Lead Data Scientist @appdiff.com

Testing is much like AI5

Testing: Apply Inputs, checking outputs.

https://www.extremetech.com/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they

6

AI: Apply Inputs, check outputs

AI for App Testing

Features

Complexity increases

exponentially as new features and

states interact with existing

featuresTests

Test coverage grows linearly

because tests can only be added

one at a time

Time

Co

mp

lexi

ty /

Co

vera

ge

COVERAGE GAP

7

Testing Needs AI

Our Story8

Definition

When will AI Start Testing?

Our Story10

AI for Testing

What We Do11

ML : Subjective or complex labels via Humans

Our Story12

AI for Testing: Input

Reduce input space

150 actions per page

35 steps -> 150^35 = 10^78 paths

#atoms in universe10^78...

Our Story13

AI for Testing

Our Story15

AI for Testing

Our Story16

AI Driven Test Flows

Our Story17

AI Driven Test Flows

Our Story18

AI Driven Test Flows

Our Story19

Coverage

Coverage

What We Do21

ML : App Changes Don’t Break Bot Testing!

Our Story22

AI bots can test almost any app. 10,000+ apps tested.

Our Story23

Testing the Conference App

Our Story24

Benchmarked Performance

portfolio

AI for Testing26

Current AI Coverage: ~2/3rds Existing Tests

Long Sequences of Dependant Actions and Verifications

Basic Tasks (Login, Search, Create Account, Add items to Card, etc.)

Specific Sequences of Events with Specific Input (search for ‘beanie babies’, etc.)

AI for Testing27

Automation Coverage: Bots soon 100%

AI for Testing28

Automation Coverage: Bots... soon after, 10X

Canonical Tests

Learning Tests Cross-app

Our Story29

Testing AI: Test Graph Generation and Traversal

What We Do30

Testing Flow with AI

What We Do31

Abstract Intent Language: AIT

Focus on Intent, not “how”

Human Readable

App-Independant

Our Story32

AI for Testing

QnAhi@appdiff.com

Jason Arbon, CEOjason@appdiff.com

Appendix

What We Do35

ML: Page Label Training

Our Story36

Appdiff Represents the Evolution of Software Quality

1

AD HOC TESTINGReactively test

2

MANUALTESTINGProactively test

3

TEST AUTOMATION

Automate repetition

4

AI-DRIVENAPPROACHAccelerate coverage

App

QA

App

Automation

QA

App