For Testers - ANZTB AI and...Relevant Context Testing Neural Net Ranker Personalized Web Search and...

AI and Machine LearningFor Testers

Jason Arbon@jarbon jason@test.ai

Relevant Context

Testing Neural Net Ranker

Personalized Web Search and Chrome Test Automation

AI for Mobile Test Automation

AITesting.org

Testing: Apply Inputs, checking outputs.

https://www.extremetech.com/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they

AI: Apply Inputs, check outputs

Features

Complexity increases

exponentially as new features and

states interact with existing

featuresTests

Test coverage grows linearly

because tests can only be added

one at a time

Time

Co

mp

lexi

ty /

Co

vera

ge

COVERAGE GAP

Testing Needs AI

HOW A BOT LEARNS TO TAP, TYPE, AND SWIPE

Key Problems in Test Automation

1. Have to Code2. Element Selection

3. Test Step Sequencing

solved by AI systems

VS

Element Identification

Test.aiAuto Labeled

Quick Visual Custom Labeling

Selenium IDEPain Finding IdentifiersHave to write code. (brittle)

Old / Past

IDENTIFYING APP STATE

AI Auto-trains to recognize all elements and pages in your app

AI REPLACES HARD-CRAFTED TEST CASES

VS

Step Sequencing

Selenium IDEHave to code exact steps.Brittle when app changes.

Test.aiFamiliar Cucumber/GherkinOnly list labels of interesting

steps and verifications.Also, GUI Editor

Old / Past

Where’s the AI? Reinforcement learning

Robust testing

Instead of prescriptive tests, capture user intent

“Go to the shopping cart”

AIT - Open - Manual Too

VS

Test ExecutionOld / Past

VS

Test Reporting

Selenium IDEMany Tests Flaky/Break as app ChangesSimple pass/fail

Test.aiQ-Learning Test Step Generation

Element Classifiers at runtimeSee Test Steps.

Presentable.

Old / Past

Test Run Cross Platform, Cross App!

HARNESSING COLLECTIVE INTELLIGENCE FOR TESTING

How to test AI?

https://www.extremetech.com/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they

30

Testing AI: Test Boundary + InputTest Data outside the range of Training Data. Errors are at the margins.

What We Do31

Testing AI: Test Boundary InputTest Data outside the range of Training Data. Errors are at the margins.

What We Do32

Testing AI: Feature Correctness

AI “input” correctness● Features computed correctly● E.g. “#words on page”● Representative Sample?● Bad/Partial data?● Sample Temporal/Domain relevance?

Credit: https://www.researchgate.net/figure/224915935_fig2_Figure-1-Neural-Network-Architectures-Used-in-the-SimulationsTwo-different-neural33

Testing AI: Each new build--consider it a new implementation

Credit: https://www.wolfram.com34

Testing AI: “Drift”

Networks are randomly initialized

New network might be just as accurate as last networkON AVERAGE

Test for the texture change.

Mapping from network output -> product output functionally correct.

Our Story35

Testing AI: Action / Flow Graphs

Our Story36

Testing AI: App Generation and Traversal

Our Story37

Testing AI: Test Graph Generation and Traversal

What is Next?

What We Do39

Testing AI: “DARPA HARD”

What We Do40

Humans vs Bots

Humans Train Bots in Course of Work

Humans Remain to Personalize and Re-train bots.

Little testing that AI cannot do “soon”

What We Do41

Abstract Intent Language: AIT

Focus on Intent, not “how”

Human Readable

App-Independant

jason@test.ai

@jarbon

EXAMPLE: HOW TO DETECT A BUG

ML : Subjective or complex labels via Humans

AI for Testing: Input

Reduce input space

150 actions per page

35 steps -> 150^35 = 10^78 paths

#atoms in universe10^78...

ML: Page Label Training