Embed Size (px)
Citation preview
AW7 Advanced Technical Wednesday, November 7th, 2018 1:30 PM
How AI Is Transforming Software Testing
Presented by:
Raj Subramanian Testim.io
‘
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/
Raj Subramanian Raj Subramanian is a former developer who moved to testing to focus on his passion. Raj currently works as a Developer Evangelist for Testim.io, that provides stable self-‐healing AI based test automation to enterprises such as Netapp, Swisscom,Wix and Autodesk. He also provides mobile training and consulting for different clients. He actively contributes to the testing community by speaking at conferences, writing articles, blogging, making videos on his youtube channel and being directly involved in various testing-‐related activities. He currently resides in Chicago and can be reached at [email protected] and on twitter at @epsilon11. He actively blogs on www.testim.io and his website www.rajsubra.com. His videos on testing, leadership and productivity can be found here – http://www.rajsubra.com/my-‐youtube-‐channel/ Â
11/1/18
1
HOW AI IS
www.testim.io
Raj Subramanian [email protected]
TRANSFORMING SOFTWARE TESTING
www.testim.io
ABOUT ME
● Developer Evangelist at Testim.io ● 12+ Years of Development and Testing ● Blogger, Speaker and Writer
○ http://www.rajsubra.com/blog/ ○ https://blog.testim.io/ ○ http://www.rajsubra.com/publications/ ○ http://www.rajsubra.com/presentations/
Twitter: @epsilon11
E-mail: [email protected]
@epsilon11
11/1/18
2
AGENDA AGENDA
LEARN THE BASICS OF AI
CURRENT CHALLENGES HOW AI CAN SOLVE THESE CHALLENGES
FUTURE OF AI
CONTEXT MATTERS
CONTEXT MATTERS
11/1/18
3
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative, Smart testing
THE FUTURE
EVOLUTION OF TESTING EVOLUTION OF TESTING
www.testim.io @epsilon11
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative, Smart testing
THE FUTURE
EVOLUTION OF TESTING EVOLUTION OF TESTING
www.testim.io
11/1/18
4
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative, Smart testing
THE FUTURE
EVOLUTION OF TESTING EVOLUTION OF TESTING
www.testim.io
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative, Smart testing
THE FUTURE
EVOLUTION OF TESTING EVOLUTION OF TESTING
www.testim.io
11/1/18
5
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative, Smart testing
THE FUTURE
EVOLUTION OF TESTING EVOLUTION OF TESTING
www.testim.io
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative, Smart testing
THE FUTURE
EVOLUTION OF TESTING EVOLUTION OF TESTING
www.testim.io
11/1/18
6
BASICS OF AI BASICS OF AI
Artificial Intelligence (AI)
www.testim.io
DE-MYSTIFYING AI DE-MYSTIFYING AI
Machine Learning (ML)
Deep Learning (DL)
@epsilon11
11/1/18
7
Size Price
Size
Bedrooms
Family Size
Zip code
Wealth
Walkable
School Quality
Price
www.testim.io
DEEP LEARNING EXAMPLE DEEP LEARNING EXAMPLE
@epsilon11
www.testim.io @epsilon11
11/1/18
8
DEMO: QUICK DRAW DEMO:
QUICK DRAW
CURRENT CHALLENGES
CURRENT CHALLENGES
11/1/18
9
Not at all, we test last, we test quality into the product
Testing involved early, but still have an extended test phase before release
We could probably release after every sprint
We could ship daily if we wanted to
We could deploy any minute of any day
13%
57%
19%
10%
2%
YOUR RELEASES? HOW FREQUENT ARE HOW FREQUENT ARE YOUR RELEASES? FROM A RECENT SURVEY TESTIM COMPLETED
www.testim.io @epsilon11
Not at all, we test last, we test quality into the product
Testing involved early, but still have an extended test phase before release
We could probably release after every sprint
We could ship daily if we wanted to
We could deploy any minute of any day
13%
57%
19%
10%
2%
YOUR RELEASES? HOW FREQUENT ARE HOW FREQUENT ARE YOUR RELEASES? FROM A RECENT SURVEY TESTIM COMPLETED
www.testim.io
11/1/18
10
Not at all, we test last, we test quality into the product
Testing involved early, but still have an extended test phase before release
We could probably release after every sprint
We could ship daily if we wanted to
We could deploy any minute of any day
13%
57%
19%
10%
2%
YOUR RELEASES? HOW FREQUENT ARE HOW FREQUENT ARE YOUR RELEASES? FROM A RECENT SURVEY TESTIM COMPLETED
www.testim.io
12345
SKILL SET
AUTHORING
MAINTENANCE
SCALE
INITIAL STATES
www.testim.io
PARADIGMS CHALLENGES
@epsilon11
11/1/18
11
THE CHALLENGE: THE CHALLENGE: #1 SKILL SET #1 SKILL SET
www.testim.io www.testim.io @epsilon11
THE CHALLENGE: THE CHALLENGE: #2 AUTHORING #2 AUTHORING
www.testim.io @epsilon11
11/1/18
12
#3 INITIALIZATION #3 INITIALIZATION
THE CHALLENGE: THE CHALLENGE:
www.testim.io @epsilon11
www.testim.io
11/1/18
13
www.testim.io
#4 MAINTENANCE #4 MAINTENANCE
THE CHALLENGE: THE CHALLENGE:
www.testim.io @epsilon11
11/1/18
14
THE CHALLENGE: THE CHALLENGE: #5 SCALE #5 SCALE
www.testim.io
HOW AI CAN SOLVE THESE CHALLENGES
HOW AI CAN SOLVE THESE CHALLENGES
11/1/18
15
www.testim.io
TESTIM.IO TESTIM.IO
www.testim.io
AI Based Locators
MULTIPLE ATTRIBUTES VS. SINGLE ONE
Hundreds of attributes used to identify elements A few changes don’t break the test
LOW MAINTENANCE
No need to (tediously) update locators
MORE EXECUTIONS => MORE STABLE
With each execution we learn the changes to the application and optimize scores
@epsilon11
11/1/18
16
Source: Jason Arbon
www.testim.io
FEATURES
Complexity increases exponentially as new features and states interact with existing features
TESTS
Test coverage grows linearly because they can only be added one at at time
COVERAGE GAP
THE COVERAGE GAP THE COVERAGE GAP IN SOFTWARE TESTING IN SOFTWARE TESTING
Time
Com
plex
ity /
Cov
erag
e
@epsilon11
FUTURE OF AI FUTURE OF AI
11/1/18
17
TESTING AUTONOMOUS AUTONOMOUS TESTING
www.testim.io
LEARN BY OBSERVATION (PRODUCTION)
AGGREGATE USER ACTIONS INTO FLOWS
TEST PRODUCED FROM FLOWS
@epsilon11
www.testim.io
VIA UI TDD AUTHOR FASTER AUTHOR FASTER VIA UI TDD
RUN DURING DEVELOPMENT
UPDATE ON FIRST GREEN TEST (MULTIPLE LOCATORS)
USE DESIGN MOCKS TO AUTHOR (IMAGE BASED)
@epsilon11
11/1/18
18
AUTOMATIC RESPONSE AUTOMATIC RESPONSE
www.testim.io
SERVERS TESTS’ UI INTERACTION
RECORD SERVER RESPONSE
CALL TO SERVER ARE RECORDED 1
RESPOND WITH MOCKS 2
SERVERS TESTS’ UI INTERACTION
PLAY SERVER RESPONSE
www.testim.io www.testim.io
HOW TESTIM IS HOW TESTIM IS PAVING THE FUTURE PAVING THE FUTURE
11/1/18
19
QUALITY VELOCITY Create much more scenarios in a shorter period of time Deploy resources much more effectively create the tests proactively Dev/QA friction is minimized root cause analysis
Easier for QA to maximize user coverage Take a risk based approach with true data driven decisions Prevent bugs as opposed to fixing them
THE QUALITY/VELOCITY DILEMMA
AUTONOMOUS TESTING ELIMINATES AUTONOMOUS TESTING ELIMINATES THE QUALITY/VELOCITY DILEMMA
www.testim.io @epsilon11
www.testim.io
https://blog.testim.io/summary-blog-how-ai-is-transforming-software-testing/
SUMMARY
TALK TALK SUMMARY
Email: [email protected] | Website: www.testim.io | Twitter: @epsilon11 | Youtube: Testim Channel
11/1/18
20
WRAP IT’S A IT’S A WRAP
Email: [email protected] | Website: www.testim.io | Twitter: @epsilon11 | Youtube: Testim Channel
