Embed Size (px)
Citation preview
W6 AI and Data Analytics Wednesday, October 17th, 2018 11:30 AM
How AI Is Transforming Software Testing
Presented by:
Raj Subramanian
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/
Raj Subramanian Raj Subramanian is a former developer who moved to testing to focus on his passion. Raj currently works as a Developer Evangelist for Testim.io, that provides stable self-‐healing AI based test automation to enterprises such as Netapp, Swisscom,Wix and Autodesk. He also provides mobile training and consulting for different clients. He actively contributes to the testing community by speaking at conferences, writing articles, blogging, making videos on his youtube channel and being directly involved in various testing-‐related activities. He currently resides in Chicago and can be reached at [email protected] and on twitter at @epsilon11. He actively blogs on www.testim.io and his website www.rajsubra.com. His videos on testing, leadership and productivity can be found here http://www.rajsubra.com/my-‐youtube-‐channel/
www.testim.io
ABOUT ME
● Developer Evangelist at Testim.io● Blogger, Speaker and Writer
○ http://www.rajsubra.com/blog/○ https://blog.testim.io/○ http://www.rajsubra.com/publications/○ http://www.rajsubra.com/presentations/○
● 11+ Years of Development and Testing
Twitter: @epsilon11
E-mail: [email protected]
AGENDAAGENDA
LEARN THE BASICS OF AI
CURRENT CHALLENGES HOW AI CAN SOLVE THESE CHALLENGES
FUTURE OF AI
CONTEXTMATTERSCONTEXTMATTERS
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative,Smart testing
THE FUTURE
EVOLUTION OF TESTINGEVOLUTION OF TESTING
www.testim.io
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative,Smart testing
THE FUTURE
EVOLUTION OF TESTINGEVOLUTION OF TESTING
www.testim.io
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative,Smart testing
THE FUTURE
EVOLUTION OF TESTINGEVOLUTION OF TESTING
www.testim.io
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative,Smart testing
THE FUTURE
EVOLUTION OF TESTINGEVOLUTION OF TESTING
www.testim.io
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative,Smart testing
THE FUTURE
EVOLUTION OF TESTINGEVOLUTION OF TESTING
www.testim.io
Manual Testing
Waterfall methodology
1980 - 1990
Bulky Automation Tools
Experimentation with Different Development Approaches
1990 – 2003
More Robust Automation tools + Open Source Frameworks
Agile Approaches, faster release cycles
2003 - 2010
More about Scale
DevOps, Continuous Testing, CI/CD
2010 - CURRENT
Autonomous Testing, Machine Learning and AI
Collaborative,Smart testing
THE FUTURE
EVOLUTION OF TESTINGEVOLUTION OF TESTING
www.testim.io
BASICSOF AIBASICS OF AI
Artificial Intelligence (AI)
www.testim.io
DE-MYSTIFYING AIDE-MYSTIFYING AI
Machine Learning (ML)
Deep Learning (DL)
Size Price
Size
Bedrooms
Family Size
Zip code
Wealth
Walkable
School Quality
Price
www.testim.io
DEEP LEARNING EXAMPLEDEEP LEARNING EXAMPLE
CURRENTCHALLENGESCURRENTCHALLENGES
Not at all, we test last, we test quality into the product
Testing involved early, but still have an extended test phase before release
We could probably release after every sprint
We could ship daily if we wanted to
We could deploy any minute of any day
13%
57%
19%
10%
2%
YOUR RELEASES?HOW FREQUENT ARE HOW FREQUENT ARE YOUR RELEASES?FROM A RECENT SURVEY TESTIM COMPLETED
www.testim.io
Not at all, we test last, we test quality into the product
Testing involved early, but still have an extended test phase before release
We could probably release after every sprint
We could ship daily if we wanted to
We could deploy any minute of any day
13%
57%
19%
10%
2%
YOUR RELEASES?HOW FREQUENT ARE HOW FREQUENT ARE YOUR RELEASES?FROM A RECENT SURVEY TESTIM COMPLETED
www.testim.io
Not at all, we test last, we test quality into the product
Testing involved early, but still have an extended test phase before release
We could probably release after every sprint
We could ship daily if we wanted to
We could deploy any minute of any day
13%
57%
19%
10%
2%
YOUR RELEASES?HOW FREQUENT ARE HOW FREQUENT ARE YOUR RELEASES?FROM A RECENT SURVEY TESTIM COMPLETED
www.testim.io
CURRENT CHALLENGESCURRENT CHALLENGESRAPID RELEASE CYCLESRAPID RELEASE CYCLES
Innovators 2.5%
Early Adopters
13.5%
Chasm
Early Majority
34%
Late Majority
34%Laggards
34%
Every two weeks Every day 5 times
a day
www.testim.io
Every 6 month
Every month
THE CHALLENGE:THE CHALLENGE:#1 SKILL SET#1 SKILL SET
www.testim.io
Innovators 2.5%
Early Adopters
13.5%
Early Majority
34%
Late Majority
34%Laggards
34%
Every two weeks Every day 5 times
a dayEvery 6 month
Every month
www.testim.io
Chasm
THE CHALLENGE:THE CHALLENGE:#1 SKILL SET#1 SKILL SET
www.testim.iowww.testim.ioSource: http://dilbert.com/
THE CHALLENGE:THE CHALLENGE:#1 SKILL SET#1 SKILL SET
www.testim.iowww.testim.io
THE CHALLENGE:THE CHALLENGE:
Innovators 2.5%
Early Adopters
13.5%
Early Majority
34%
Late Majority
34%Laggards
34%
Every two weeks Every day 5 times
a dayEvery 6 month
Every month
Chasm
#2 AUTHORING#2 AUTHORING
Chasm
www.testim.io
#3 INITIALIZATION
Innovators 2.5%
Early Adopters
13.5%
Early Majority
34%
Late Majority
34%Laggards
34%
Every two weeks Every day 5 times
a dayEvery 6 month
Every month
Chasm
#3 INITIALIZATIONTHE CHALLENGE:THE CHALLENGE:
Chasm
www.testim.io
www.testim.io
www.testim.io
#4 MAINTENANCE#4 MAINTENANCE
Chasm
Innovators 2.5%
Early Adopters
13.5%
Early Majority
34%
Late Majority
34%Laggards
34%
Every two weeks Every day 5 times
a dayEvery 6 month
Every month
THE CHALLENGE:THE CHALLENGE:
Chasm
www.testim.io
Innovators 2.5%
Early Adopters
13.5%
Early Majority
34%
Late Majority
34%Laggards
34%
Every two weeks Every day 5 times
a dayEvery 6 month
Every month
Chasm
THE CHALLENGE:THE CHALLENGE:#5 SCALE#5 SCALE
Chasm
www.testim.io
HOW AI CAN SOLVETHESE CHALLENGESHOW AI CAN SOLVE THESE CHALLENGES
www.testim.io
TESTIM.IOTESTIM.IO
www.testim.io
AI Based Locators
MULTIPLE ATTRIBUTES VS. SINGLE ONE
Hundreds of attributes used to identify elementsA few changes don’t break the test
LOW MAINTENANCE
No need to (tediously) update locators
MORE EXECUTIONS => MORE STABLE
With each execution we learn the changes to the application and optimize scores
Source: Jason Arbon
www.testim.io
FEATURES
Complexity increases exponentially as new features and states interact with existing features
TESTS
Test coverage grows linearly because they can only be added one at at time
COVERAGEGAP
THE COVERAGE GAPTHE COVERAGE GAPIN SOFTWARE TESTINGIN SOFTWARE TESTING
Time
Com
ple
xity
/ C
over
age
FUTUREOF AIFUTURE OF AI
TESTINGAUTONOMOUSAUTONOMOUSTESTING
www.testim.io
LEARN BY OBSERVATION (PRODUCTION)
AGGREGATE USER ACTIONS INTO FLOWS
TEST PRODUCED FROM FLOWS
www.testim.io
VIA UI TDDAUTHOR FASTERAUTHOR FASTERVIA UI TDD
RUN DURING DEVELOPMENT
UPDATE ON FIRST GREEN TEST(MULTIPLE LOCATORS)
USE DESIGN MOCKSTO AUTHOR (IMAGE BASED)
AUTOMATIC RESPONSEAUTOMATIC RESPONSE
www.testim.io
SERVERSTESTS’ UI INTERACTION
RECORDSERVER RESPONSE
CALL TO SERVER ARE RECORDED1
RESPOND WITH MOCKS2
SERVERSTESTS’ UI INTERACTION
PLAYSERVER RESPONSE
www.testim.iowww.testim.io
HOW TESTIM ISHOW TESTIM IS PAVING THE FUTUREPAVING THE FUTURE
QUALITY VELOCITY
Create much more scenarios in a shorter period of time
Deploy resources much more effectively create the tests proactively
Dev/QA friction is minimized root cause analysis
Easier for QA to maximize user coverage
Take a risk based approach with true data driven decisions
Prevent bugs as opposed to fixing them
THE QUALITY/VELOCITY DILEMMAAUTONOMOUS TESTING ELIMINATESAUTONOMOUS TESTING ELIMINATES THE QUALITY/VELOCITY DILEMMA
www.testim.io
WRAPIT’S AIT’S AWRAP
Email: [email protected] | Website: www.testim.io | Twitter: @epsilon11 | Youtube: Testim Channel
www.testim.io
https://blog.testim.io/summary-blog-how-ai-is-transforming-software-testing/
SUMMARYTALKTALKSUMMARY
Email: [email protected] | Website: www.testim.io | Twitter: @epsilon11 | Youtube: Testim Channel
