Embed Size (px)
DESCRIPTION
A look at testing as it used to be and the way it can be done in an enterprise where the benefits of continuous integration combined with multi-level test automation are understood.
Citation preview
1
Test Automation
The Backbone of an Agile Software Development Team
Towo Toivola
Lead Quality Engineer, F-Secure Corporation
F-Secure Corporation
Provocative
UnclearInteractiv
e
16 SW yearsQuality
Agile Lean
Towo
© F-Secure Confidential4
Bonus: Ricardo Kekki
Senior Quality Engineer
5
Are you in?
6
ROI!
7
”Kyseenalaistakaa kaikki mitä sanon,
mutta kyseenalaistakaa avoimella, älkää
jääräpäisellä mielellä”
8
Contents• In the olden days..• What testing used
to be• What testing used
to be for
• Test Automation compared to manual testing•New world of working together
•New value of testing: CI+TA=status•Great advantages of TA•Modern Test Automation is..• Imagine 100 people working on a Trunk..• ’Done includes TA’
9
In the Olden Days..
10
In the olden days..
Testing was.. Manual Boring Unrespected Tedious Slow Expensive Out-sourced Off-shored
11
In the olden days..
Organizations used testing for..
1. Find out what work is still needed
2. Verify that we are good to release
Practically this meant: Find list of bugs we may decide to fix later Last mile before releasing, a ’check for
confidence’- Usually a preventer of release..
12
In the olden days..
Feedback cycles were weeks or months
More calendar time means: More effort in between More uncertainty More investigation cost More discord between testing and programming More open bugs More wasted effort in testing and programming More requirements change Less customer feedback
How is this supposed to
work?
Reproduce it with the
latest version.
We need PSG
decision on bugfixing.
EXPENSIVE!
13
Test Automatio
n
Towo Toivola 2009
Man is Superior to the Machine
Works around problems Common sense Creativity Understanding of the user Understanding of priorities Understands change Looking outside the box The most important bugs are found in the
requirements and assumptions
This was rarely
leveraged in the olden
days..
Towo Toivola 2009
The Machine is Superior to
Man Automation is faster Automation has more attention to detail Automation can run through more data Automation can execute more
combinations Automation can measure more exactly Automation does not get tired Automation works through the night Automation provides exact logs
Software
Tool
Test
16
The New Age of
Cooperation and
Results
17
New Age..The Value of
Testing
CI + TA = status
© F-Secure Confidential18
Test on multiple levels
”Are we stable? Can we
move on?”
Does it make sense to perform
exploratory testing now?
Can we give this to others
now?
New Age..CI+TA=status
Legend
Scanbox version maturity state
Environment / tool
Activity / test
Decision
Scanbox build smoketests Jenkins + Xingu
”Not broken”
Flow Time
<10 min
”Stable”
Binary Repository
Scanbox regression TA Jenkins + Xingu <120 min
Malware detection rate TA Jenkins + Orange Scanbox performance & reliability TA Jenkins + Xingu
”Super Stable”
Promote to Alfa
”Alfa”
Nightly / on-demand
Deployment, Long test, monitoring, perf check
Alfa environment in AWS
User group
Other dev teams
1-3 times per week
1-10 days
Promote to Beta
”Beta”
Deployment, monitoringBeta environment in AWS
External beta use
1-10 days
Promote to Production Release
Out.
From the test automation
radiator we see how our release
process is doing.
Feedback cycles are minutes!
20
Great Advantages of
Test Automation ”If it doesn’t have test automation, I will not
touch it.” ”How can I fix it if I can’t tell if it works?” Brings the skills of software engineers and test
engineers together Shifts focus from ’How’ to ’Why’ Guides architecture and scope decisions
21
Better than Specs!
It is specific It is unambiguous It is version controlled You can verify if software meets it You can verify your ’fix’ is an improvement It is written by people who know the software You can go and fix it if it is broken
22
Modern Test Automation
Fast Disciplined Jointly owned Business critical The priority!
Few things are as
expensive as broken
test automation.
23
How do you get it?
24
Are some of your projects large?
© F-Secure Confidential25
Facebook:
© F-Secure Confidential26
Facebook:
27
Imagine 100 people working on
a trunk..
28
My local code
My system tests
Team system tests
I am mister
programmer My unit
tests
Team code
Project code
Project system tests
Good version
Exploratory system tests
Beta version
GIT
Jenkins
Eucalyptus
Python
Jira
29
April 9, 202330
Thank you for your time.
PLEASE FEEL ENCOURAGED TO APPROACH ME FOR A DISCUSSION.
