Upload
maurizio-mancini
View
384
Download
1
Embed Size (px)
Citation preview
MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY
How to build a QA Team that matters
in an
Agile or Incremental and Iterative (IID) world.
GIANCARLO BISCEGLIA • SENIOR QA MANAGER • YELLOW PAGES CANADA
Who are we?
• Head of QA for Yellow Pages Canada (YP)
• Also responsible for Software Process Deployment at YP
• 25 years of software engineering experience
Maurizio Mancini
Giancarlo Bisceglia
• Senior QA Manager for Yellow Pages Canada (YP)
• 15 years of software engineering experience
About Our Team
• YP has ~ 2300 employees and ~ 1 Billion in Revenue
• Our team of 100 QA are embedded into 15 development teams
• Our team is geographically disbursed in Montreal (4 locations) and in Toronto (2
locations)
• Our team works on projects using all types of software development process
(Agile, Incremental & Iterative (IID), Waterfall)
Agenda
• Yellow Pages Yesterday and Today
• QA “State of the Nation” from our perspective
• How to hire and build a QA Team that will succeed in any SDLC
• Agile Testing – What’s that about?
• Test Automation – Fundamental Building Block
• The Testing Pyramid – Where does QA fit in?
• Different Layers of Testing
• Where else does QA fit in an Agile or IID World?
Yellow Pages Canada
• 3 years ago no real QA Team existed
• A few scattered QA souls left to fend
for themselves
• A lot of consultants
• Almost all testing was manual, very
little automation
• A lot of “End to End” testing
• Most QA was done by BA’s and The
Business
Yesterday Today• Team of 100 QA Engineers, Test
Automation Developers, and QA
Analysts
• A full QA practice that uses common
tools and QA approaches
• On average our testing is 50-50
Automated/Manual
• QA Teams focus on System Testing and
BA/Product Owners focus on UAT
• QA teams are Embedded with the Dev
Teams
QA “State of the Nation”
• Over the last 30 to 40 years, the QA software industry has gone from being a
“nice to have”
“a must”
“developers will do it all” in the Agile world
QA “State of the Nation”Like all extremes, we believe that “pendulum of sensibility” is starting to settle in
the middle
The Death of QA… One of the heated debates in the industry is that QA is dead with Agile becoming
the predominant software development process…
We like to say that Agile has finally brought balance to the QA world…
Agile has brought balance to the QA world
More developers than testers, too much testing,
not enough time for QA to test everything…
Agile has helped rebalance the inequities that existed when it came to testing
software….
Agile has brought balance to the QA world
• Agile has helped refocus development teams on the importance of testing.
• Agile has re-asserted what Software Engineering 101 has said all along…
Unit Testing is a MUST!
TESTING PYRAMIDTraditional Goal
End to End GUI Tests
Functional Tests
Integration Tests
Unit Tests
GUI Tests
Functional Tests
Integration Tests
Unit Tests
Q
A
DEV
Donald Knuth
He once warned in a letter to one of his fans,
"Beware of bugs in the above code; I have only proved it correct, not tried it."
Knuth has been called the "father of the analysis of algorithms"
‘To tweak what Knuth said,
"Beware of the above code. I have only tested that it works. I haven't actually tried it.“’
http://jjinux.blogspot.ca/2012/05/unit-tests-dont-find-bugs-death-of-qa.html
Shannon Behrens
HiringWhich Roles should you be hiring for the current world of Software Development
that includes Agile, IID and sometimes waterfall?
• QA Engineer
• Also known as QA Specialist, QA Developer, Engineer in Test
• Test Automation Engineer
• QA Analyst
Hiring
What should the mix be?
• QA Engineer (75%)
• Also known as QA Specialist, QA Developer, Engineer in Test
• Test Automation Engineer (10%)
• QA Analyst (15%)
Candidate Profiles
Early Career – Look for people that think they want to be a developer.
Mid Career – Look for someone that may have coded before and has
discovered the love of testing.
Late Career – Look for someone with a passion for testing and breaking
software.
QA EngineerAlso known as a QA Developer, QA Specialist, or Engineer in Test
Key skills and qualities you should be looking for:
Coding and/or scripting – Experience in Record and Playback not good enough
API and Web services background – ex: REST and SOAP
SQL and database background
Networking background – Should have the basic knowledge of networking
Linux/UNIX and/or Server background
Passion for testing
Solid understanding of QA and how it differs from Dev
Understands that there is more to testing than “executing a test”
Someone that likes to get under the hood QA Mechanic
Test Automation DeveloperThis will be the person that will put in place your test automation framework
• Helps to implement test automation best practices for QA Engineers
• Will support the team as they use the framework in their project
• Helps conduct QA Teams code reviews
Test Automation DeveloperKey skills and qualities you should be looking for:
This is a pure developer that also has a passion for testing
Object Oriented coding skills
Scripting language skills
Deep understanding of QA Automation Tools
API and Web services background
SQL and database background
Networking background
Linux/UNIX and/or Server background
Solid understanding of QA and how it differs from Dev
Understands that there is more to testing than “executing a test”
Someone that likes to get under the hood QA Mechanic
QA AnalystThis is your manual tester
• This is the person that will also drive Exploratory Testing
• This resource can play other roles – More on this later
QA Analyst
API and Web services background
SQL and database background
Networking background
Solid understanding of QA and how
it differs from Dev
Understands that there is more to
testing than “executing a test”
Has a natural tendency to want to
understand how something works
Key skills you should be looking for:
Passion for testing
Not just a GUI tester, someone that likes
to get under the hood QA Mechanic
Drives Technology in his/her home
(Hacks home technology)
Examples:
• Uses Opensource firmware on their
home router
• Pushes usage of his/her smart phone
beyond everyday usage
• Able to clean a computer, infected with
a virus, without the help of a tool
Hiring Team
• You should form a Hiring Team
• Formulate a hiring plan – Who you are looking for, what are the criteria
• Each member of the team focuses on a set of skills to look for
• Have a battery of questions that the team uses. For example:
Linux/UNIX
SQL
Networking
Database
Coding
Actually test something live with bugs
• This will ensure that you are hiring consistently and that you build that awesome
QA Team
The Interview• QA Engineers and Test Automation Developers should be interviewed by a
Developer too
• Questions must include coding questions
• Your questions must include “white boarding”
• Include multi-dimensional questions
• Multi-System
• Many Technologies
• System Architecture Questions – Are they able to visualize it?
• Goal is to reveal if the candidate has a QA Mechanic mentality
• Do they have a QA mindset? More on this later.
• Do they have a desire to break software?
• Are they technically sound in the technical skills that you are looking for?
• Do they have the passion for QA?
Central QA but with an Embedded Focus
• Common set of Automation Tools
• Shared Automation Framework
• Common standards
• Common Hiring practices
• Able to shift a QA resource when that resource wants to advance their career
• Helps to avoid unproductive Agile Teams
Benefits of a Central QA Team
Benefits of an Embedded QA Team
• QA involved in the daily team interactions
• QA testing early and often
• QA helps to keep a process focus within the team
• Get the benefits of a true agile team
TESTING PYRAMIDTraditional Goal
End to End GUI Tests
Functional Tests
Integration Tests
Unit Tests
GUI Tests
Functional Tests
Integration Tests
Unit Tests
Q
A
DEV
UNIT Testing is great but you need other types of testing!
• Integration Testing
• Stubbing external dependencies to facilitate UI automation
• Exploratory Testing
• Performance Testing
• UAT using BDD’s
Unit and
Integration
Stubbed
Functional Testing
System
Integration
Performance UAT
Owner Dev QA Engineers QA Engineers
QA Analysts
QA Engineers Product Owner
QA Analyst
Type White Box Black Box Black Box Black Box Black Box
Environment None Local Deployment
or CI Server
“Prod-like”
QA
Environment
Performance
Environment
Staging or
Training
Environment
Complexity
Focus
Low Level
Correctness
Maintainable
Negative
inputs
Mid Level
Extreme cases
Negative tests
UI Permutations
Failure Tests
Mid Level
Positive Tests
Exploratory
tests
System
Integration
High Level
Scalability
Robustness
Reliability
High Level
Business
Flows
Layers of Testing
• Many paid tools are based on open source libraries nowadays.
• Don’t think that because you pay for a tool, it’s best of breed.
• Beware of Scriptless tools. You still need people that can code reusable classes. Yes you get a library but how complete is it?
• Record and playback just encourages end to end testing.
• Automation project needs to be run like a dev project, including code reviews
• Invest in a custom framework which uses libraries or frameworks that can be swapped out.
Don’t tie yourself to one technology
Ideal for helping Product Owners and QA Analysts define business requirements in an English like syntax that can be automated.
Limitations of BDD’s• Good to use for top of pyramid tests, flows that don’t change often• Not a replacement for unit testing or automated functional tests• Best if it’s a layer you add on top of your existing automation
framework• Syntactically discourages code reuse• Test Steps are expensive to maintain
BDD Usage in AutomationBehavior – Driven Development
• Time for Automation is considered expendable
• Environment Deployment Automation
• Coding Standards for the software being developed
Test Automation Challenges
• Most expensive testing you can do
• Too late in the Cycle
• Hard to coordinate and manage
• Don’t make this your primary method of testing
• Use it as a last resort
End to End Testing
• No more excuses, QA unblocks itself
• Think outside the box, to find ways to test partial implementations (UI not implemented)
• Test system components in isolation using stubs
• Test Early, Test Often
• Push for continuous integration
• Deploy software and maintain QA environments
• Work with DevOps on deployment Automation
The New QA Mindset
Now What?
• Train Existing Staff – On the Job, Classroom, Self Study
• Hiring new people – QA Engineers, Test Automation Developers
• New Approaches to Testing – Testing Pyramid
• Embed your team(s) if you haven’t already
• Change in Roles and New Roles
How do you present your new approach to your existing staff?
QA is there to make developers and
development better.
We don’t ensure better software – we
enable developers to build better
software.James Whittaker
TESTING PYRAMIDTraditional Goal
End to End GUI Tests
Functional Tests
Integration Tests
Unit Tests
GUI Tests
Functional Tests
Integration Tests
Unit Tests
Q
A
DEV
What is the difference between QA and QC?
Quality Assurance – Is focused on the development, implementation and
measurement of software processes in an organization.
Focused on Defect Prevention
Quality Control – Is focused on the comparison of a product against the
requirements and ensuring that the requirements have been met.
Reactive approach focused on defect detection and removal
LESS
MORE
QA Coaching Role
QA can play a vital role coaching the
Development Team on how to test…
…so that Developers can create better
Unit and Integration tests.
But Manual Testing is still important…
Automated Tests are great but
look around, it is still humans
using the software…
Exploratory Testing is
essential… and QA is key in
this role!
Let’s not forget System Integration Testing
Most systems today integrate
with many other systems
SIT is essential…
and QA is key in this role!
User Community of TestersIf your systems and applications allow it , then why not create a
User Community of real testers?
Have QA manage your user
community of testers!
Employees
Early Adopters