QA Challenges in an Agile World

Presentation for Southern California Quality Assurance AssociationPresented by Yousef Abazari(yousefabazari@yahoo.com)

November 17, 2015

What is covered in this presentation Introduction About me Brief overview of Agile Brief overview of Scrum Multi-sited scrum teams Scrum without test

automation Using waterfall methods in

Scrum Lack of code review Lack of unit testing

Not capturing bugs in bug reports

Ambiguous user stories Testing a US without AC Limiting QA testing to AC Testing without TCs using AC Falling behind in testing US Q&A

Introduction I will go over some of the challenges that

software QA professionals encounter in the context of a Scrum methodology while performing their assigned scrum tasks

I will provide some of the solutions (or suggestive recommendations) to address those issues

About me I have an Electrical Engineering degree from McGill

University (Montreal, Canada) I am an experienced software QA manager. I have managed QA teams at Accurate Background

(2y), Fujitsu(11y), Ericsson (3y), Avocette (1y) and Infowave (<1y).

My special interest is managing quality, starting new QA teams, new test automation teams, and defining a new QA process and methodology that works best for a client.

What is Agile? Agile is a set of

principles and values upon which a family of incremental and iterative software development methodologies are based.

Agile Manifesto: Individuals and

Interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

What is Agile? – ContinuedAgile is based upon: Adaptive planning Quick feedback cycles Evolutionary development

and testing Customer collaboration Encourage rapid & flexible

response to change Self organizing & Cross

functional teams Early delivery Continuous improvement

Some of the Software Development Methodologies that are based upon Agile (note that this is not an exhaustive list):

Scrum Kanban Lean Extreme Programming Design Driven Development(D3) Test Driven Development (TDD) Crystal

What is Scrum? According to Merriam-

Webster: In Rugby, a way of starting

play again in which players from each team come together and try to get control of the ball by pushing against each other and using their feet when the ball is thrown in between them

A large group of people who are close together in one place

It was discussed for the first time in 1986 in an article (1)

It is one of the most popular Agile-based software development methodologies

Product or application is built in a series of time-boxed iterations called sprints

Product is built using a product backlog, which is a prioritized list of features and functionality updated continuously

(1) Scrum’s origin goes back to a 1986 article published in Harvard Business Review by Takeuchi & Nonaka.

Jeff Sutherland and his team created the Scrum software development methodology in 1993 at Easel Corp.

What is Scrum? - Continued At times Scrum is used

synonymously to Agile Work is done by a small group

of 5-7 cross- functional (multi-skilled), members sitting physically next-to-one another

At the end of each sprint, the team has a potentially (increment of) deployable product

Scrum activities are repeated in cadence (flow of planned activities repeated from one sprint to another)

Scrum Roles;- 1. Product owner- 2. Scrum master- 3. Team member (Dev, QA, DBA,

…)

Scrum Activities:- 1. Sprint execution- 2. Sprint Planning- 2. Daily scrum- 3. Sprint Review- 4. Sprint Retrospective- 5. Product Backlog grooming- 6. Product (increment) deployment

What is Scrum? - Continued Scrum artifacts:- 1. Product Backlog- 2. Product Backlog Items:

User Story (Feature or part of a feature), Defect, Task, Test task, Design spike, Knowledge Acquision task

- 3. Sprint Backlog (a subset of Product Backlog that is being implemented in the current sprint)

- 4. Shippable (increment of) product

Scrum team participants and corresponding numbers in a team of 5-7 members:

- 1. Product owner (0.5-1)- 2. Scrum master (0.5-1)- 3. Developers (3-5)- 4. Testers (1-2)- 5. DBA (0-1)

It is important to keep the scrum team size small and the number of participants between 4-7 to allow it to operate effectively.

Sprint Life Cycle ActivitiesSprint

Planning

Acceptance Testing

Reg /Other QA

Testing (1)

Development & Coding

Sprint Review

Sprint Retrospective

Sprint Deployment

BacklogGrooming

Sprint Activities

(1) This can also be Single Sprint Level Acceptance Test (Product Increment Acceptance Test) and/or Multi-Sprint Level Acceptance Test (Product Acceptance Test) and/or Non-Functional Acceptance Test.

P#1: Multi-sited scrum teams Problem: Placing members of the scrum

team physically in separate locations (offshore/ onshore/ different cities/buildings/floors) impacts the team’s daily interaction, cohesion & communication

Solution: Co-locate all team members in one place, otherwise, plan regular visits, start daily team chat board, daily calls, 1:1 messaging between team members, regular updates to tickets’ progress on scrum board.

PO: Product Owner

P#2: Scrum without test automation Problem: Lack of smoke/ regression/ functional

test automation impacts quality and adequate test coverage of earlier user stories implemented in this/previous sprints

Solution: Automate AC for each US, automate Smoke/Reg/Functional TCs, Identify/reduce dependency of USs, Involve developers in TA, Make TA a mandatory part of Scrum activities, Create/run sprint-level/multi-sprint-level E2E/Reg test cases

AC: Acceptance Criteria E2E: End-to-End Reg: Regression TA: Test Automation US: User Story

P#3: Using waterfall methods in Scrum Problem: Making USs available for testing at the end

of sprint, making many of US un-testable, developing/ testing many (not 1 or 2) USs by one person at the same time, lack of cross-functional approach by scrum members, changing the scope of a sprint mid-way, changing sprint length, Lack of pre-defined cadence/rhythm for sprint activities, Not applying the lessons learned (coming out of Sprint Retrospective) from this sprint to the next sprint, Lack of deployable product increment at the end of a sprint that stops the team from getting the customer/business stakeholder’s feedback.

US: User Story

P#3: Using waterfall methods in Scrum Solution: Avoid Waterfall method inclusion in Scrum if you can:

• Make user stories available for testing early in the sprint. • Make all user stories testable by providing Acceptance Criteria.• Each team member should only work on 1-2 US at the time and no more.• Each available scrum team member should help others performing their

tasks. QA help Developers and Developers helping QA, etc.• Do not change the scope of a sprint mid-way.• Do not change the sprint length while the sprint is in progress.• Run sprint activities as planned (i.e. Sprint Planning on day 1, One daily

scrum in the morning, Sprint Retrospective on the last day of sprint, etc.)• Collect feedback from the team in Sprint Retrospective and apply

Lessons Learned to the next sprint. Target problems and not a person.• Target to have a deployable product increment at the end of each sprint

in order to collect customer/business stakeholder’s feedback and also identify issues earlier.

US: User Story

P#4: Lack of code review in Scrum Problem: Lack of code review (specially for complex

code being written/modified) results in build failures, creating sub-optimal code, introducing catastrophic run-time errors, adversely impacting existing code, implementing user stories that go back between QA and Dev multiple times, and impacting team velocity.

Solution: Ensure senior developers review the code created by junior or mid-level developers. Capture review time in user story estimation during sprint planning. You can add a state such as Ready for Code Review on Scrum board. Capture review comments. Avoid bottle necks for code review.

P#5: Lack of unit testing in Scrum Problem: Lack of unit testing results in build failures, creating

new code that does not work, bugs being uncovered later in the process while testing a US, breaking existing functionality when the code is changed later, burdening QA by making it busy dealing with rudimentary bugs that should have been caught earlier, forcing QA to expand the AC for the US.

Solution: Write unit test for each public method, include the unit test code with the base code and release it to the code repository, make unit tests independent from one another, consider unit test as an additional documentation of new code base, run unit test before or as part of creating a new build & ensure unit test covers both happy path and error scenarios. When providing estimate for US, consider unit testing as well. Code review is not a replacement for unit test and vice versa.

AC: Acceptance Criteria US: User Story

P#6: Not capturing bugs in bug reports Problem: Communicating bugs (without capturing them

in bug reports) to developers verbally while running acceptance criteria for user stories is a common tendency among scrum participants. This saves them a bit of time up front. However, creating bug reports allows collecting QA metrics, provides a history of an issue, allows issues to be viewed by people outside the scrum team, allows bug reports to be added to the product/sprint backlog and proves that the QA analyst is doing his/her job right.

Solution: Capture all bugs in bug reports and link them to the user story if you can. While estimating a user story, take into account the time needed to open defects and to verify them later on.

Defining what the software should and should not do is a business decision (decided by the PO), not a technical decision.

P#7: Ambiguous User Stories Problem: The problem with implementing an ambiguous user

story is not knowing clearly what needs to be implemented. This happens when a PO is not familiar with the functionality being built, or doesn’t have clear business requirements to create the user story, or lacks the time to investigate the subject matter area to ‘cook’ the user story. Also, this happens when there is not enough info provided by the customer about a functionality.

Solution: The scrum team should be courageous enough to reject a US when it is not ready to be implemented or ask the PO to provide more details/info. A good US has 5 (SMART) or 6 (INVEST) characteristics:

SMART: Specific, Measurable, Achievable, Relevant, and Time-boxed.

INVEST: Independent, Negotiable, Valuable, Estimateable, Small, and Testable.

An ambiguous US has no AC, lacks details, lacks user information, is not testable and/or lacks business value for the customer.

P#8: Testing a US without AC Problem: A user story without an AC is considered an

untestable US or ambiguous US. The problem with such a US is that there is no written agreement on when it is considered Done. In addition, it is harder to size a US without AC. Lack of AC in a US makes it without adequate content to work on.

Solution: Make sure every US has one or more AC. Acceptance Criteria within a US: defines what (and not how it) is being implemented, identifies the boundary of a US, provides the definition of Done for a US, allows QA to test the US to see if “It Works as business Intended” and not “I think it works or it works as coded”, and helps to size a US. Use the format GIVEN, WHEN, THEN layout to fill out the Acceptance Criteria information.

Acceptance Criteria are the functional/non-functional conditions/statements with a clear pass/fail result that are specified within a US that must be met by an implemented software in order for the US to be considered “Done” (completed).

P#9: Limiting QA testing to AC Problem: There are many circumstances within which regular US specific AC

testing may not be adequate:- Implementing a new US may break (under certain

circumstances) prior implemented user stories in this sprint.- Implementing a new US may break (under certain

circumstances) prior implemented user stories from previous sprints

- When multiple scrum teams work on the same application and their code can impact one another

- When scrum team is not performing in-scope negative scenarios, boundary conditions and non-functional testing (i.e. performance, stress, load, soak, security, browser dependency testing)

- When scrum team is not performing Product level acceptance testing.

AC: Acceptance Criteria US: User Story

P#9: Limiting QA testing to AC - Continued Solution: In such cases additional testing is to be added in

the form of:- Introducing test-only spikes or test-only USs with focus on

types/areas of testing that are missing.- Introducing Hardening sprint(s) with focus on types/areas

of testing that are missing.- Involving business SMEs and/or POs and/or QA in such

test activities- Creating & running an auto smoke/regression test- Use metrics to measure the success/impact of these

additional testing on quality improvement- Additional Test spikes or USs or tasks can be performed

each sprint or the last sprint depending on the type of testing being performed and scope of testing (To be decided by the scrum team in Sprint Planning meeting)

AC: Acceptance Criteria PO: Product Owner SME: Subject Matter Expert US: User Story

P#10: Testing without TCs using AC Problem: Limit Scrum testing to AC provides a good coverage of a product

but under certain circumstances (listed below) it may not be adequate:- There are no detailed test steps, environment, data, setup,

precondition, post-condition and expected result information that appears in a TC.

- There are no product level user stories and acceptance test- Important business scenarios (E2E) that traverse over several

functional areas/applications are not captured in any user stories- Negative data/operation/flow/browser/functional scenarios are not

captured in any user stories.- Non-functional test scenarios are not covered in any user stories.

Solution: Create appropriate test-only spikes/USs and include them on the Scrum board during the Sprint Planning meeting. It is very helpful to have a traceability matrix/page added to TCs mapping US to TCs.

AC: Acceptance Criteria E2E: End-to-End TC: Test Case US: User Story

P#11: Falling behind in testing USs Problem: Members of the scrum team at times fall

behind in coding or testing USs due to various reasons including incorrect estimation, unexpected issues, late availability of many USs for testing, or interrupting urgent issues that take scrum members’ time and bandwidth.

Solution: For non-urgent US, if you cannot complete a US, complete it in the following sprint. However, in most cases, less busier members of the scrum team should pitch-in and help in coding or testing of the USs which are at risk of not being completed. Remember that the essence of Scrum is a cross-functional team who is willing and capable of taking different roles and helping members whenever needed.

US: User Story

Questions and Answers If you have any questions, please pose it now.