Xbosoft Agile Test Management · we’ll demonstrate how to implement agile test management in TFS using a scrum background. We can use the Scrum template (contains backlog, task,

AGILE TEST MANAGEMENT

WITH VISUAL STUDIO

1

any companies are implementing an agile methodology, but often still have waterfall based tools. We’ve been working on

several agile projects, one of which we collaborate with our client using the .Net platform with Visual Studio. We use VS

together with TFS (Team Foundation Server) and MTM (Microsoft Test Manager) to seamlessly manage the entire testing M

1WWW.XBOSOFT.COM CONTACT US AT [email protected]

life cycle including task assignment, test case design, test execution and results tracking. Although we use many agile methods with

our clients, probably the most common is scrum, or some form of it. Scrum, originally from an abbreviation for scrummage, is a

method of restarting play in rugby where players pack closely together with their heads down while attempting to gain possession

of the ball. Hence the roots of one of the most popular agile development methods with close collaboration.

This white paper discusses Visual studio and its components, Microsoft Test Manager and Team Foundation Server, in using them to

implement an agile testing methodology with a particular focus on scrum. First we’ll describe the scrum roles and process because

these terms are used later with VS. Then we’ll describe and show how it is implemented using Visual Studio.

Scrum is one of several agile methods (including XP, Kanban, Crystal Methods, FDD) that have periodic sprints (iterations) and

specific roles in an iterative and incremental framework for managing product development.

SCRUM PROCESS OVERVIEW

FIGURE 1: BASIC SCRUM PROCESS

» In summary, scrum projects consist of a series of "sprints“ where each sprint is an iteration and with each iteration, the

software is delivered as ‘working’ to the product owner. The definition of ‘working’ varies amongst organizations but in

most cases means that the software feature or function has been completed through testing. Sprints are usually referenced

or named as: current sprint, future sprint, past sprint. Sprints are typically 1-4 weeks in length where actual duration

depends on the organization’s development environment and team. The main point is to have ‘shorter’ versus longer

iterations as per the agile manifesto.

» The Product Owner, who represents the end users and stakeholders, decides what goes into the product backlog as

shown in Figure 1. The product backlog is an ordered list of software requirements that is maintained for a product.

Requirements are basically work items, ‘things to do’. This could include anything it takes to get the product out the

door. It consists of features, bug fixes, non-functional requirements, etc. Product backlog items are prioritized by the

Product Owner based on considerations like risk, business value, dependencies, date due, etc.

» For each sprint, the Product Owner will move the task from the product backlog to a specific sprint, according to the

product requirements, the features that will be designed, built and tested during a sprint. A sprint should also include the

work needed to fix defects as decided by the team. Part of this step in the process involves estimating the work involved and

assigning tasks out to individuals on the team. A customer or product owner should not change the requirements or work

activities during a sprint, but occasionally this does happen. That is one of the features of a short iteration. That is, to focus for

a short period without change.

» Then each day, there is a daily scrum meeting where the team members put their heads together (not exactly like in rugby

but you get the idea), and discuss basically what they will do for the day, what they accomplished the day before and what

problems or issues they have. They have the daily scrum meetings until the end of the sprint.

» At the end of the sprint, there is a demo of the “working” product to the product owner, where “working” and “done”

should be defined in the beginning of the project. Depending on the definitions of ‘working’ and ‘done’, the product

may be delivered to the end user as a product increment. There is also usually a retrospective where the whole team

discusses what worked, what didn’t work, and improvements that could be made.

1


SCRUM PROCESS OVERVIEW

The person who is responsible for

creating and prioritizing the product

Backlog and reviewing the ‘working’

product (with other stakeholders) at

the end of the Sprint.

PRODUCT OWNER

Scrum is facilitated by a Scrum Master,

who is accountable for delivering the product

goals. The Scrum Master can also be a team

member that also participates in product

development (not just management). The

scrum master manages the whole process

keeping everyone focused and serves the

team to remove blocking issues.

SCRUM MASTER

Team members committed to achieving

a Sprint Goal and do whatever it takes to

achieve the goal. Team members should be

multi-disciplinary, but most will gravitate toward

their strong suit which is also best for the team.

Team members include product managers,

developers, QA, etc.; basically whoever is

involved and delivering the product.

TEAM MEMBERS

1


Team Foundation Server (commonly abbreviated to TFS) is a Microsoft

product which provides source code management , reporting,

requirements management, project management , automated builds

and testing. In other words, it covers the entire application lifecycle. TFS

enables teams work together in different locations via a web portal,

meaning that testers, developers and product owners can work and

collaborate together on an agile project.

Microsoft Test Manager (MTM) is part of the Microsoft Visual Studio

toolset designed for test managers and testers to plan, manage, and

execute both manual and automated tests. It used to be a separate

component, but since VS 2012, it has been merged into Visual Studio.

FIGURE 3: TFS AND VS (MTM) FUNCTIONS

Since MTM (Microsoft Test Manager) was merged into Visual Studio Studio 2012, all the functions that were provided by MTM also can

be done in the VS. Therefore from this point forward, we will combine the MTM and VS together and will not discuss MTM separately.

FIGURE 2: UNDERSTANDING HOW TFS, VS

AND MTM WORK TOGETHER

Create backlog items

Create tasks

Design text cases

Link cases with tasks

Associate automation script

Create test plan

Execute test cases

Check result

FUNCTION TFS MTM VS

PRIMER ON TFS, MTM AND VS

RELATIONSHIP OF TFS, MTM AND VS

WHAT IS MTM?

For a task that is a new feature to be developed, we usually divide

the requirements into different user stories of how that feature

would be typically used. For those created tasks for the user story,

some of the tasks are development related work while others are

testing related work. Only after all of the tasks of the user story

have been finished, then this user story is considered ‘done’.

1


Now that we’ve provided the basic introduction to some basic TFS functions and terms,we ’ l l demonstrate how to implement agi le test management in TFS us inga scrum background.

We can use the Scrum template (contains backlog, task, bug ……), which is created by Microsoft for convenience, but you can also

do your own, or customize the one provided. As in any agile project, we need to manage the Product Backlog. We do this by first

creating Backlog items. Backlog items are all the things that we are going to do in the project. By simply clicking “+ Product

Backlog Item” button, you can create a backlog item. Go to the “Work” page as shown in Figure 4 and you can see the item has

been created successfully.

CREATE BACKLOG ITEMS IN TFS

FIGURE 4: CREATING BACKLOG ITEMS IN TFS

TO CREATE A PRODUCT BACKLOG

Click the “+ Product Backlog Item”

button. The “Work” page will show

your newly created Backlog.

FIGURE 5: CREATING TASKS IN TFS

1


CREATE TASKS IN TFS

1

2

3

4

5

6

7

As mentioned earlier, a key part of scrum is breaking down work into tasks with estimates on amount of work or hours per task.

Let’s create a task for this backlog item.

Go to the “Board” page under Work .

Click the “+” icon to add a task. Fill in the necessary information, such as ‘Assigned To’, ‘Activity’, ‘Original estimate’ and

‘Remaining Work’.

‘Assigned To’ means who will be responsible for the work. As shown below in Figure 5 as an example, Angela will responsible

for this work. We will show this work in VS later on.

In the ‘Activity’ column we can choose the work type. If we choose ‘Developer’, this means this task belongs to a developer.

From Figure 5, we can see the task is a testing task.

‘Original Estimate’ is the time that we originally estimate for how long the task will take. We fill the actual time that we have

spent for this task in ‘Completed Work’ column as well as hours in the ‘Remaining Work’, if more time is still needed for

accomplishing the work.

In the ‘Backlog Priority’, usually the Product Owner will define the priority of a developers’ task or it will be discussed in the

planning meeting. For testing work, we determine the priority based on workload and time to release.

FIGURE 6: CREATING TEST CASES IN TFS

1


CREATE TEST CASES IN TFSHere, we show how to create and manage your test cases in TFS & VS and the relationships and differences between them.

» To create a test case in TFS, click Work Items > New > Test Case.

Then complete the test case steps and expected results on the right-hand side.

TIP: When we design the case, we usually put the actual operation in ‘Action’ column and in the

‘Expected result’ column we usually put the expected result according to the product requirements.

TIP: Only after you update your TFS to TFS 2012 Update 3, then you can add steps here. Otherwise,

the “Click or type here to add a step” won’t work; and you have to use the following way.

Before design a test case in VS (MTM), we need to connect to TFS. Let ’ s see how to do th is step by step :

FIGURE 7: CONNECT TO TEAM FOUNDATION SERVER TO DESIGN TEST CASES

FIGURE 8: MY WORK ON THE PROJECT

Once connected to TFS, click “My Work” under Team Explorer Home.

1


DESIGNING TEST CASES IN VS

To create a test case in Visual Studio, with the assigned user...Connect to Team Foundation Server -> select the correct project (Meetingpoint4 as example).

TIP: You should have a TFS account with correct permissions

1


In scrum, there are different kinds of tasks and roles in TFS as we have mentioned before. For example, a developer will see

developers’ tasks. As a tester, we would see the testing tasks. Here we examine a tester’s work.

» ‘In Progess Work’ is a task status, which means the tasks are being done by me now. If you have to stop your

current work to look at another bug, you can just hit ‘Suspend’ button. This will move the work

to ‘Suspended Work’ and save your current working environment. After you have finished the

other bug, then you can click the ‘resume’ button and you are back where you left off.

» ‘Suspended Work’ is a task status, which means those tasks are stopped for some reason and cannot be continued.

» ‘Available Work Items’ is also a task status, which contains all the To-Do work that belongs to me.

FIGURE 9: WORK LIST FOR THE USER LOGGED IN- A TESTER EXAMPLE

We drag the task “69197 – Perform test” to

“In Progress Work” > click New > Test Case

FIGURE 10: ADDING A TEST CASE

After clicking the ‘Test Case’, now you will see the following

screenshot as shown in Figure 11 (on next page). However, even

though you can fill in the test case name and save it in Visual

Studio, you cannot add steps for this case in VS.

Below is my To-Do work list:

1


FIGURE 11: ADDING STEPS TO A TEST CASE

FIGURE 12: ADDING STEPS TO A TEST CASE

This is most likely a defect in Visual Studio. Microsoft provides a workaround – clicking “Edit with Microsoft Test Manager” to edit in

MTM. Figure 12 shows how it looks in MTM where you click Insert Step to add a step to a test case.

Usually there are many test cases that can use parts of other test cases. This is where you need to think carefully about test

organization and design. When you want to use common parts of test cases, use ‘Insert shared steps’. This enables you to share the

common actions for different cases.

1 Click the ‘Organize’ -> ‘New’ -> ‘Save and close’ button after finishing the shared steps as shown in Figure 13.

2 Click on the ‘Steps’ -> ‘Insert shared steps’ as shown in Figure 14.

FIGURE 13: CREATING SHARED STEPS FOR TEST CASES

FIGURE 14: INSERTING SHARED STEPS FOR TEST CASES

1


USING SHARED STEPS — COMMON ACTIONS FOR TEST CASES

FIGURE 15: LINKING TEST CASES WITH TASKS

You should link your test cases to their original tasks such that every test case is connected to a task. By linking all of the cases to

corresponding tasks, it makes it convenient to manage the test cases. This can be done in TFS, VS, or MTM as shown in Figure 15.

The following screenshot demonstrates linking test cases to tasks in VS. Remember that MTM and TFS work the same.

1


LINK TEST CASES WITH TASKS

With agile, we want to automate as much as possible if it makes sense. So if you have fairly stable features and code, we usually start

to think about what parts of the testing can be automated. If you have created a CodedUI script for a test case, you can associate these

scripts. You can use VS to create automated tests of the user interface (UI) known as coded UI tests. These tests provide functional

testing of the UI and validation of UI controls. Automated UI tests enable you to test that the UI is functioning correctly after code

changes. Coded UI tests are quicker to run than manual tests, meaning that you can run them more frequently and faster.

ASSOCIATE AUTOMATION SCRIPT

FIGURE 16: SIMPLE EXAMPLE OF TEST AUTOMATION WITH CODEDUI

FIGURE 17: ASSOCIATING AUTOMATION WITH A TEST CASE

However, automated testing totally can’t take the place of the manual testing. If the requirement always changes (as they often do

in agile) or the logic is complex, then you may continually need to make changes to your code to either optimize or maintain. So,

you have to look at each feature and function to decide if automation is suitable.

Learning to write automated tests in CodedUI takes some time. Here in Figure 16, we show a simple verification log using CodedUI.

1


Please note: » Associating an automation script can ONLY be done in Visual Studio.

» You need to rebuild the solution (rebuild solution is a function provided by VS) before associating

the automation scripts.

FIGURE 18: CREATING A TEST PLAN

Please note: » Regarding how to organize the test plan, we organize the test plan by Company Products. This way it is

convenient for us to see how many User Stories were created for a product as shown in Figure 18.

» The ID of a test case is generated automatically, which means that each test case has a unique ID.

» Test suite: in software development, a test suite is a collection of test cases that are intended to be used

to test a related function of a software program.

1


Up to this point, we have created test cases. Many think that test plans are not needed in agile which is true to an extent. You don’t need

a detailed plan spelling out everything, but it’s still good to group test cases together to make them easier to manage into test suites. Here,

we show you how to create a test plan to execute the test cases. The name of the test plan is up to you, but it should be easy to understand,

reference and find not only for you, but for the whole team.

CREATE TEST PLAN

TO START TEST CASE

Click Start Test as shown in Figure 19. Then as shown in Figure 20, you can fill “Pass” or “Fail” for each

step. If you find a defect, click the “Create Bug” icon to add a defect, also shown in Figure 20, then add the steps as shown in Figure 21.

FIGURE 20: PASS/FAIL FOR EACH STEP

FIGURE 19: EXECUTING TEST CASES

1


The benefit of putting test cases into a test plan is that you can execute them together. To do this, go to the Test tab in MTM. Make sure

you are at the correct test plan, right click a test case and start execution.

EXECUTE TEST CASES

If the test case is not automated, you will see a window like this:

FIGURE 21: ENTERING DEFECT STEPS

FIGURE 22: TFS OVERVIEW CONNECTING BUGS AND TASKS

Actually, it’s more convenient to create the bugs in TFS because in TFS shows an overview of the bugs with the user story or task

as shown in Figure 22.

1


1


If the test case you want to run is associated with an automation script, right click the case, then select “Run with options” as

shown in Figure 23.

FIGURE 23: EXECUTING AN AUTOMATED TEST CASE

FIGURE 24: CHOOSING THE EXECUTION ENVIRONMENT FOR AUTOMATED TEST CASES

You are then able to select on which build and environment

you want to use for executing the test case as shown above.

FIGURE 25: EXECUTING AN AUTOMATED TEST CASE

Here’s what it looks like when the test case starts to run.

It will invoke the associated CodedUI automation script,

and run the test in the script as shown in Figure 25.

1


After test case execution, go to Plan > Results to check the results. You can change the view based on Test Suites, Test Configuration, and

Testers separately. Figure 26 shows comprehensive results for all of the sprints.

We can find lots of useful information in Figure 26. Let’s examine some of the most important elements.

TEST SUITES (TEST PLAN): We could choose a specific sprint to check all of the results for that sprint as shown in Figure 27.

CHECKING RESULTS

FIGURE 26: EXAMINING TEST CASE EXECUTION RESULTS

FIGURE 27: EXAMINING RESULTS FOR A SPECIFIC SPRINT

FIGURE 28: EXAMINING RESULTS FOR DIFFERENT CONFIGURATIONS

1


TEST CONFIGURATIONS: If you have different configuration options for the test plan, then you can see various configurations. In this

example however, there is only one configuration:

Wow. That was a lot. We started off by providing an overview of agile and the scrum process. After defining some common terms in scrum

such as roles and sprints, we showed some typical agile test management tasks using TFS.

We’ve shared some lessons learned along the way via tips and tricks that we’ve discovered. We hope that you’ve found this brief tutorial

helpful. If you have any questions or comments, please let us know via email; [email protected].

CONCLUSION

SOME OF THE TASKS WE DEMONSTRATED INCLUDED:

» Create Backlog Items in TFS

» Create Tasks in TFS

» Creating Test Cases in TFS

» Using Shared Steps – Common Actions for Test Cases

» Link Test Cases with Tasks

» Associate Automation Script

» Create Test Plan

» Execute Test Cases

» Checking Results

ABOUT XBOSOFT

XBOSoft is a full-service software and application testing company serving both the US and European markets. With a proven track

record working for Fortune 100 companies to small independent developers, XBOSoft delivers a full range of software testing

services. XBOSoft’s testing services range from Functional testing to Security and Mobile Usability and UX consultation with

collaboration tools for secure 24/7 access to project information.

WEBSITE: www.xbosoft.com

EMAIL: [email protected] XBOSOFT

Documents

Xbosoft Agile Test Management · we’ll demonstrate how to implement agile test management in TFS using a scrum background. We can use the Scrum template (contains backlog, task,