MTLM Visual Studio 2010 ALM workshop - day1

DAY 1 MTLM

Testing Practices with ALM tools

Tune the value of MTLM, its capabilities and the processes it supports…

GOAL deep dive:

Know the value of MTLM and know how to use its capabilities …

GOAL general course:

Agenda Day 1 Start Topics

9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test

configuration 10:45 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM

Lot of discussions and hands-on lab’s

Environments setup TFS supported operating system Installed: • Team Foundation Server with Basic configuration • Visual Studio 2010 with SP1 • Microsoft Test Manager 2010 (comes with VS10 Ultimate)

You get an USB with: • Demo applications (we create a Team Project for them, see lab 1)

• Microsoft Test Agents.msi (we install this together)

• Hands-on lab documents • This deck, talk through and useful information

Lab 1

Create one team projects for the demo applications

Timebox: 15 minuts

People Introductions

Clemens Reijnen Sogeti Netherlands | Management Consultant http://www.ClemensReijnen.nl @ClemensReijnen

Start Topics



Application Lifecycle

Management

Understand the term ALM, know it’s value and Visual Studio’s ALM support

GOAL:

Suite of Application Lifecycle Management tools

http://msdn.microsoft.com/en-us/library/fda2bad5.aspx





application lifecycle management

http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx





work together

__with work packages __on alm artifacts

Testing Lifecycle

See also on www.tmap.net: - The Test Lifecycle Model - The 4 essentials

http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/index.jsp

http://eng.tmap.net/Home/TMap/The_4_essentials/index.jsp




http://msdn.microsoft.com/en-us/library/bb385832.aspx





Start Topics



Work items

Test Activities Planning…

Testing Lifecycle

See also on www.tmap.net: Planning Phase

http://www.tmap.net/

http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/Planning_Phase.jsp







VS2010 Process Support with Process Templates M

ult

iple

pro

cess

tem

pla

te

Artifact types

Work items

Reports

Workbooks

Dashboards

Metrics

User Story Work Item Type

Work Item Type Flow

User Story

Testing processes in the Lifecycle

Testing processes in the Lifecycle

Process template with work items, flows, links, queries and more

support | drive | enforce a way of work

Lab 2

Create user stories (2 or 3) and development and test tasks (optional) for the employee application.

Timebox: 5 minuts

Workitem Queries

and Reports

Queries

create your own

Create your own – advanced –

Query Types

Default Reports

Excel Reports

create your own

Excel basic

create your own

Report Wizard Excel Pivot


See also on www.tmap.net: Planning Phase




Microsoft Test Manager

Microsoft Test Manager 2010 The UX

Start Topics



Test Case Management

See also on www.tmap.net: Control Phase


http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/Control_Phase.jsp



Test Plans Test Suites Test Configurations

Organize the testing effort in a Test Execution Plan Create dynamic test suites for changing selection criteria Define various test configurations and share automation across those

Test (Execution) Plans

Test (Execution) Plans

Test Plan Organization Naming of test plans for agile environments: Iteration number and name and one extra test plan for end to end tests.

Test Configurations

Test Configurations

Test Suites

Test Suites Organization Naming of test suites for agile environments: Suite per user story (requirement based) and one additional test suite for regression tests.(query based?) (test cases are copied to the regression test suite)

Proposed organization

Iteration 1 Create test plan for iteration and a ‘end to end’ test plan. Create requirement based test suites which hold test cases for that suite.

Iteration 1 Create two additional suites, one for regression sets, which will be taken along during the project and one specific for automation.

Copy Test Cases

ID 21 ID 21

ID 22

Copy a Test Case

ID 21

Copy the reference to a Test Case

End iteration 1, some important test case are promoted to the ‘end to end ’ test plan.

Iteration 2, test suites are created for the requirements and the regression test suit is copied (link or hard?)

During the execution of iteration 2 the regression test suit grows and an automation suit is added.

Again important ones are promoted to the ‘end to end’ test plan

TCM.exe

TCM plan /list

TCM suites /lists planid:16

Lab 3

Create Test Plan, Test Configurations and make a test suite organization. Optional: use TCM.exe to list the lab results

Timebox: 10 minuts

Share MTM Artifacts

mtm://clemenes-pc:8080/tfs/defaultcollection/p:MSF/Testing/testplan/connect?id=19

mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67

Open Test results Open Test Cases (or any work item) Open Test Runs Open Test Plans Connect to test plans

mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center group>/<group specific>

http://blogs.msdn.com/b/vstsqualitytools/archive/2010/02/09/sharing-content-in-microsoft-test-manager.aspx












Start Topics


configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM

BREAK TILL:

11:00

(extra time to finish the labs :)

Start Topics



Test Case Specification

See also on www.tmap.net: Specification Phase


http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/Specification_Phase.jsp




Test Case Shared Steps Test Data Iterations

Link test cases and requirements to provide requirement traceability Data drive test cases with different parameters Refactor common shared steps in test cases

Test Case

Test Case Naming of test case: Purpose: • Finding the right test case • Common understanding • Intention from the title • Scan test plans more effectively and efficiently

Proposed solution #1 make use of linked work items #2 maximal length of the title 20 characters #3 format <functionality identifier> <action> <additional information> #4 make useful queries

Test Case from Excel

Test step types

http://www.clemensreijnen.nl/image.axd?picture=Windows-Live-Writer/Microsoft-Test-Manager-Test-case-specifi/5BABF6E6/image.png

Test steps Make them descriptive Not every step is a validation step Use the step attachment

Test steps copy-past

Test Data Iterations Data drive test cases with different parameters

Test Data Iterations

Test data copy-past

Shared Steps Refactor common shared steps in test cases Reduce test maintenance by sharing test steps across test cases

Shared Steps Shared Step candidates?

• REUSE – Many steps, reused for many test cases . Benefit: time, retyping, boring work.

• EXPECTED CHANGES – for example environment variables change [URL]. Benefit: Time, test cases

which need this step are updated automatically, also action recording.

• PRE- CONDITIONS – bring the system under test to a certain state and execute the different

verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case

• BACK TO INITIAL STATE - after test execution bring the environment back to the initial state for

new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test cases.

Shared Steps what about parameters and expected result in shared steps?

Lab 4

1. Specify multiple test cases. 2. Add some test data iterations. 3. Create shared step and reuse it.

Optional: play with the parameters and expected result steps in the shared steps…

Timebox: 15 minuts

Start Topics



Test Execution

Test Runner

A test that you will want to run multiple times. Different manual tests that contain common steps. Verifying a bug is fixed by using a manual test.

Test Case Execution

See also on www.tmap.net: Execution Phase


http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/Execution_Phase.jsp




Test Case Execution

Action recording

TCM run /create

Test Case Execution Results

View Test Results Using Microsoft Test Manager 1: from the ‘Run Tests’ 2: from the ‘Analyze Test Runs’

From Analyze Test Runs Test Results for multiple Test Cases

From Run Tests Test Result for a Test Case

Test Plan

From TCM run /export id:239 /resultsfile:”c:\runresult”

Share with mtm://

mtm://clemens-pc:8080/tfs/defaultcollection/p:MSF/Testing/testrun/open?id=292

Lab 5

Execute the created test cases, view the results Optional: run them from TCM.exe

Timebox: 10 minuts

Test queries

Test reports

Excel Reports for Tests

• POINT COUNT TREND - I want to see the results for test in a particular test run

• RESULT COUNT - I want to see trend report for my tests regardless of build or plan

• RESULT COUNT TREND - I want to see trend report for my tests regardless of build or plan

• TEST CASE COUNT - I want to see the number of test cases that have been planned for automation

Microsoft Test Manager Test Artifact organization

http://www.clemensreijnen.nl/image.axd?picture=WindowsLiveWriter/MicrosoftTestManagerTestArtifactorganiza/6F9D5EBE/testartifactorg.png

Test Plans and TFS items

Server

TCM Server Plan “Iteration 2” Suite US1 Suite US2

Test Case A Test Case B Test Case C Test Case D

MTLM Client

Plan “Iter 1”

Plan “Iter 3”

TFS

User Story 2

Test Case A

Test Case B

Test Case C

Task X

Task Y

Task Z Test Case C

Lab 6

Create excel pivot table to list test points per Tester.

Timebox: 10 minuts

Start Topics


configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM

LUNCH TILL:

12:45


Start Topics



Controlling the testing effort BUG’s

File a BUG

Diagnostic Data Adapters File a Bug File an Exploratory Bug Verify a Bug

Diagnostic Data Adapters

File a Bug Submit a Bug using Test Runner

Ways to file a Bug

Submit a Bug from the Test Runner Submit a Bug from the Verify Bugs View Submit a Bug from Test Results for a Test

Verify a Bug Verify a Bug Fix Using Microsoft Test Manager

The Bug Flow

T

E

S

T

D

E

V

Link build 2 test plan. Configure Data collectors.

Setup build and deploy

Build definition is

the intergration

Execute Manual Tests

File Bug

Resolve Bug

Checkin Build Deploy

Verify fix by using playback

Close or reopen Bug

Lab 7

Create a build File a bug Resolve a bug Verify a bug

Timebox: 15 minuts

Bug Reports

Bug Status Report Bug Trends Report Reactivations Report

Bug Status Report

Bug Trends Report

Reactivations Report

Start Topics



Test Impact

T

E

S

T

Execute ‘passed’ Tests


Test Impact

D

E

V

Assign build 2 test plan. Configure Data collectors.

Setup build and deploy

Build definition is

the intergration


Make code changes

Checkin Build Deploy

Assign ‘NEW’ build 2 Test Plan

Evaluated and reset tests impacted by code churn

Lab 8

Attach build to test plan Execute tests Make code change Kickoff build Attach new build to test plan Evaluate impacted tests Timebox: 15 minuts

E2E Scenario

Iteration N

T

E

S

T

Sprint plan

Write tests for US1

Write tests for US2

D

E

V

Sprint plan

Build 1 Build 2 Build 3

Implement User Story ‘1’

(US1)

Test US1 & file bugs

Implement US2

Build 4

Fix bugs

Build 6 Build 7

Regress impacted

tests

Build 5

Implement US2

Fix bugs


Verify fixes

Verify fixes

T

E

S

T

Sprint plan

Write tests for US1

Write tests for US2

D

E

V

Sprint plan



(US1)


Build 4

Fix bugs

Build 6 Build 7

Regress impacted

tests

Build 5

Implement US2

Fix bugs


Verify fixes

Verify fixes

Sprint plan

Write tests for US1

Sprint plan



(US1)


Sprint plan

Write tests for US1

Key Value: Requirements Planning

Build 3


(US1)


User Stories and corresponding done dev tasks. Create test tasks and assign to someone else.

T

E

S

T

Sprint plan

Write tests for US1

Write tests for US2

D

E

V

Sprint plan



(US1)


Build 4

Fix bugs

Build 6 Build 7

Regress impacted

tests

Build 5

Implement US2

Fix bugs


Verify fixes

Verify fixes

Sprint plan

Write tests for US1

Sprint plan



(US1)


Sprint plan

Write tests for US1


Build 3


(US1)


Execute the Test Specification Tasks and other test tasks

T

E

S

T

Sprint plan

Write tests for US1

Write tests for US2

D

E

V

Sprint plan



(US1)


Build 4

Fix bugs

Build 6 Build 7

Regress impacted

tests

Build 5

Implement US2

Fix bugs


Verify fixes

Verify fixes

Sprint plan

Write tests for US1

Sprint plan



(US1)


Sprint plan

Write tests for US1


Build 3


(US1)


Execute the Test Execution Tasks and file bugs (if any)

T

E

S

T

Sprint plan

Write tests for US1

Write tests for US2

D

E

V

Sprint plan



(US1)


Build 4

Fix bugs

Build 6 Build 7

Regress impacted

tests

Build 5

Implement US2

Fix bugs


Verify fixes

Verify fixes


Fix bugs

Verify fixes

Build 3 Build 5

Fix bugs


Build 5

Verify fixes

Key Value: Virtuous Bug Cycle Dev solves the bugs. Test Verifies tests

T

E

S

T

Sprint plan

Write tests for US1

Write tests for US2

D

E

V

Sprint plan



(US1)


Build 4

Fix bugs

Build 6 Build 7

Regress impacted

tests

Build 5

Implement US2

Fix bugs


Verify fixes

Verify fixes

Key Value: Virtuous Bug Cycle Check Results

Start Topics



BREAK TILL:

14:30


Start Topics



Create test from Bug

File an Exploratory Bug Submit an Exploratory Bug using Test Runner

Record scenario by business user with MTM on application under test. Action recording is captured.

1

Create test steps (with parameters) in MTM by using the action recording.

2

Save test case and execute it by a tester on application under test, by using MTM.

3

CON: - Manual actions and tests. - Test steps text are technical names - Action recording (step 1) must be clean. - Unknown what is validated, during step 2, need

additional knowledge. - The business user must think of the important

scenarios The action recording during step 1 can have unnecessary steps, these can be removed during the execution (1) or during the specification (2).

PRO: - Reuse scenario knowledge business user - No typing detailed test steps - Easy to parameterize - Writing standards for test steps - MTM capabilities in step 3. (bug, codedUI,

recommended test etc …) Most benefit during step 2 (test case writing), benefit in step 1: run once in common situation. Benefit in step 3: common MTM benefits (record playback, fast forward, parameters etc) and benefit of well written steps.

Lab 9

Timebox: 10 minuts

Create an exploratory bug and create a test case from it

Start Topics



Test Infrastructure

Where and how to run the automated test?

Test Environments

See also on www.TMap.net: - Checklist "Intake Test Environment“ - Checklist "Test Environment“ - Infrastructure Phase


http://eng.tmap.net/Images/Checklist Intake Test environment TMap Next v1.0_tcm9-47157.doc



http://eng.tmap.net/Images/Checklist Test environment TMap Next v1.0_tcm9-47160.doc



http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/Infrastructure_Phase.jsp

Test lab Visual Studio 2010 Lab Management

physical test environments virtualized test environments

Where and how to run the automated test?

TFS 2010

Physical Test Client

MTLM Client

Physical Test Client

Hyper-V Host

VM

System Center Virtual Machine Manager (SCVMM)

TFS 2010

VM

Library Shares

share

share

MTLM Client

physical test environments

virtualized test environments

Build application

Provision ‘clean’ env

Deploy application

Run tests

physical test environments versus virtualized test environments

Start Topics



Visual Studio

Test Manager

Team Foundation Server

Build Server

Test Agent

Test Agent

Test Agent

Lab for Physical Environments, the different flavors…

A

Flavor A: Execution from VS2010…

Configurations for automated test execution:

Real developer test / test automation dry run (Triggered by: right mouse click – run tests, in test project) http://msdn.microsoft.com/en-us/library/dd286580.aspx No additional configuration needed* * Data Diagnostic Settings in the Local.Testsettings file are configurable.

Pro: Easy setup

Debug-able test automation

Con: No collection of test results in TFS Run on developer environment

http://msdn.microsoft.com/en-us/library/dd286580.aspx





B

Flavor B: Execution during build with Build controller…


Part of Build Verification Tests (triggered by: build) Set the Build Service to run in interactive mode http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType * Data Diagnostic Settings in the *.Testsettings file are configurable.

Pro: Easy setup

Test results in build result

Con: No collection of test results in TFS Build Controller needs to run in interactive mode Tests are executed on build environment Run on build environment

not recommended, strange to run UI tests on build



http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx














http://msdn.microsoft.com/en-us/library/ms182465.aspx



C

Flavor C: Execution during build with Test controller…


Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B)

Pro: Test run on test environments

Tests run on multiple environments Test Results in Build result

Con: No collection of test results in TFS Harder to configure Need for specific test client environments Test Settings from VS

Preferred configuration above flavor B











D

Flavor D: Execution from Microsoft Test Manager…


Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html


Tests run on multiple environments Test Result in MTM and TFS

Test Settings from MTM

Con: Manually triggered by Tester Hard to configure Need for specific test client (same as MTM?)

Other type of test than BVT



http://msdn.microsoft.com/en-us/library/ee390842.aspx







http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html


















E

Flavor E: Execution from MTM during Build…


Part of BVT (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx


Tests run on multiple environments Test Result in MTM and TFS

Triggered by build Test Settings from MTM

Con: Hard to configure Need for specific test client (same as MTM?)

Preferred configuration above flavor C Flavor D and E can be configured together



http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx



















Test Controller

Test Agent

Lab 10

Install and configure the test controller and test agent

Timebox: 20 minuts

http://msdn.microsoft.com/en-us/library/dd648127.aspx http://www.microsoft.com/downloads/en/details.aspx?FamilyID=a3216d2a-0530-4f6c-a7c9-0df37c54a902





http://www.microsoft.com/downloads/en/details.aspx?FamilyID=a3216d2a-0530-4f6c-a7c9-0df37c54a902










Start Topics



New physical environment


RECAP

Test Case Management RECAP

Test Case Specification

RECAP

Test Case Execution

RECAP

Test Case Management RECAP

Start Topics

9:00 Recap

9:15 Create Basic CodedUI, from Test Case and Manual

9:45 UIMap and customize the code for optimization

10:15 Data driven Tests and assertions

10:45 break

11:00 Troubleshoot CodedUI, Common Practices and questions

11:30 Configure the build to execute the CodedUI tests, test settings from VS2010, Execute from VS2010 with test agents and analyze the results

11:30 Associate test automation with test case, execute automated test case from MTM, configure the build to run from the build within MTM and analyze the results

12:30 break

13:15 MTLM, Scrum and other project Methodologies

13:45 Lab Management - Virtualized Test Environment and execution Demo

14:15 Test info in the OLAP Cube

14:30 Extensibility - test Scribe, TCMIMport

14:45 Create your own Data Diagnostic Adapter

15:15 MTLM usages with Azure projects

16:00 Closing

DAY 2

Technology

MTLM Visual Studio 2010 ALM workshop - day1