Technical Meeting Automated Testing with

VS2010

Suite of Application Lifecycle Management tools

http://msdn.microsoft.com/en-us/library/fda2bad5.aspx

application lifecycle management

http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx

work together

__with work packages __on alm artifacts

Testing Lifecycle

See also on www.tmap.net: - The Test Lifecycle Model - The 4 essentials

http://msdn.microsoft.com/en-us/library/bb385832.aspx

Microsoft Test Manager 2010

Test Case Management

See also on www.tmap.net: Control Phase

Test Plans Test Suites Test Configurations

Organize the testing effort in a Test Execution Plan Create dynamic test suites for changing selection criteria Define various test configurations and share automation across those

Test (Execution) Plans

Test (Execution) Plans

Test Plan Organization Naming of test plans for agile environments: Iteration number and name and one extra test plan for end to end tests.

Test Configurations

Test Configurations

Test Suites

Test Suites Organization Naming of test suites for agile environments: Suite per user story (requirement based) and one additional test suite for regression tests.(query based?) (test cases are copied to the regression test suite)

Proposed organization

TCM.exe

TCM plan /list

TCM suites /lists planid:16

Share MTM Artifacts

mtm://clemenes-pc:8080/tfs/defaultcollection/p:MSF/Testing/testplan/connect?id=19

mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67

Open Test results Open Test Cases (or any work item) Open Test Runs Open Test Plans Connect to test plans

mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center group>/<group specific>

http://blogs.msdn.com/b/vstsqualitytools/archive/2010/02/09/sharing-content-in-microsoft-test-manager.aspx

Test Case Specification

See also on www.tmap.net: Specification Phase

Test Case Shared Steps Test Data Iterations

Link test cases and requirements to provide requirement traceability Data drive test cases with different parameters Refactor common shared steps in test cases

Test Case

Test step types

Test steps Make them descriptive Not every step is a validation step Use the step attachment

Test steps copy-past

Test Data Iterations Data drive test cases with different parameters

Test Data Iterations

Test data copy-past

Shared Steps Refactor common shared steps in test cases Reduce test maintenance by sharing test steps across test cases

Shared Steps Shared Step candidates?

• REUSE – Many steps, reused for many test cases . Benefit: time, retyping, boring work.

• EXPECTED CHANGES – for example environment variables change [URL]. Benefit: Time, test cases

which need this step are updated automatically, also action recording.

• PRE- CONDITIONS – bring the system under test to a certain state and execute the different

verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case

• BACK TO INITIAL STATE - after test execution bring the environment back to the initial state for

new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test cases.

Test Execution

Test Runner

A test that you will want to run multiple times. Different manual tests that contain common steps. Verifying a bug is fixed by using a manual test.

Test Case Execution

See also on www.tmap.net: Execution Phase

Test Case Execution

Action recording

TCM run /create

From TCM run /export id:239 /resultsfile:”c:\runresult”

Share with mtm://

mtm://clemens-pc:8080/tfs/defaultcollection/p:MSF/Testing/testrun/open?id=292

Controlling the testing effort BUG’s

File a BUG

Diagnostic Data Adapters File a Bug File an Exploratory Bug Verify a Bug

Diagnostic Data Adapters

File a Bug Submit a Bug using Test Runner

Ways to file a Bug

Submit a Bug from the Test Runner Submit a Bug from the Verify Bugs View Submit a Bug from Test Results for a Test

Verify a Bug Verify a Bug Fix Using Microsoft Test Manager

The Bug Flow

T

E

S

T

D

E

V

Link build 2 test plan. Configure Data collectors.

Setup build and deploy

Build definition is

the intergration

Execute Manual Tests

File Bug

Resolve Bug

Checkin Build Deploy

Verify fix by using playback

Close or reopen Bug

Test Impact

T

E

S

T

Execute ‘passed’ Tests

Execute ‘passed’ Tests

Test Impact

D

E

V

Assign build 2 test plan. Configure Data collectors.

Setup build and deploy

Build definition is

the intergration

Execute ‘passed’ Tests

Make code changes

Checkin Build Deploy

Assign ‘NEW’ build 2 Test Plan

Evaluated and reset tests impacted by code churn

Create test from Bug

File an Exploratory Bug Submit an Exploratory Bug using Test Runner

Record scenario by business user with MTM on application under test. Action recording is captured.

1

Create test steps (with parameters) in MTM by using the action recording.

2

Save test case and execute it by a tester on application under test, by using MTM.

3

CON: - Manual actions and tests. - Test steps text are technical names - Action recording (step 1) must be clean. - Unknown what is validated, during step 2, need

additional knowledge. - The business user must think of the important

scenarios The action recording during step 1 can have unnecessary steps, these can be removed during the execution (1) or during the specification (2).

PRO: - Reuse scenario knowledge business user - No typing detailed test steps - Easy to parameterize - Writing standards for test steps - MTM capabilities in step 3. (bug, codedUI,

recommended test etc …) Most benefit during step 2 (test case writing), benefit in step 1: run once in common situation. Benefit in step 3: common MTM benefits (record playback, fast forward, parameters etc) and benefit of well written steps.

Test Activities Planning…

RECAP

Test Case Management RECAP

Test Case Specification

RECAP

Test Case Execution

RECAP

Test Case Management RECAP

Intellitrace

F5

1 Open the employee application Set the intellitrace options Set a breakpoint at …

2 When on the breakpoint, save the trace file on your desktop

3 Close Visual Studio and double click the intellitrace file

4 In the intellitrace summary, open the main threat …

5 … and search for The values that where entered during the previous run.

6 And now from Microsoft Test Manager

6 Set the diagnostic adapter

7 Configure the build… it needs a drop location for the symbols (pdb files)

8 Run the test case and see it crash… file a bug.

9 See the itrace file in the bug report...

10

Close MTM, open VS (with or without the employee application). Got to the bug and click the itrace file… See the exception and start debugging…

11

Go back in time

11

Go back in time … and find the entered values

Regression Test Candidates

See also on www.tmap.net: Completion Phase

Test Automation

Test automation Where? When? How? Who? Why? What?

met Visual Studio 2010

Where? When? How? Who? Why? What?

Where? When? How? Who? Why? What?

Where? When? How? Who? Why? What?

VS2010 Test Tools UI Test Automation Capabilities

Fast Forward CodedUI

Test Infrastructure

1

2

3

No investment in any kind of automation. Just click and test.

Clean up of the Action log maybe re-execute, for more smooth Fast Forward.

Basic CodedUI. Only use the default generation, add your own assertion.

Advanced CodedUI. Customize the CodedUI and UIMap for optimization.

0

Test Automation Investment Levels

1 Clean up of the Action

2

VS2010 ALM MTLM usages patterns for Windows Azure hosted application development

Basic CodedUI

3 Advanced CodedUI

Fast Forward

Cleaning the action recording When: executing the same test case on multiple environments or when using test data iteration or … and don’t forget the shared steps

CodedUI

CodedUI from existing action recording from Manual Tester

CodedUI from existing action recording from Manual Tester

CodedUI from existing action recording from Manual Tester

1

3

4

2

CodedUI Create a new action recording by Developer

UIMap

UIMap

Data-Driven Coded UI Test

Test Infrastructure

Where and how to run the automated test?

Test Environments

See also on www.TMap.net: - Checklist "Intake Test Environment“ - Checklist "Test Environment“ - Infrastructure Phase

Test lab Visual Studio 2010 Lab Management

physical test environments virtualized test environments

Where and how to run the automated test?

TFS 2010

Physical Test Client

MTLM Client

Physical Test Client

Hyper-V Host

VM

System Center Virtual Machine Manager (SCVMM)

TFS 2010

VM

Library Shares

share

share

MTLM Client

physical test environments

virtualized test environments

Build application

Provision ‘clean’ env

Deploy application

Run tests

physical test environments versus virtualized test environments

Visual Studio

Test Manager

Team Foundation Server

Build Server

Test Agent

Test Agent

Test Agent

Lab for Physical Environments, the different flavors…

A

Flavor A: Execution from VS2010…

Configurations for automated test execution:

Real developer test / test automation dry run (Triggered by: right mouse click – run tests, in test project) http://msdn.microsoft.com/en-us/library/dd286580.aspx No additional configuration needed* * Data Diagnostic Settings in the Local.Testsettings file are configurable.

Pro: Easy setup

Debug-able test automation

Con: No collection of test results in TFS Run on developer environment

B

Flavor B: Execution during build with Build controller…

Configurations for automated test execution:

Part of Build Verification Tests (triggered by: build) Set the Build Service to run in interactive mode http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType * Data Diagnostic Settings in the *.Testsettings file are configurable.

Pro: Easy setup

Test results in build result

Con: No collection of test results in TFS Build Controller needs to run in interactive mode Tests are executed on build environment Run on build environment

not recommended, strange to run UI tests on build

C

Flavor C: Execution during build with Test controller…

Configurations for automated test execution:

Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B)

Pro: Test run on test environments

Tests run on multiple environments Test Results in Build result

Con: No collection of test results in TFS Harder to configure Need for specific test client environments Test Settings from VS

Preferred configuration above flavor B

D

Flavor D: Execution from Microsoft Test Manager…

Configurations for automated test execution:

Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html

Pro: Test run on test environments

Tests run on multiple environments Test Result in MTM and TFS

Test Settings from MTM

Con: Manually triggered by Tester Hard to configure Need for specific test client (same as MTM?)

Other type of test than BVT

E

Flavor E: Execution from MTM during Build…

Configurations for automated test execution:

Part of BVT (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx

Pro: Test run on test environments

Tests run on multiple environments Test Result in MTM and TFS

Triggered by build Test Settings from MTM

Con: Hard to configure Need for specific test client (same as MTM?)

Preferred configuration above flavor C Flavor D and E can be configured together

Test Controller

Test Agent

New physical environment

Execute CodedUI During the build with Visual Studio 2010

C

Flavor C: Execution during build with Test controller…

Configurations for automated test execution:

Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B)

Pro: Test run on test environments

Tests run on multiple environments Test Results in Build result

Con: No collection of test results in TFS Harder to configure Need for specific test client environments Test Settings from VS

Preferred configuration above flavor B

1 Configure Test Controller.

2 Add new Test Settings file Set remote execution and test controller.

3 Create a new build definition Set the Automated Test Properties

4 Queue a new Build Watch the Test Agent and Test Environment

Execute CodedUI Within a test case as an associated automation manualy

D

Flavor D: Execution from Microsoft Test Manager…

Configurations for automated test execution:

Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html

Pro: Test run on test environments

Tests run on multiple environments Test Result in MTM and TFS

Test Settings from MTM

Con: Manually triggered by Tester Hard to configure Need for specific test client (same as MTM?)

Other type of test than BVT

1 Configure Test Controller.

2 Associate the CodedUI with a Test Case.

2 Create Test Environment

3 Set test Settings

4 Associate the CodedUI with a Test Case.

5 Run Automated Tests from Test Manager

Execute CodedUI Within a test case as an associated automation from the build

E

Flavor E: Execution from MTM during Build…

Configurations for automated test execution:

Part of BVT (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run MTC or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx

Pro: Test run on test environments

Tests run on multiple environments Test Result in MTM and TFS

Triggered by build Test Settings from MTM

Con: Hard to configure Need for specific test client (same as MTM?)

Preferred configuration above flavor C Flavor D and E can be configured together

Execute automated test cases on different environments

0 Same steps as previous.

TFS 2010

Physical Test Client

MTLM Client

Physical Test Client

Win 7 IE8 configuration

Win 7 IE9 configuration

1 Configure additional environments in Test Plan properties.

Win 7 IE8 configuration

Win 7 IE9 configuration

TC

1 Get Test Plan ID Get Test Suite Get Test Configurations with TCM.exe

2 Create command line for Test Run for the different Test Configurations

"run /create /title:""Run on Host for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=13"" /settingsname:""Automated Test on Windows 7 Environment IE9"" /testenvironment:""Win 7 IE9"" /collection:http://clemenes-pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"

"run /create /title:""Run on test env for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=12"" /settingsname:""Automated Test on Windows 7 Environment IE8"" /testenvironment:""Win 7 IE8"" /collection:http://clemenes-pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"

3 Add build commands

6 Queue a new Build Watch the Test Agent and Test Environment

Build application

Provision ‘clean’ env

Deploy application

Run tests

• Reproduce the exact conditions of a bug or other development issue.

• Build, deploy, and test applications automatically in a clean environment.

• Reduce the time required to create and configure machines for testing an application.

• Run multiple copies of a test or development at the same time.

• Enable members of a team to create and manage virtual environments without requiring system administrator privileges.

VS2010 Lab Management

System Center Virtual Machine Manager (SCVMM)

TFS 2010

Hyper-V Host

VM

VM

Library Shares

share

share

MTLM Client

VM

Lab Agent

TFS 2010

staat voor resultaat