04b | Manage Test Execution (2 of 2)
Steven Borg | Co-founder & Strategist, Northwest CadenceAnthony Borton | ALM Consultant, Enhance ALM
• Run Tests
• Perform Exploratory Testing
• Manage Bugs
• Use Lab Center
• Analyze Recommended Tests
• Perform Analysis
• Manage Work Items
Module Overview
Click to edit Master subtitle style
Microsoft Virtual Academy
Manage Bugs
What the Study Guide says…
• Manage bugs– tracking bug metrics (bug trends, status)– verifying bugs (create Test from bugs)– analyzing bug reports–managing bug workflow
DEMO
Microsoft Virtual Academy
Manage Bugs
Click to edit Master subtitle style
Microsoft Virtual Academy
Use Lab Center
What the Study Guide says…
• Use Lab Center– creating new environments– creating copies of environments– running Tests on remote environments– executing Test Case in a Lab Center Environment
Lab management automation
Lab management services
Environment templates Environment provisioning Environment snapshots
Development and Testing
Environments Development environments
Repro environmentsTest environments
Continuous Build-Deploy-TestModern applications need to be deployed and tested in multiple environments.
Whether you store your test environments on bare metal or in the cloud, Lab Manager enables automation and increases collaboration, reducing the build-deploy-test cycle time.
Create a new environment using virtual machine templates in
SCVMM
Create an environment using machines running on
SCVMM
Create new environments on existing machines
Automation: engageWhen developers and testers collaborate, quality wins.
Developers build the application, deploy it to an environment and run automated test cases in a single step, preparing the way for testers.
Quickly choose multiple test suites to run after deployment
Choose your environment, and build, deploy and test in that environment
from a single dialog
Manage EnvironmentsTesters spin up environments to run manual tests, and test agents record all the data required to recreate any bugs.
Snapshot environments and attach to bug reports so developers can easily reproduce in the exact environment where the defect was found.
Snapshot environments during manual testing to increase
reproducibility
Spin up virtual test environments from within Test Manager, or attach a virtual
environment to a manual testing session
Intelligent analysisEnable data collectors and generate detailed test results to ensure any bugs that are found are actionable.
Team Foundation Server keeps track of all the pieces, keeping your entire team on the same page and reducing time to repair.
Test run results are attached to specific builds and environments
Collect all the data required to recreate bugs from virtual
environments
DEMO
Microsoft Virtual Academy
Use Lab Center
Click to edit Master subtitle style
Microsoft Virtual Academy
Analyze Recommended Tests
What the Study Guide says…
• Analyze Recommended Tests– selecting the build in use– comparing the current build to a previous build– viewing Recommended Tests– analyzing related Work Items
DEMO
Microsoft Virtual Academy
Analyze Recommended Tests
Click to edit Master subtitle style
Microsoft Virtual Academy
Perform Analysis
What the Study Guide says…
• Perform analysis– analyzing reports (Requirements-User Stories)– analyzing by Test Suite– analyzing by configuration– identifying areas where quality is low– identifying Test Plan status
Software testing with Visual Studio
Team Foundation Server
Visual Studio IDEMicrosoft Test Manager
Unit testing UI automation testing
Performance testing Load testingCode quality
analysisTest planning and
managementTest case
management Lab manager Manual testing
Exploratory testing
Microsoft Feedback Client
User testing
Work item tracking Source control Lab management services Test execution services Build services Reporting and BI
Pluggable components
Test runners Build engines Lab environmentsDiagnostic data collectors
Monitoring Quality Metrics
Users Solution managers
Developers Testers Operations Stakeholders
• Actionable feedback • Exploratory testing
• Requirements management
• Acceptance criteria definition
• Developer testing• Build automation• Actionable
diagnostics
• Test planning and management
• Test execution and defect reporting
• Lab management
• Production monitoring• Integrated incident
managementQuality metrics and
reporting
Role tailored tools feed data to transparent quality reports
Data Warehouse
Stories Overview
Test Failure Analysis
Bug Reactivation Bug Trends Custom
SSRS/SSIS Pre-built and fully customizable cross project reporting
Visualize qualityThe build report is a barometer of quality and a harbinger of success or failure.
As the development cycle nears the end, defects and failed builds should trend towards zero. Build quality reports provide teams clear insight into the development process and can indicate where teams need to focus their efforts.
Filters allow you to control which builds you see
View the test and code coverage results for each build to determine
quality
Right out of the gateWaiting until the end of a cycle to address defects can adversely impact schedule and resources.
Desired ATDD behavior should see a high spike in failing acceptance tests early in the cycle with the spike tapering down as the cycle progresses indicating that tests are being run early and uncovering defects early on.
Bug bashingDevelopment teams can use defect tracking reports to estimate levels of deliverable code quality and to provide insight into a team’s ability to address defects.
A defect should ideally be detected and solved once, with a corresponding test implemented to ensure the fix addresses the desired behavior. Bug reactivations should remain close to zero.
Filling in the gapsThe only thing constant is change and code that changes must be tested.
To ensure your code base is fully covered by test leverage the code coverage report and identify your testing gaps. Establishing a code base fully covered by tests ensures new changes will not break your code!
Line-by-line highlighting illustrates which code is covered, uncovered, or
partially covered by test
Results show which methods are covered by test and total coverage is
indicated by a percentage
I can see clearly nowA picture is worth a thousand words and a Stories Overview report gives that big picture!
The Stories Overview gives a comprehensive view of each implemented user story including completion status, test results and bug status. A thousand words, indeed!
Track total percentage hours completed by user story
View real-time status of each user story
Test results quickly display quality status of each user story
Track active and resolved bugs by user story
DEMO
Microsoft Virtual Academy
Perform Analysis
Click to edit Master subtitle style
Microsoft Virtual Academy
Manage Work Items
What the Study Guide says…
• Manage Work Items– validating requirements–Work Item relationships (eg, what it means when a test
case is associated with a requirement)– creating Work Item queries– performing bulk updates in Microsoft Excel
DEMO
Microsoft Virtual Academy
Manage Work Items
• Become very familiar with work items, work item queries, and how work items are related
• Understand the lifecycle of a bug and its associated test case
• Know what Excel reports are available for each process template
• Know what SSRS reports are available for each process template
• Understand the basics of the Lab Center tab, including environments and an overview of Lab Management
EXAM BEST BETS
©2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.