INC REA S E P ROD UC TIV IT Y A ND S OF TW A RE QUA L ITY W IT H A Z U RE A ND

V S 2 0 1 0 A LM

Fa st er , b e tt er w it h h ig h er qu a l i t y de s i g n, d eve lo p, b ui l d , t e st an d d e ploy Az u r e c lou d ap pl ica t io ns wi th Ap pl ica t ion L i f ecyc le Ma na g em en t .

AZURE PROVIDES NEW OPPERTUNITIES FOR BUSINESSES. MANY ORGANISATIONS ARE STARTING TO

DEVELOP CLOUD APPLICATIONS. MICROSOFT VISUAL STUDIO 2010 AND TEAM FOUNDATION SERVER

2010 ARE THE APPLICATION LIFECYCLE MANAGEMENT TOOLS TO DEVELOP AZURE APPLICATION.

THIS PAPER DISCUSSES, HOW APPLICATION LIFECYCLE MANAGEMENT PROCESSES AND TOOLS CAN BE

USED IN AN OPTIMAL WAY, BY MAKING USE OF THEY KEY CHARACTERISTICS OF THE AZURE PLATFORM

TO RAISE THE PRODUCTIVITY AND QUALITY OF SYSTEMS DEVELOPED FOR THE CLOUD.

ALM 4 Azure

ALM 4 Azure a presentation covering different levels of ALM automation for Azure cloud services development. An important notice: the topic covers modern software development tools and practices which you need to know while developing systems which run on the Azure cloud (4 Azure). This is something completely different, but related, than ALM on Azure, where we use the tools out of the cloud (SaaS) to execute a modern system development. You can use them together ‘on’ and ‘for’ Azure, but this presentation only covers the ‘for’ azure. How do we need to execute our practices and use our on premise tools for cloud services?

What is Application Lifecycle Management, why do we want it and what are the goals we pursue. What are the specific characteristics of Azure and Cloud Computing. Where can these characteristics help us to reach the goal and where do they give us some challenges. The main topic in the agenda are 5 different ALM 4 Azure scenario’s and supporting technologies explained.

Application Lifecycle Management

All tool vendors and methodologies have their own definition of Application Lifecycle management. What can we learn from these definitions? A wide variety in scope. ITIL is focused on the operational side of ALM, the Wiki and Forrester descriptions are more focused on the Software Development Lifecycle [SDLC] and Microsoft takes a bigger scope with business, development and operations, although the tooling and the assessment are focused on SDLC. Borland is also talking about a wider scope, when you look at the RUP like model. But the main pro is their focus on "many processes and many tools" so it should fit more then one environment. Beside this difference in scope, everybody agrees on terms like: measurable, predictable, traceable, manageable, monitored etc etc... smells like "in control" We use this image when defining ALM and it is about "in control" but it is even more about communication. It is about how the different ALM roles who are all responsible and accountable for the success of a software development project communicate in a seamless manner.

When talking about Application Lifecycle Management [ALM] terms like accountability, governance and compliance are used. All of them refer back to “working together”, how do we work together during the application lifecycle. ALM is not about tools, it’s about working together. Working together seamless and flexible while staying in control, being measurable and responsible. All the roles in the application lifecycle have a part in this collaboration effort. Tools can help but isn’t’ the core driver. There are lots of examples of wrong interpreted business requirements, miss communication between development and test, applications which won’t run in production, operations who don’t understand the applications. All of them resulting in more work, more faults, more costs or even only costs and no application because the project was unplugged. Most of these projects faults, these extra costs are a slip in communication. Having a strategy how people have to collaborate within the lifecycle is one important piece of Application Lifecycle Management. Most organizational parts have already some kind of process / methodology in place. Having an approach how to share information and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raises the business value of IT investments. Tools can help with this goal. Having gear in place which supports and stimulates collaboration is a driver for a successful Application Lifecycle Management. But without a plan, how people should collaborate and communicate, tools are useless.

Creating new modes of collaboration supported by technology can only be done by addressing the human aspect. More specifically, we need to address some of the worries and obstacles people encounter when collaborating using technology.

The three most important concerns are:

Trust. Trust is a condition for social interaction. People will only work

with people, companies, tools and information they know they can

trust. Before we can expect collaboration to take off online, there must

be a way for people to get this “trust.” And a topic closely associated

with trust when it refers to people is Identity.

Collaborative culture. If one individual is the greatest collaborator in

the world, he or she is probably not getting anywhere. Only when all

people involved are part of the same collaborative culture will new

levels of creativity and productivity be reached. A collaborative culture

consists of many things, including:

o Collaborative leadership;

o Shared goals;

o Shared model of the truth; and

o Rules or norms.

Reward. Changing the way people work takes effort, so it must be

clear for the parties involved what they will gain, at a personal level,

from collaborating in a new way. Surprisingly, a “reward” for successful

collaboration is most often of a non-financial nature.

When working together with work packages, seamless communication

need to address these challenges.

All the different roles in the Application Lifecycle create artifacts, product

also these products need to work together, they need to fit, a single point of

truth. Requirements, designs, the business case, the test cases, the source

files and the operational information… all need to work together as one

consistent product. When one gets out of sync then the involved roles

should get a notification. Tools can help with this.

The Visual Studio 2010 family is made up of a central team server, and a small selection of client-side tools. The team server—Team Foundation Server 2010—is the backbone of the application lifecycle management, providing capabilities for source control management, (SCM), build automation, work item tracking and reporting. In this release Microsoft expanded the capabilities of Team Foundation Server by adding a true test case management system and extended it with Lab Management 2010—a set of capabilities designed to better integrate both physical and virtual labs into the development process. On the client-side for developers, you can choose between Visual Studio 2010 Professional, Premium or Ultimate. For testers and business analysts there is Test Professional—a new integrated test environment designed with manual testers in mind. For those people who participate in the development efforts, but for whom Visual Studio—the IDE—is not appropriate, including Java developers, project managers and stakeholders the Team Foundation Server extensibility model enables alternative interfaces. These include both Team Explorer—a standalone tool built with the Visual Studio shell—and Team Web Access. These tools enable anyone to work directly with Team Foundation Server. And there is cross-product integration capabilities with Microsoft Office® and Microsoft Expression and SharePoint Server with new SharePoint dashboard.

Azure

Windows Azure™ is a cloud services operating system that serves as the development, service hosting and service management environment for the Windows Azure platform. Windows Azure provides developers with on-demand compute and storage to host, scale, and manage web applications on the internet through Microsoft® datacenters. Windows Azure has several unique characteristics as a platform. 1. Hosted services allow deploying to two identical but independent environments. The stating environment and the so called production environment. When you deploy a service you can choose to deploy to either the staging environment or the production environment. A service deployed to the staging environment is assigned a URL with the following format: {deploymentid}.cloudapp.net. A service deployed to the production environment is assigned a URL with the following format: {hostedservicename}.cloudapp.net. The staging environment is useful as a test bed for your service prior to going live with it. In addition, when you are ready to go live, it is faster to swap VIPS to move your service to the production environment than to deploy it directly there. http://msdn.microsoft.com/en-us/library/gg433118.aspx 2. Guest OS Versions are identical for every instance. In the configuration of the hosted service the OS version is set and every instance will be built of this same image. This results in the very hard to accomplish on premise situation where, test, acceptance and production environments are identical. The Windows Azure guest operating system is the operating system that

runs on the virtual machines (VMs) that host your service. The guest operating system is updated monthly. You can choose to upgrade the guest OS for your service automatically each time an update is released, or you can perform upgrades manually at a time of your choosing. All role instances defined by your service will run on the guest operating system version that you specify. http://msdn.microsoft.com/en-us/library/ff729422.aspx 3. Don’t assume your state is safe. Instances, VM’s are recycled on a ‘for us’ random base. Windows Azure manages that the application is always accessible, but local stored information isn’t. 4. in place upgrades. Windows Azure role instances can be easily upgraded. Windows Azure organizes instances of your roles into virtual groupings called upgrade domains. When you upgrade one or more roles within your service in-place, Windows Azure upgrades sets of role instances according to the upgrade domain to which they belong. Windows Azure upgrades one domain at a time, stopping the instances running within the upgrade domain, upgrading them, bringing them back online, and moving on to the next domain. By stopping only the instances running within the current upgrade domain, Windows Azure ensures that an upgrade takes place with the least possible impact to the running service. http://msdn.microsoft.com/en-us/library/ee517255.aspx 5. clear environment costs.

Azure applications are developed local. It is also possible to run the Azure application in this local environment by using emulators. The Windows Azure compute emulator enables you to run, test, debug, and fine-tune your application before you deploy it as a hosted service to Windows Azure. http://msdn.microsoft.com/en-us/library/gg432968.aspx The Windows Azure storage emulator provides local instances of the Blob, Queue, and Table services that are available in the Windows Azure. If you are building an application that uses storage services, you can test locally by using the storage emulator. http://msdn.microsoft.com/en-us/library/gg432983.aspx When deploying the Azure application to the Azure platform a package and a configuration file needs to be created. The package contains all files and the configuration information about the guest OS it needs and other configuration information.

Environment configuration. To start developing Azure Applications and to run the application local or deploy and run it on the Azure platform there are some needs.

1. a version of Visual Studio 2010 2. the Azure SDK

http://www.microsoft.com/windowsazure/getstarted 3. Set up a Windows Azure Subscription

For the demo’s… you need more (for a windows 7 environment)

- TFS Basic - Microsoft Test and Lab Manager - Build Service, controller and agent - Test controller and test agent - Visual Studio SP1 - Feature Pack 1 and 2 - Windows Virtual PC - Image with test agent configured - Powershell - …

ALM 4 Azure

The main goal when configuring the technical tool support for your processes is that it must support the same goals. For example when you execute an agile process with your development team. So your team is flexible, efficient and can deliver new functionality to the business against a predefined quality, repeatable and fast. The tools you are using must support and drive this goal. When your process of delivering new functionality takes days to deploy and configure even when you are capable of realizing this functionality, your team still can’t comply the goals. Let’s assume we executing an agile like processes and we have a risk driven mindset. We can write down several goals the tools must support… every team has its own goals, but the ones written down in this list are very common. Every change every technical tool support must support and drive towards these goals.

No team is the same. This list of scenarios isn’t a maturity level kind of list, it are more advancement levels of tool support. Not every team wants to go through the knowledge gathering, hardware and software investment necessary to implement a specific scenario. It is definitely a balance between effort, money and profit, benefit.

1: Engineering only

The developer only scenario is for really small teams, one developer which also does the testing. Most exercises and hands-on labs make use of this scenario.

The engineers create functionality in Visual Studio 2010. The source code is checked in Team Foundation Server. Engineers can make use of work items in TFS but isn’t necessary.

Source code is compiled and unit test if any are executed. Other code quality checks can be performed. Implementation checks by the layer diagram are interesting but not necessary. No real quality gates.

Engineers deploy the application from Visual Studio by; or creating a package and configuration file or by using the ‘single click’ deployments from within Visual Studio.

Soon on: http://www.youtube.com/user/clemensreijnen

A common deployment flow. 1. Local development in emulator environments. 2. Hybrid of local and Windows Azure, when storage is stable. Different

engineers can work against the same data source. The compute emulator is almost similar as the hosted service environment on Azure, but the storage emulator has some big differences compared with the Azure storage.

3. Everything in Windows Azure in staging. 4. Swap from Staging to Production.

• Debugging is not currently supported in Windows Azure, intelitrace is. • Set breakpoints & debug in Local Development Fabric. • Test initially with development storage, but test with Windows Azure

storage to test with large volumes of data whilst still keeping your roles local for debugging

• Once you are happy with the Worker/Web Roles running locally deploy everything to Staging and run tests in this environment

• Once all tests in staging pass, promote everything to production • Worker roles in the “Staging” project are operational – and as such will

process messages from queues etc. You should design for this. • Staging also cost money. • Storage costs are way less as compute costs. Test as much local as

possible.

The developer only scenario has a lot of benefit and many organizations start with it because its painless. But it’s also faults can slip in easy and time consuming. Pro: Easy installation and configuration Single click deployment from VS2010 Con: No collaboration Easy deployment errors (configuration) What about test and ops

2: Developer with manual tester

A bit bigger team with a test specific role. Developers have their quality gates within the build. Testers analyses the quality of the system and help the team with risk classifications. It can gets a bit challenging when entering this scenario. Testers and engineers often have different approached, different methodologies and different goals. Now they have to start working together. Work with workitems and supported process templates will help the adoption, providing the team with the benefits the technical support can give them when working together will also help. But mainly it’s a cultural process (see first alm slides) people have to work together and take a shared responsibility for the success of the system.

Engineers and testers work together with workitems In Visual Studio Team System 2010 all test roles are provided with clear and better support within the Application Lifecycle. Testers do not use their own separate technical tools anymore, but use integrated tools that are used by architects and developers. Effectively tearing down the wall between developers and testers. But good tools are not enough. Also, a clear separation of roles, tasks, and authorizations are necessary. Finally and most importantly, a structured approach determines how successful you are with your test strategy. For example the role of the tester and the usages of work items in collaboration with engineers. During the planning phase of the project, also called iteration 0 [first blue piece], user stories are collected / brainstormed / defined /… in VSTS this information is collected in the work item type ‘user story’ During the planning of the iteration the team starts to breakdown the user stories [which are selected for that iteration] in implementation tasks. Within VSTS this is done in the implementation tab of the user story Work item. The new 2010 functionality of hierarchies between work items is used for this. More reading: http://www.clemensreijnen.nl/post/2009/09/03/Agile-Testing-with-VSTS-2010-and-TMap-Part-01-User-stories.aspx More reading: http://www.clemensreijnen.nl/post/2009/04/21/Testing-in-the-Application-Lifecycle-with-Visual-Studio-2010-Test-Edition.aspx

As in scenario 1, the engineers create functionality in Visual Studio 2010. The source code is checked in Team Foundation Server. The test role is added to the team. They specify and execute manual test cases in Microsoft Test Manager.

Same as scenario 1

Same as scenario 1

Tests are executed against the Azure staging environment. Bugs are filled in TFS. Using work items together with engineers is a must, starting with bugs followed by user stories test cases and tasks.

Microsoft Test Manager 2010 is for testers what Visual Studio is for developers. That is to say, where Visual Studio is an IDE – an integrated development environment, Test Manager is an ITE – an integrated test environment. This is the interface that a tester will use to create test cases, organize test plans, track test results, and file bugs when defects are found. Test Manager is integrated with Team Foundation Server, and is designed to improve the productivity of testers. While I am not going to do a deep-dive of all that Test Manager can do, it is important to understand how it integrates with the Visual Studio Agents to make the most of test case execution, and ensure that when a tester files a bug, it is actionable with very little work on the tester’s part. Test cases are work items with a specific tab where test steps can be defined. These test steps can only be edited from within MTM. You can create test cases for your manual tests with both action and validation test steps by using Visual Studio 2010 Ultimate or Visual Studio Test Professional. You can add test cases to your test plan using Microsoft Test Manager. More information: http://msdn.microsoft.com/en-us/library/dd380712.aspx http://msdn.microsoft.com/en-us/library/dd286729.aspx

Where to execute tests? The challenge of this scenario is that the staging environment usages cost money and with some serious testing this will grow every sprint. To balance what is tested where these costs can be minimized and kept in a comfortable range. Executing all the tests in the staging environment isn’t an option, too expensive. There are two other environments where to execute tests. One is the environment the tester uses to specify the tests, the other one the build environment. With Azure the challenge of running an Azure application outside of azure is the availability of the compute emulator. While the build environment is a server environment this isn’t the recommended place to execute manual tests. Only automated, like unit tests, are interesting to execute on the build. Balancing the testing effort over the environments is challenging. What is tested at developer level, with unit tests, isn’t useful to test during functional testing. And what should be tested in the staging environment while all functional and system tests are already executed in the testers environment.

Execute tests in test environment by using the compute and CSRun. To execute tests, on a tester’s environment without Visual Studio 2010 installed, you need to set and install several things. The easiest way is to let the environments run their own compute emulator and version of the cloud application under test and make use of the azure storage for data. You need to install the Azure SDK and have IIS7.0 available on the test machine. (2) With CSRun.exe testers can launch the cloud application within their own compute storage with the CSX folder and the CSFG file (1). The only challenge left is the port number (3) created by the CSRun.exe command. Microsoft Test Manager record and playback capability will be useless when this one changes over time, and it will change. Test case should be rewritten and executed. This change of URL makes the use of shared steps (4) in Microsoft test manger a must. You can re-record them without breaking record and play back notion of the previous executed test cases. Shared steps will aslo proof there use when you want to execute a test case in staging and production after you have record it in the emulator.

Soon on: http://www.youtube.com/user/clemensreijnen

Collect data from the Azure instance while execution a test A challenge with testing cloud systems is that instances recycle on a regular basis, it’s not sure that the environment on where the tests were executed is still the same as when the engineers the bug tries to reproduce and resolve. So, when an engineer wants to find the bug in the system it is not sure he got access to the log files of the system, for example IIS logs, trace logs etc. When the cloud application has diagnostics enabled you can create a custom diagnostic adapter which collects information and queries the logs for the test execution time frame and add these logs to the test result for the engineer. See: http://www.clemensreijnen.nl/post/2011/01/05/MTM-Azure-Data-Diagnostic-Adapter-a-nice-ALM-for-Cloud-scenario-e280a6.aspx

See demo WAD MTM Adapter: http://www.youtube.com/watch?v=xKxFtfKh6yo

Microsoft Test Manger capabilities for cloud system testers. Microsoft Test Manager has some very useful features and capabilities for testers of cloud systems. Beside the usages of work items which help the testers to get the same heartbeat as engineers, these capabilities are helping managing the testing effort with, test plans, configurations, suits and the bug workflow. Two features are really useful while testing cloud systems, the shared steps and the diagnostic adapter extensibility. Shared steps make it easy to handle the different environments. First tests are executed on the compute emulator, second on staging and finally the production environment all with different URL’s. The diagnostic adapter extensibility makes is easy to collect environment information for bug solving. Two MTM features aren’t working for cloud systems in relation to MTM. Test Impact analysis and Intellitrace. Intellitrace does work from within Visual Studio as the deployment is configured to use it, but not from the MTM.

The developer with manual tester scenario has a lot of benefit. The main benefit is that the system is going to be tested in a well thought manner. That engineers and testers are connected and have the same heartbeat will solve some big project management challenges and will save a lot of time. For test execution the biggest challenge is not to test everything on Azure, but balance it. Only platform verification tests on azure the other system and functional tests on the compute emulator. This needs some configuration on the test environments. Pro: Easy installation and configuration Single click deployment from VS2010 Testers connected, same heartbeat as dev Proven quality Con: Easy deployment errors (configuration) Time consuming (deploy and test) Not repeatable (annoyed testers) Testers connected

3: Developer with manual tester and deployment build

To drive forward when the team grows or when we need to but some more effort in the stability of the system development process, we have to look at the build process on the build server and the deployment. Making the deployment of cloud systems repeatable for different environments will make the whole process more stable. We as a team can deliver functionality in a faster proven way. When we work agile we want to deliver functionality in a fast flexible way, having an automated process in place which supports this will raise our quality bar.

Same as scenario 1 for engineers and testers, they specify test cases and write source code, test cases are executed on the compute emulator. In collaboration with operations build and deployment scripts are configured.

During the build, unit test are run, deployment packages and configurations are made.

As a final step of the build the cloud system is deployed to the Azure staging environment.

Tests are executed against the Azure staging environment. Bugs are filled in TFS. Using work items together with engineers is a must, starting with bugs followed by user stories test cases and tasks.

Automating deployment Manual deploying Azure systems is error-prone, changing configuration files and connection strings can go wrong, resulting in an instable deployment with annoyed testers (system isn’t ready for testing) and users (we can’t show them anything). There are several different ways to deploy a system to azure, the powershell CmdLets are the easiest to use. The Cmdlets can be downloaded from this location http://archive.msdn.microsoft.com/azurecmdlets More reading: http://msdn.microsoft.com/en-us/library/ff803365.aspx http://scottdensmore.typepad.com/blog/2010/03/azure-deployment-for-your-build-server.html http://blogs.msdn.com/b/tomholl/archive/2011/02/23/using-msbuild-to-deploy-to-multiple-windows-azure-environments.aspx

For the next demo we are using cmdlets the build target and msbuild to create packages and deployment as described in http://blogs.msdn.com/b/tomholl/archive/2011/02/23/using-msbuild-to-deploy-to-multiple-windows-azure-environments.aspx. Note: You don’t want to configure the automatic deployment on a continuous integration build. This definitely won’t work, release builds or sprint review builds aren’t run that often can do the deployment.

Soon on: http://www.youtube.com/user/clemensreijnen

Having automated deployment in place is a boost for the quality of the system delivery process. Although it can be challenging to get the initial configuration right, allot of different technologies must be used, the benefit the team gets from it us high. Pro: Easy installation and configuration No click deployment from build Repeatable ‘proven’ deployments* Testers connected, same heartbeat as dev Proven quality Con: Time consuming testing Application can contain ‘annoying’ bugs Build workflow knowledge necessary Powershell, ccproj tweaks, target files, certificates

4: Developer with automated regression tests, manual tests and deployment build

With automatic deployment in place we can start to configure automatic testing. Although you can run automatic UI test cases from within Visual Studio with the compute emulator on the developer environment, so automatic testing could be in place earlier. But running automatic UI tests on a developer’s environment executed by the developer and with the result collected in Visual Studio is more for bug reproducing and solving than testing, where you say something about the quality of the system. You want the tester to execute them on a test environment and with result in the test plans. Automatic testing will speed up system development, testers and developers have the same hard beat, but the further in a project you get the more regression test cases will need to be executed. This execution of regression tests will take more and more time bringing friction to the same heartbeat mindset.

Engineers write source code and testers specify and execute test cases on the compute emulator. In collaboration with engineers manual test cases are automated with CodedUI and associated with Test Cases. Test case automations are ‘tested’ (dry run) on the developer’s environment.

During the build, unit test are run, deployment packages and configurations are made.

As a final step of the build the cloud system is deployed to the Azure staging environment.

Tests are executed against the Azure staging environment. Bugs are filled in TFS. Using work items together with engineers is a must, starting with bugs followed by user stories test cases and tasks.

Associated test case automations are executed from MTM on the compute storage.

There are different technologies available to create tests within Visual Studio. Two of them are suitable for the automation of manual tests. Web tests and CodedUI tests. The main difference between them is that CodedUI really interacts with the UI, it uses IEDOM for automation. Web tests uses HTTP get and post to automate the tests. The CodedUI functionality is strongly connected with Microsoft Test Managers action recordings (created while executing a test case in MTM).

CodedUI tests can better be used for functional tests and web test can better be used for performance tests and load tests.

Execute tests as soon as possible in the lifecycle. We can divide the different tests technologies in development tests, load performance tests, automated UI tests and manual tests, and create a test specific sub category for the automated and manual tests with the categories: Functional Testing, Integration Testing, Acceptance Testing and Platform Testing (Azure specific, answers the question: will it run in the cloud and is the deployment and configuration correct test). Development tests are executed during development and in the CI build. For load and performance we need a full blown environment, this is easy with Azure but we do need a complete feature we need to test available in the cloud. So, these kinds of tests when part of the Definition of Done probably will be executed at the end of a sprint. Automated UI tests are really valuable, automated as soon as possible. These tests can cover just one feature (Functional Testing) and can be created and executed in the compute emulator as soon as the feature is implemented. Automated Integration tests are harder to execute within a sprint, because they often cover more scenario’s, when integration testing is part of the Definition of Done, these should be moved to the undone list (see: Agile Test practices with Microsoft Visual Studio 2010 [TMap and Scrum] ). Platform testing focus on specific things like if the deployment and configuration is correct and does it run correct in the cloud, often these are very common tests and can be executed within a sprint after the build, automated as soon as possible. Acceptance testing is often done outside the team by the business users, keeping those tests connected with the team is very useful for bug solving, test coverage and automation.

Generated CodedUI tests from Microsoft Test Managers action recordings have all the steps as methods in the CodedUI test. To make the test suitable to be executed in different environments (emulator, staging, production) without having to change the code constantly or change the test data parameters settings.

Another way to make customizations to the behavior of the codeUI is by using the UImap editor, which can be found in Feature Pack 2. It helps you tune the search conditions for controls but more important for Azure Applications it helps you extract methods like ‘open application’ out of the xml and code generation to a partial class where you can edit the behavior for all test methods who use this method.

Soon on: http://www.youtube.com/user/clemensreijnen

There are multiple ways to execute your test automation effort, you can run them from and on Microsoft Test Manager, from and on Visual Studio, during the build and on the Build Server or during the build on Test Agents configured with Test Controllers or … many flavors. So, where should you execute your automated tests to test an azure application, and how should you configure your test infrastructure. One thing to keep in mind when making a decision, testing on Azure costs money. You can configure VS2010, MTM or the Build server to execute the automated tests against an azure deployment. But it’s cheaper to run most of them against an emulator deployment. For sure you need to balance this decision; there are tests which have to run against the Azure deployment, so called Platform Verification Tests. They verify is the app is running correct in the cloud, also if the app is configured correct in the cloud. All the other test, functional system etc, can be executed on an emulator deployment.

Both test executions (emulator deployment and azure staging / production deployment) can be configured to run from VS2010, MTM or the Build server, with or without the use of test agents. Each execution platform has it’s pros and cons. For example from VS2010 is more a developer test, which verifies a bug fix or repro. Test results aren’t collected in TFS or Test Manager and no connection with linked user stories. It’s an easy way to dry run your tests, specially because the emulators are already in place an loaded by VS2010. No additional actions need to be made. Execution on the build server is a kind of strange; execute manual tests on a server. It also will get challenging when you want to load the emulator during the build to run tests against. Also there will be no collection of test results in Test Manager (no reporting on test points) http://www.clemensreijnen.nl/post/2011/02/21/Running-Automated-Tests-on-Physical-Environments-the-different-flavorse280a6.aspx

The preferred way of execution automated tests is from within Microsoft Test Manager with associated automation in manual test cases. Another benefit is that you can add scripts and deployment actions which run before and after an automatic test.

Flavor E: Execution from MTM during Build… Purpose: Part of BVT. Preferred configuration above flavor C. Flavor D and E can be configured together. Triggered by: Build Information: Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx How to: Run Test Cases with Automation from the Command Line Using Tcm http://msdn.microsoft.com/en-us/library/dd465192.aspx Pro Test run distributed over test environments. Tests can be configured to run on different configured environments Test Result in MTM and TFS Triggered by build Test Settings from MTM Con Hard to configure maintenance of TCM commands in build

Flavor D: Execution from Microsoft Test Manager Purpose: Part of Regression Tests (other type of test than BVT). Triggered by: MTM user, right mouse click on test case, run Information: Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode) Configure Lab Center in MTM to use test controller and create test ‘agent’ physical environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html Pro Test run distributed over test environments Test Result in MTM Test Settings from MTM Full control by the tester Con Test Controller needs to be configured with a Project Collection (one controller per collection) Manually triggered by Tester (or pro) Hard to configure Hard to see which test case is automated

Soon on: http://www.youtube.com/user/clemensreijnen

Pro: No click deployment from build Repeatable ‘proven’ deployments* Testers connected, same heartbeat as dev Proven quality Automated BVT on different Environments Comfortable Acc Testing Done Done Con: Build workflow knowledge necessary Powershell, ccproj tweaks, target files, Certificates Test Infrastructure knowledge necessary A balanced thinking of test automation

5: Developer, Automated Tests, Build, Deploy, Acceptance test and Operations

The final scenario also added acceptance testing and operations to the process. Acceptance testing often done by the business users and often very disconnected with the team. In the previous scenarios there was a lot of focus on setting up environments so testers won’t get annoyed that the test environment isn’t ready. This is even more important having annoyed business users who can’t test the system isn’t very well for adoption. One big benefit of Azure is that all environments are the same, the same guest OS. So, deployment packages and configuration files in one environment will also work on other environments. Operations needs to provide the business with valuable information how the system is used, so the business can make decisions about the project portfolio. For cloud applications also the monetization of the usages is interesting.

Team development. The Team implements the requested features, specifies test cases and determines operational SLA and usages parameters. On local environments ‘compute and storage emulators’ execution of unit tests, dry run CodedUI tests (customize code to handle different environments) and associate CodedUI tests with a MTLM test case, execute the automated test cases from MTLM (make use of CSRun). As soon as possible emulator storage to the Azure storage, due to environment differences and same test- development data storage for the team. (green line) Engineering and design should also focus on tracing and diagnostics, this is important during testing and operation of the cloud application.

Build, Unit test, Deploy, UItest flow, manual test During build, not the CI build but for example a Sprint Review Build deploy the application automatic to the staging environment. First compile, unit test and the creation of the deployment package and configuration files for different environments. After deployment, automated platform/ staging tests are run. These are CodedUI tests which verify the installation and stability on the Azure environment. Test infrastructure can be configured to run the test from different environments to distribute the tests for time saving or to tests the azure application with different client configurations. The test are executed during the build with collection of the result in MTLM by using TCM.exe http://msdn.microsoft.com/en-us/library/dd465192.aspx So, during the build ‘Build Verification Tests’ and after deployment ‘Environment Verification Tests’ are executed, when these are successful you could VISPSwap the Cloud application from staging to production for additional manual testing or for the sprint review (you can use the Management API for the VISPSwap or CSManage.exe).

Release Drop. The package created during the build is reused in another Azure subscription for security the keys in this environment aren’t used by testers or developers. “Adatum uses the same package file to deploy to the test and production environments, but they do modify the configuration file. For the aExpense application, the key difference between the contents of the test and production configuration files is the storage connection strings. This information is unique to each Windows Azure subscription and uses randomly generated access keys. Only the two key people in the operations department have access to the storage access keys for the production environment, which makes it impossible for anyone else to use production storage during testing accidentally.” (from: Moving Applications to the Cloud on the Microsoft Azure™ Platform http://oreilly.com/catalog/0790145308795/ ) The business users execute there acceptance tests against the staging environment of the production subscription. By using MTLM they can execute manual tests, automated tests, and exploratory tests while still being connected with the TFS repository. This still gives the capabilities to provide very rich bug reports to the team. When acceptance testing is done, the azure application is manually swapped to production.

Operations (see this PDF Monitoring and Diagnostic Guidance for Windows® Azure™–hosted Applications ) The goal of application monitoring is to operationally answer a

simple question: Is the application running efficiently within its defined SLA parameters and without errors? If the answer to this question is no, then the Operations team needs to be made aware of this condition as soon as possible. Effective placement of monitoring on critical application and system breakpoints will help manage the hosted solutions. This document is intended for development and operations teams that need to monitor applications hosted on the Windows® Azure™ Services Platform, enabling incident, problem, and knowledge management.

Same sources are used for the MTLM Azure Diagnostic Data Collectors.

Two more interesting scenario’s. Not specific belonging to the ALM 4 Azure story, but it give some thoughts. TFS on Azure announced PDC2010 … http://blogs.msdn.com/b/bharry/archive/2010/10/28/tfs-on-windows-azure-at-the-pdc.aspx Other ALM Infrastructure on Azure … the opportunities are endless..

Thanks for reading all comments are welcome. This is work in progress….