Alm 4 Azure

A u t o m a t e d A L M f o r A z u r e C l o u d D e v e l o p m e n t ALM 4 Azure

1

• ALM • Azure • ALM 4 Azure

AGENDA

2

• Sogeti | Management Consultant

www.ClemensReijnen.nl

@ClemensReijnen [email protected]

Clemens Reijnen

3

application lifecycle management

http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx

4





5

ALM - Agile | Lean

Application Lifecycle Management Collaborate over Processes and Products with tools.

Agile, Lean and also TMap, DyA, ITIL Methodologies, principles customized, adopted and merged from all ALM roles.

optimized environments for supporting the ALM team goals

work together __with work packages __on alm artifacts

6

http://msdn.microsoft.com/en-us/library/bb385832.aspx

7





identical – independent environments

Guest OS Versioning

8

Config *.cscfg

Development & Deployment

SDK

*.cspkg

9

Start

10

ALM 4 Azure Goals: Collaborative

High Quality Flexible Automated Repeatable Efficient C

lou

d S

yste

ms

11

Scenario 1: Dev only Scenario 2: Dev and Test Scenario 3: Dev, Test and Build, Deploy Scenario 4: Dev, Auto Tests and Build, Deploy Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test and Operations

12


13

Scenario 1: Developer only

Staging Production

Developers implementeren het systeem Local unittesting

TEAM

Source Repository

(TFS) C# (implementatie + unittests)

‘SprintReview’Build Compile Run Unit Tests

1

2

3

14

Stages of Service Deployment

Taken from:

Deploying a Cloud Service from Visual Studio

DEMO

16

Scenario 1: Developer only

Pro: Easy installation and configuration

Single click deployment from VS2010

Con: No collaboration Easy deployment errors (configuration) What about test and ops

17


18

Scenario 2: Developer with manual tester

Staging Production


Testers specificeren testcases “Dry” run van tests tegen Development Fabric

TEAM

Source Repository



1

2

3

4

19

Microsoft Test Manager

21

Staging Production

TEAM

Source Repository

(TFS)

1

2

3

4

Functional, system, manual tests executed in compute emulator

with MTM.

Build Verification Tests (unit tests) and other quality gates,

executed in the build.

Platform accepatance tests executed on Azure stating with

MTM.

1 2 4

4

2

3

Get the CSX folder and CSFG file 1

1

Test a Cloud Service with Microsoft Test Manager

DEMO

24

Diagnostic adapters

http://www.clemensreijnen.nl/image.axd?picture=Windows-Live-Writer/3e88be580125/76A3C516/image.png

Azure Logs Diagnostic Data Adapter

DEMO

26

Capabilities • Link test cases to requirements • Fast Forward • Test data Iterations • Test case management • Test configurations

Interesting capabilities 4 Azure • Shared steps ( 4 VISP Swap ) • Diagnostic adapters ( Rich bug reports )

Not working capabilities in Azure • Test Impact 4 manual tests • Intelli Trace 4 manual tests (although possible custom

adapter) 27

Scenario 2: Developer with manual tester


Single click deployment from VS2010 Testers connected, same heartbeat as dev

Proven quality

Con: Easy deployment errors (configuration) Time consuming (deploy and test) Not repeatable (annoyed testers) Testers connected

28


29

Scenario 3: Developer with manual tester and deployment build

Staging Production


Testers specificeren testcases Dry run van tests tegen Development Fabric

TEAM

Source Repository



Package (CSRun.exe, CSPack,exe, CSManage.exe) Deploy 2 Staging (CMdLets, Powershell)

1

2

3

4

30

Management API CSPack CSRun CSManage PowerShell CmdLets Build Targets

Automatic Deployment

http://scottdensmore.typepad.com/blog/2010/03/azure-deployment-for-your-build-server.html http://blogs.msdn.com/b/tomholl/archive/2011/02/23/using-msbuild-to-deploy-to-multiple-windows-azure-environments.aspx

31

http://scottdensmore.typepad.com/blog/2010/03/azure-deployment-for-your-build-server.html












http://blogs.msdn.com/b/tomholl/archive/2011/02/23/using-msbuild-to-deploy-to-multiple-windows-azure-environments.aspx


















1. Change .ccproj project • New build target which uses the built-in CorePublish target

2. Create PowerShell script • Called by build target and uses Azure CMDLets

3. Edit Build Config • Add the necessary parameters

4. Do something with certificates • Put the Azure certificate on the build server

Deploy during build

32

Deploy a Cloud Solution within the Build

DEMO

33


No click deployment from build Repeatable ‘proven’ deployments*

Testers connected, same heartbeat as dev Proven quality

*demo needs some additional environment tweaks for different csfg files

per environment

Con: Time consuming testing Application can contain ‘annoying’ bugs Build workflow knowledge necessary Powershell, ccproj tweaks, target files, certificates

Scenario 3: Developer with manual tester and deployment build

34


35

Scenario 4: Developer with automated regresion tests, manual tests and deployment build

Staging Production


Testers specificeren testcases Dry run van tests tegen Development Fabric Automation van testcases

(moeten gedraaide testcases zijn voor CodedUI) UI automation tests run tegen Development Fabric

en associëren met test plan

TEAM

Source Repository

(TFS) C# (implementatie + unittests) Test cases (automated UI tests)


Package (CSRun.exe, CSPack,exe, CSManage.exe) Deploy 2 Staging (CMdLets, Powershell)

1

2

3

4

5

36

CodedUI

Where? When? How? Who? Why? What?

Where? When? How? Who? Why? What?

40

Different URL’s

1

2

41

UIMap Editor

Create automatic Tests for the Cloud Application with CodedUI

DEMO

42

a) At the Build server?

b) In the Development environment?

c) In the Test environment?

d) In the Azure environment?

Where to run the auto test?

43

Visual Studio

Test Manager

Team Foundation Server

Build Server

Test Agent

Test Agent

Test Agent

44

Configurations for automated test execution:

VS 2010

MTM 2010

TFS 2010

Build Controller

Test Controller

Build Agent

Test Agent

On physical environments (no lab)

45

46

Automated Test Settings

A

Flavor A: Execution from VS2010…


Real developer test / test automation dry run (Triggered by: right mouse click – run tests, in test project) http://msdn.microsoft.com/en-us/library/dd286580.aspx No additional configuration needed* * Data Diagnostic Settings in the Local.Testsettings file are configurable.

Pro: Easy setup

Debug-able test automation

Con: No collection of test results in TFS Run on developer environment 47

B

Flavor B: Execution during build with Build controller…


Part of Build Verification Tests (triggered by: build) Set the Build Service to run in interactive mode http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType * Data Diagnostic Settings in the *.Testsettings file are configurable.

Pro: Easy setup

Test results in build result

Con: No collection of test results in TFS Build Controller needs to run in interactive mode Tests are executed on build environment Run on build environment

not recommended, strange to run UI tests on build

48

http://msdn.microsoft.com/en-us/library/dd286580.aspx


http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx














http://msdn.microsoft.com/en-us/library/ms182465.aspx



C

Flavor C: Execution during build with Test controller…


Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B)

Pro: Test run on test environments

Tests run on multiple environments Test Results in Build result

Con: No collection of test results in TFS Harder to configure Need for specific test client environments Test Settings from VS

Preferred configuration above flavor B

49











D

Flavor D: Execution from Microsoft Test Manager…


Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html


Tests run on multiple environments Test Result in MTM and TFS

Test Settings from MTM

Con: Manually triggered by Tester Hard to configure Need for specific test client (same as MTM?)

Other type of test than BVT

50



http://msdn.microsoft.com/en-us/library/ee390842.aspx







http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html


















E

Flavor E: Execution from MTM during Build…


Part of BVT (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run MTC or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx


Tests run on multiple environments Test Result in MTM and TFS

Triggered by build Test Settings from MTM

Con: Hard to configure Need for specific test client (same as MTM?)

Preferred configuration above flavor C Flavor D and E can be configured together

51



http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx



















Automatic test a Cloud Service with Microsoft Test and Lab Manager infrastructure

DEMO

52

Pro: No click deployment from build

Repeatable ‘proven’ deployments* Testers connected, same heartbeat as dev

Proven quality Automated BVT on different Environments

Comfortable Acc Testing Done Done

*demo needs some additional environment tweaks for different csfg files

per environment

Con: Build workflow knowledge necessary Powershell, ccproj tweaks, target files, Certificates Test Infrastructure knowledge necessary A balanced thinking of test automation

Scenario 4: Developer with automated regresion tests, manual tests and deployment build

53


54

Staging Production

Developers implement the system Local unit testing

Testers specify test cases Dry run tests cases against Development Fabric Automation of test cases

UI automation tests execute against Development

Fabric and associate with a test case Operations determine instrumentation data DFO, SLA Usages Metrics, payment

‘Release ’Drop Package (same as test env

package) Acc testing met MTM en

Diagnostic Data Collectors

Automated ALM for Azure Cloud development Efficient, repeatable with proven quality fast flexible delivering of useful functionality … Making optimal use of the characteristics of Azure: - Identical environments (Guest OS versioning). - Separate in depended staging production environments. - Easy up and downgrade of applications. - Unambiguous deployment mechanism - rich diagnostic capabilities - Simplified Application Deployment & Management - …

TEAM

BUILD (TFS)

Acceptation

C# (implementation + unittests) Test cases (automated UI tests)

‘SprintReview’Build Compile Run Unit Tests Package (CSRun.exe, CSPack,exe, CSManage.exe) Deploy 2 Staging (CMdLets, Powershell) Run automated Test Suites (MTC.exe) with different

environments (test controller – agents infrastructure) Collect and publish test results “Build Verification Tests” green -> deployment 2 ‘test

'production

1 2

3

4

55

Scenario 1: Dev only Scenario 2: Dev and Test Scenario 3: Dev, Test and Build, Deploy Scenario 4: Dev, Auto Tests and Build, Deploy Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test and Operations Scenario 6: TFS on Azure Scenario 7: Other Infrastructure on Azure

56

57

Technology

Alm 4 Azure