The Release Pipeline ModelApplied to Managing Windows Server and Microsoft Cloud
Michael GreenePrincipal Program ManagerMicrosoft Enterprise Cloud CAT
whoamiMichael GreenePrincipal Program ManagerEnterprise Cloud Engineering CAT Team(Customer Feedback, Technical Enablement)CAT PM for PowerShell & DevOps
12 yrs at Microsoft3 yrs BPOSD/O365D Ops
@migreeneMgreenegithttp://aka.ms/thereleasepipelinemodel
When does a concept become a revolution?
First, the new candidate must seem to resolve some outstanding and generally recognized problem that can be met in no other way.Second, the new paradigm must promise to preserve a relatively large part of the concrete problem-solving ability that has accrued to science through its predecessors.
(The Structure of Scientific Revolutions, Thomas S. Kuhn)
Reading http://stevenmurawski.com/devops-reading-list/
Customer Landscape
patterns & practices
how do we include Windows Server?
that’s a little more on the dev side, than IT
Pro?
Treat Servers like PaaS
service consistency
fabric consistency
Software As a Service
consume service
Platform As a Service
deploy apps, jobs, data
Infrastructure As a Servicecustom baselinemanage binaries
SQL Azure
Azure App Services
Azure Virtual Machines
application consistency
Increasing opportunity
for configuration
drift
Physical/Virtual Servers
Configuration as Code• Lots of API's• Complex scenarios should be property:value• Living documentation• PowerShell DSC is the PLATFORM that all solutions can
use to deploy and manage Windows Server• Azure Resource Manager is also Configuration as Code• Nothing replaces knowledge about the systems you
intend to manage
The Release Pipeline
Who changed the environment?
What did they change, exactly?
When did the change occur?
What we did before
Prototype in VMSubmit change formPresent to CABArgue that the change is worthwhileConflict oversight by committeeUpdate docs
What we do now
• Document everything in configuration as code• Including automated deployment and testing
• Clone latest to workstation•Make change• Run tests locally• Push to Source• Who, What, When, Why (comments)
•Merge with Master, resolve conflicts if needed
How will I catch problems at the earliest possible moment?
Can elements be combined cleanly to produce the correct results?
How will I be notified of a problem?
What we did before
• Remote in on Saturday•Make sure servers are listed in RDCMan• Connect to servers and make changes• Reboot in order• Say a prayer / squeeze rabbit’s foot / perform tribal dance• Check that services still work correctly
What we do now
• Trigger Build service• Webhook/agent, schedule, manual
• Run scripts stored in Source• Validate quality• Perform work• Capture output
* opportunity to implement Just Enough Administrationhttp://aka.ms/JEA
Engineer for Resiliency• What will this project require on Day 2? Day 30?
Day 100?• Monitoring rules• Monitoring alerts• Automation runbooks• JEA endpoints• Onboard to backup services• Operational Validation Tests
How do we check for regulatory issues?
How do I know this change will not cause an outage?
Will this change work across every variation I have in my environments?
Does this configuration meet our business requirements?
What we did before
• Create VM• Install OS• Install app/service• Implement change• Manual, test OU, run script
• Try app/service• Make an educated guess whether it still works
What we do now
• Build runs Scripts from Source• Linting - Script Analyzer
• Must meet guidelines set by organization• Unit Tests – Pester
• Functionally should work as expected• Integration Tests – Pester / Kitchen
• Should work across matrix of diverse combinations• Operational Validation – Pester / OVF
• Service should do what it is supposed to do• TDD
Integration Testing with Pester
All examples courtesy of the following blog post - https://pshirwin.wordpress.com/2016/04/08/active-directory-operations-test/
$ADConfiguration = @{ Forest = @{ FQDN = 'pshirwin.local' ForestMode = 'Windows2012R2Forest' GlobalCatalogs = @( 'DC-DSC-01.pshirwin.local' ) SchemaMaster = 'DC-DSC-01.pshirwin.local' DomainNamingMaster = 'DC-DSC-01.pshirwin.local'
} Domain = @{ NetBIOSName = 'PSHIRWIN' DomainMode = 'Windows2012R2Domain' RIDMaster = 'DC-DSC-01.pshirwin.local' PDCEmulator = 'DC-DSC-01.pshirwin.local' InfrastructureMaster = 'DC-DSC-01.pshirwin.local' DistinguishedName = 'DC=pshirwin,DC=local' DNSRoot = 'pshirwin.local' DomainControllers = @('DC-DSC-01') }
Describe 'Active Directory configuration operational readiness' {
Context 'Verifying Forest Configuration'{ it "Forest FQDN $($ADConfiguration.Forest.FQDN)" { $ADConfiguration.Forest.FQDN | Should be $SavedADReport.ForestInformation.RootDomain } it "ForestMode $($ADConfiguration.Forest.ForestMode)"{ $ADConfiguration.Forest.ForestMode | Should be $SavedADReport.ForestInformation.ForestMode.ToString() } }
Context 'Verifying GlobalCatalogs'{ $ADConfiguration.Forest.GlobalCatalogs | ForEach-Object{ it "Server $($_) is a GlobalCatalog"{ $SavedADReport.ForestInformation.GlobalCatalogs.Contains($_) | Should be $true } } }
POSHSpecDescribe 'Services' { Service w32time Status { Should Be Running } Service bits Status { Should Be Stopped }}
Describe 'Files' { File C:\inetpub\wwwroot\iisstart.htm { Should Exist } File C:\inetpub\wwwroot\iisstart.htm { Should Contain 'text-align:center' }}
Describe 'Registry' { Registry HKLM:\SOFTWARE\Microsoft\Rpc\ClientProtocols { Should Exist } Registry HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ "SyncDomainWithMembership" { Should Be 1 } Registry 'HKLM:\SOFTWARE\Callahan Auto\' { Should Not Exist }}Describe 'Http' { TcpPort localhost 80 PingSucceeded { Should Be $true } TcpPort localhost 80 TcpTestSucceeded { Should Be $true } Http http://localhost StatusCode { Should Be 200 } Http http://localhost RawContent { Should Match 'X-Powered-By: ASP.NET' } Http http://localhost RawContent { Should Not Match 'X-Powered-By: Cobal' }}Describe 'Hotfix' { Hotfix KB3116900 { Should Not BeNullOrEmpty} Hotfix KB1112233 { Should BeNullOrEmpty}}
https://github.com/Ticketmaster/poshspec
How do I make changes without granting long term administrative access?
Does anyone need to sign-off before deployment?
How do I keep services consistent across all my environments?
Can I integrate service management?
What we did before
• Request/create VM• Request/manage environmental changes• Install OS• Install app/service (or defer to app/service owner)• Onboard to patching, anti-malware, backup,
monitoring• Validate app/service is working• Go Live
What we do now
• Build runs scripts from Source• Potentially automatically releases when Tests pass• Deploy artifacts to environment• Simplify complex work through automation• Promote through stages• QA, Prod, Green/Blue
Getting Started Guide
• Use Source Control• Don't expect to cutover instantly• Require tests• Keep options open• Old and new tools can be integrated
Make change visible
• If change is happening fluidly, everyone needs to know• Your peers are not going to go check reports, the way
they check Facebook• Notify all stakeholders in the way they collaborate
Managing outages without randomizing servers
Map your requirements
Plaster
PSake
Pester
POSHSpec
OVF
PSDeploy
Lability
Community Projects
PowerShell Slack
Finally
Everyone must be onboard for long term success but it starts with one passionate engineer