Datacenter Services Datacenter Automation Design … · Web viewDesign Architecture Datacenter Automation Table of Contents 1Introduction3 1.1Background3

Design ArchitectureDatacenter Automation

<Author Name><Date>

Table of Contents1 Introduction.......................................................................................................................

1.1 Background..................................................................................................................1.1.1 Traditional automation methods........................................................................1.1.2 Optimizing IT service delivery and increasing quality of service – through orchestration..............................................................................................................

1.2 Customer and User Profiles..........................................................................................2 Solution Architecture.........................................................................................................

2.1 Orchestrator 2012 Components...................................................................................2.2 Orchestrator Architecture............................................................................................

2.2.1 DataBus.............................................................................................................2.2.2 Integration.........................................................................................................

2.3 Orchestrator 2012 System Requirements..................................................................2.3.1 Hardware Requirements...................................................................................2.3.2 Security Planning.............................................................................................2.3.3 Network Requirements.....................................................................................

2.4 Backup and Recovery Considerations........................................................................2.4.1 Management Server.........................................................................................2.4.2 Orchestrator Database.....................................................................................2.4.3 Monitoring Plan................................................................................................2.4.4 Services and Event log monitoring...................................................................2.4.5 Performance monitoring...................................................................................2.4.6 Antivirus exceptions.........................................................................................

2.5 Scalability Considerations..........................................................................................2.5.1 Runbooks Scalability........................................................................................2.5.2 Runbook Servers Scalability.............................................................................2.5.3 Orchestrator 2012 Performance Tuning...........................................................2.5.4 Resource Availability Requirements.................................................................2.5.5 Database Sizing...............................................................................................

2.6 High Availability Considerations.................................................................................2.6.1 Management server.........................................................................................2.6.2 Orchestrator database.....................................................................................2.6.3 Orchestrator Web Service................................................................................2.6.4 Orchestration Console......................................................................................2.6.5 Runbook Servers..............................................................................................

3 Orchestrator 2012 Architecture Patterns.........................................................................3.1.1 Design Pattern #1 – Single Server Orchestrator 2012 Infrastructure...............3.1.2 Design Pattern #2 – High Availability Orchestrator 2012 Infrastructure...........3.1.3 Design Pattern #3 – High Availability and Multi-Site Orchestrator 2012 Infrastructure............................................................................................................

Page 3

3.1.4 Design Decision................................................................................................4 Evaluate Solution Requirements........................................................................................5 Appendix A – Sample Use Cases........................................................................................

5.1 Use Case 1: Alert Remediation.....................................................................................5.1.1 Free Space Remediation....................................................................................5.1.2 Detect and repair secure channel......................................................................

5.2 Use Case 2: Maintenance.............................................................................................5.2.1 Advanced patching (patching machines in order, with pre- and post- checks)

65.2.2 SQL Server maintenance tasks...........................................................................

5.3 Use Case 3: Change management and provisioning....................................................5.3.1 Provisioning : Provision virtual machines...........................................................5.3.2 Provisioning: Application deployment..............................................................5.3.3 Administration: Put machines in Operations Manager maintenance mode.......

5.4 Use Case 4: Cross-Technology Integration.................................................................5.4.1 Connecting System Center Operations Manager to IBM NetCool (simple example)..................................................................................................................5.4.2 Connecting System Center Operations Manager to IBM NetCool (advanced example)..................................................................................................................

5.5 Use Case 5: Dynamic Resource Allocation.................................................................5.5.1 Web Farm demo: Scaling out a service to handle additional load....................

5.6 Use Case 6: Line of business and other.....................................................................5.6.1 New hire onboarding........................................................................................

Page 4

1 IntroductionSystem Center 2012 - Orchestrator is a workflow management solution for the data center. It enables automation of creation, monitoring, and deployment of resources in your environment. This document describes System Center 2012 - Orchestrator planning and deployment and presents key concepts and captures business and environmental requirements for the automation of datacentre tasks using System Center 2012 Orchestrator. This document provides a summary of the topics, techniques, and design patterns to help deploy and use System Center 2012 Orchestrator.

1.1 BackgroundToday customers are facing a broad range of challenges in their datacenters. On one hand the complexity of their environments is increasing every year due to server sprawl, various systems and platforms from different vendors, while on the other hand customers having a lot of pressure to reduce costs at the same time.

IT decision makers are constantly looking to streamline IT operations and processes, reduce the burden on IT resources, and improve their ability to meet the complex needs of the businesses that they support. They can accomplish this by automating time-consuming manual processes, a method used to keep the world’s largest and most efficient data center facilities operating with minimal manual oversight.

Automation allows IT departments to improve agility and response times and increase performance to better meet service-level agreements (SLAs). Decision makers can also adopt and maintain quality processes and performance that give them the consistency that they need to meet business requirements, while maintaining compliance with industry and government regulations. As these improvements are implemented, decision makers will have to make sure that their deployments can span heterogeneous environments—promoting adherence to similar management models and tools.

System Center 2012 Orchestrator automation capabilities will free up resources to focus on more strategic efforts and potentially reduce overall head count associated with specific tasks. Reduced reliance on human intervention or manual processes will improve

Page 5

efficiency, and overall quality through the reduction of human errors. In addition, automation will ensure that proper configuration management processes are followed and enforced, providing invaluable tracking of change control. Finally it provides a means and framework which will be used as a blueprint on how to achieve higher productivity.

1.1.1 Traditional automation methodsTraditional automation methods, namely custom coding and scripts, are useful for running simple tasks, however, they typically lack best practices, change management, documentation, and the flexibility required in an operations environment, where business rules and configuration settings change frequently.

Traditional automation methods, namely job schedulers, run and monitor batch jobs. Although job scheduling comprises an important function in a production-computing environment, it is not well suited to automate operational processes or run book procedures, as they provide little to no integration with surrounding systems.

Due to the recent recognition of this market by both IT organizations and leading industry analyst firms alike, some companies have attempted to accomplish the equivalent of Runbook Automation, by running lengthy scripts with a job scheduler. This technique is costly, unreliable, and error-prone.

While scripts and schedulers work well for small tasks, they can rarely scale to handle complex environments. They also lack sophisticated dependencies and reporting that allow users to keep audit trails of processes. As process requirements grow, and more functionality is added, the result is a complex mix of scripts, programs, and utilities that only a few people actually understand. More concerning is that home-grown scripts can quickly turn into a fulltime programming commitment as well as a time-consuming and costly management burden.

1.1.2 Optimizing IT service delivery and increasing quality of service – through orchestration

Runbook Automation solutions include many of the features and capabilities users require in a job scheduler, while also providing more advanced functions. IT process automation software can automate any administrative, maintenance, or business processes, such as restarting services, rotating logs, backing up data, deleting temporary files, and e-mailing files. It can also run several jobs on multiple machines, modify accounts, query databases, up load data and filter/read/send e-mail. In addition to standard enterprise requirements like load balancing, failover, failure routines, error handling, and logging, ITPA should also provide integration, orchestration, and process workflow.

As part of System Center 2012, The Runbook Automation component “Orchestrator” reduces operational costs and improves IT efficiency by delivering services faster, with fewer errors. This is achieved by replacing manual, resource-intensive and error prone activities with standardized, automated processes.

Orchestrator automates the end to end operational processes that traverse organizational boundaries – bridging historic IT silos (vendor specific backups and hardware, heterogeneous environments, physical and virtual environments, multiple management solutions, service desks,…). Where it was common in the past to implement multiple rigid “point to point” solutions, Orchestrator provides a rich and

Page 6

flexible 1:N automation between System Center components as well as non-Microsoft technology stacks in the datacenter.

Well defined workflow processes are the basis for implementing efficient incident, change, configuration and release processes in an IT environment. Repeatable IT processes lessens the risk for IT by eliminating opportunities for human error, as well as freeing up valuable IT staff from constantly performing mundane tasks. Thanks to its capabilities to integrate with many other technologies including monitoring, service desk and change management solutions, Runbook Automation enables the implementation of consistent operations processes in support of best practices and standards initiatives such as ITIL.

1.2 Customer and User ProfilesUsers of the proposed solution will be limited to the following core groups:

▪ Operations team (Users). These users will be the main consumers of the solution and will derive the biggest benefits from the automation provided. This group is responsible for performing all operational tasks and is currently developing all scripts for the automation of these tasks.The operations team will be the recipient of information and control gate status messages in the form of email messages.

▪ Solution Administrators. These users will be the administrators of the solution and will ensure system maintenance is performed, troubleshoot errors, and restore availability of the solution in the event of an outage.

▪ Runbook Designers. These users will be the designers of the system and will address the need to incorporate new features, capabilities and routines into the solution.Runbook designers will develop the runbooks and activities needed for the solution to achieve the stated objectives. This may include changes to runbooks in support of new functions, features or even to support new versions. These users will have a strong development background and should not be permitted to operate on or with the production systems. Their work should be completed in the development environment where design work can be completed and tested before being approved for use in the production environment.

Page 7

2 Solution ArchitectureSystem Center 2012 Orchestrator is a workflow management solution for the data centre. Orchestrator 2012 automates tasks for creation, monitoring, and deployment of resources in the environment.

2.1 Orchestrator 2012 ComponentsAn Orchestrator 2012 solution consists of multiple components as presented in Table 1.Table 1: Orchestrator 2012 Solution Components

Name Description Management Server

The Management Server is a service that provides a communication layer between the Runbook Designer and command line tools to the Orchestrator Datastore.

Orchestration Database

A SQL Server database that contains all of the deployed runbooks, the status of running runbooks, log files, and configuration data for Orchestrator 2012.

Orchestration Console

The Orchestration Console is a web-based tool with which an operator can view the list of runbooks, view current running status and start/stop runbooks.

Orchestration Web Service

The Orchestration Web Service is a Representational State Transfer (REST)-based service that enables custom applications to connect to Orchestrator 2012 to start and stop runbooks, and retrieve information about operations by using custom applications or scripts. The Orchestration Console uses this web service to interact with Orchestrator 2012.

Runbook Designer The Runbook Designer is the tool used to build, edit and manage Orchestrator 2012 runbooks.

Runbook Server Runbook Server is where an instance of a runbook runs. Runbook Servers communicate directly with the orchestration database. Multiple Runbook Servers can be deployed per Orchestrator 2012 installation to increase capacity and redundancy.

Runbook Server Monitor

The self-monitoring feature which monitors runbooks, activities, database connections and more to ensure that these items are running as expected.

Runbook Tester Runbook Tester is a run-time tool used to test runbooks developed in the Runbook Designer.

Deployment Manager

The Deployment Manager is a tool used to deploy Integration Packs, Runbook Servers, and Runbook Designers.

Integration Pack (IP)

An integration pack is a collection of custom activities specific to a product or technology. Microsoft and other companies provide integration packs with activities to interact with their product from an Orchestrator 2012 runbook.For further details about Orchestrator 2012 Integration Packs refer to http://technet.microsoft.com/en-us/library/hh295851.aspx.

Orchestrator Integration Toolkit (OIT)

The Orchestrator Integration Toolkit extends default Orchestrator 2012 activities library beyond the collection of standard activities and integration packs. The Integration Toolkit has wizard-based tools to create new activities and integration packs for Orchestrator. Developers can also use the Integration Toolkit to create integration packs from custom activities that they build by using the Orchestrator Software Development Kit (SDK).

Page 8

http://technet.microsoft.com/en-us/library/hh295851.aspx

2.2 Orchestrator ArchitectureThe following diagram illustrates each of the Orchestrator features and the communication between each.

The orchestration database is the center of the Orchestrator installation containing all runbooks, configuration settings, and logs. The management server is required as a communication layer between the Runbook Designer and the orchestration database. One or more runbook servers communicate directly with the database to retrieve runbooks to run and store information about the jobs created from the runbooks. The web service also communicates directly with the orchestration database and provides a web browser connection for the Orchestration console.

2.2.1 DataBusIntegration DataBus, with full flexibility on process branching and parallelism. The Orchestrator engine – called the “databus” – provides the following features:

Pass data between systems in one click with the unique Orchestrator publish/subscribe data bus for rapid integration

Create context-adaptive workflows with intelligent decision making logic to automate even the most complex processes

Run multiple, concurrent branches of a workflow for high volume processing

Page 9

Page 10

2.2.2 Integration 2.2.2.1 ExtensionsThe following shows multiple strategies available for extending the functionality provided by a standard installation of Orchestrator:

Name DescriptionIntegration Pack (IP) An integration pack is a collection of custom activities specific

to a product or technology. Microsoft and other companies provide integration packs with activities to interact with their product from an Orchestrator runbook.See Integration Packs for System Center 2012 - Orchestrator http://technet.microsoft.com/en-us/library/hh295851.aspx for more information


The Orchestrator Integration Toolkit lets you extend your library of activities beyond the collection of standard activities and integration packs. The Integration Toolkit has wizard-based tools to create new activities and integration packs for Orchestrator. Developers can also use the Integration Toolkit to create integration packs from custom activities that they build by using the Orchestrator SDK.

2.2.2.2 Integration with the System Center suiteWhile its automation capabilities are not restricted to Microsoft environments, Orchestrator also provides best-of-breed automation capabilities for the other System Center components. In particular, runbook automation is often tied to ITIL process execution (incidents, problem, changes), and in System Center 2012, the Service Manager service catalog comes with a built-in connector for Orchestrator runbooks. This eliminates the need for complex scripting and development to tie runbooks into change requests. With System Center 2012, once the manual or review steps of a change have been processed, Service Manager can hand off the request seamlessly to Orchestrator, which can update the change request and the CMDB in return. This integration is key in the way System Center components work together to achieve Private Cloud scenarios, especially for Infrastructure as a Service. Microsoft provides a “Process Pack” leveraging this feature, as a download.

2.2.2.3 Standard activitiesThere are a number standard activities to enable integration:

Code-free integration with other solutions and home-grown applications (run command line, run SSH, run script, work with databases, web services, etc…)

File system interactions (monitor, copy, move…files and folders) Perform schedule-based activities Monitor processes or system-level events Send notifications (including email) Manipulate text files Work with the databus, manage and interconnect runbooks

Page 11


2.2.2.4 Partners Integration PacksMore and more vendors are providing integration packs for Orchestrator, including Dell for Advanced Infrastructure Manager, NetApp, Cisco for UCS, DoubleTake, etc.

2.2.2.5 Community Integration PacksThe Orchestrator community is always expanding, here are some community contributions as of today, for example there are many contributions on Codeplex.

Page 12

http://orchestrator.codeplex.com/

2.3 Orchestrator 2012 System Requirements

This section describes Orchestrator 2012 system requirements and supported operating system. Table 2 details the software configuration required to install individual Orchestrator 2012 components.Table 2: Orchestrator 2012 Software Requirements

Name Operating System/Software

Requirements

Comments

Management Server

Windows Server 2008 R2Windows Server 2012

Management server collocated with Runbook Server will use the same database.


Microsoft SQL Server 2008 R2Microsoft SQL Server 2012

Only Database Engine SQL Server feature is required. The instance of SQL Server can either be installed locally on the management server or on a separate dedicated database server. Orchestrator 2012 requires SQL_Latin1_General_CP1_CI_AS database collation.



Orchestrator 2012 setup will enable the internet Information Server (IIS) role if it is not already enabled.

Orchestration Web Service


Orchestrator 2012 setup will enable the IIS role if it is not already enabled.

Runbook Designer

Windows 7 32-bit or 64-bitWindows Server 2008 R2Windows Server 2012

Orchestrator 2012 setup installs and enables .NET Framework 3.5 Service Pack 1 if it is not installed and enabled.

Runbook Server


Management server collocated with Runbook Server will use the same database.

2.3.1 Hardware RequirementsThis topic describes the hardware requirements for installation of the System Center 2012 - Orchestrator SP1 components. The server hardware requirements for Orchestrator are dependent on the number, size and complexity of the runbooks being executed. The following are the minimum hardware requirements:

2.3.1.1 Management Server 1 gigabyte (GB) of RAM minimum, 2 GB or more recommended 200 megabyte (MB) of available hard disk space Dual-core Intel microprocessor, 2.1 gigahertz (GHz) or better

2.3.1.2 Runbook Server 1 gigabyte (GB) of RAM minimum, 2 GB or more recommended 200 megabyte (MB) of available hard disk space Dual-core Intel microprocessor, 2.1 gigahertz (GHz) or better

2.3.1.3 Orchestrator Web Service 1 gigabyte (GB) of RAM minimum, 2 GB or more recommended 200 megabyte (MB) of available hard disk space Dual-core Intel microprocessor, 2.1 gigahertz (GHz) or better

Page 13

2.3.1.4 Management Server 1 gigabyte (GB) of RAM minimum, 2 GB or more recommended 200 megabyte (MB) of available hard disk space Dual-core Intel microprocessor, 2.1 gigahertz (GHz) or better

2.3.2 Security PlanningThis section describes the service account and user account requirements, as well as security considerations for Orchestrator 2012 deployment.

2.3.2.1 Service accountsThe following are Service accounts that are required for the services listed. These accounts have to be created before installing the features that use them.

Orchestrator 2012 Service

Comments

Orchestrator Management Service

The Orchestrator Management Service is installed on the management server. Its service account is specified during the installation of Orchestrator 2012. This is the same account used by the Orchestrator Management Service and Orchestrator Runbook Service on each computer to access system resources. The Orchestrator Management Service is responsible for maintaining the orchestration database, communicating with the Runbook Designers, and communicating with the Deployment Manager.The account used for the Orchestrator Management Service can be a local account on the management server if the database is installed locally. However, this configuration might not allow access to other network resources. If the database is located on another server, the account must be joined to the Active Directory domain so it can access the database server. This service account does not have to have domain administrator privileges, but it should be a member of the local Administrators group on the computer where the Orchestrator Management Service and Orchestrator Runbook Service are installed.The service account for the Orchestrator Management Service must have the following permissions:

▪ Permission to log on to the management server as a service. This permission is automatically granted during the installation process.

▪ Member of the Microsoft.SystemCenter.Orchestrator.Admins role in the orchestration database. The account is automatically added to this role during the installation process.

Orchestrator Runbook Server Monitor Service

The Orchestrator Runbook Server Monitor is installed on the management server and is responsible for monitoring the health of Runbook Servers. It uses the same account as the Orchestrator Management Service and requires the same permissions.

Orchestrator Runbook Service

The Orchestrator Runbook Service is installed on each Runbook Server. When Runbook Server is collocated with the Management Server, Orchestrator Runbook Service and Orchestrator Management Service use the same account (Orchestrator Runbook Service account). If additional Runbook Servers are deployed, a different service account can be specified. The service is responsible for running runbooks and for communicating with the orchestration database.By default, all activities in a runbook run under the service account of the Runbook Server on which they are running. Some activities can specify

Page 14


Comments

different credentials to be used for individual actions as required. Because runbook activities often access resources on other computers, the account used for the Orchestrator Runbook Service should be a member of Active Directory groups that have sufficient access to these external resources.The account for the Orchestrator Runbook Service must have the following permissions:

▪ Permission to log on to the management server as a service. This permission is automatically granted during the installation process.

▪ Depending on the resources that the runbook activities access, the service account might require additional credentials on remote computers. Specific activities can also be configured with alternate credentials if the service account does not have access to particular resources.

▪ This service account does not have to have domain administrator privileges, but it should be a member of the local Administrators group on the computer where the Orchestrator Management Service and Orchestrator Runbook Service are installed.

2.3.2.2 Runbook securityOrchestrator 2012 maintains security information in the Orchestration Database. Orchestrator 2012 imposes a security model that uses Active Directory user and user group accounts to grant specific Orchestrator 2012 security permissions to Orchestrator 2012 objects.

Orchestrator 2012 also leverages Distributed Component Object Model (DCOM) security rights granting local and remote launch, activation, and access permissions on Omanagement DCOM application to Orchestrator Users group during the installation. Because the Orchestrator Web Service uses the same group for authorisation, a domain group is required if the Orchestration Console is not installed on the management server. If the Orchestration Console is installed on the management server, the group can be a local group on the management server. The decision whether to use local or domain group depends on the group membership management requirements. Typically using a domain group provides better centralised access to the group as opposed to managing it locally on the management server.

Any user account added to this group is granted permission to use the Runbook Designer and Deployment Manager tools. By default, users in this group have the authority to perform the following actions:

▪ Create new runbooks.▪ View, change, and run existing runbooks.▪ Deploy new Runbook Servers.▪ Deploy new Runbook Designers.▪ Register and deploy integration packs.▪ View and change global settings for a Runbook Server.

When a user connects to the Orchestrator 2012 via Orchestration Web Service or Orchestration Console the user account permissions are automatically validated against the Orchestration Database.

Page 15

When a user connects to the Orchestrator 2012 via Runbook Designer, the user account automatically connects to Omanagement DCOM application resulting in the validation of the account for DCOM permissions. If the user is not a member of Orchestrator Users group, access to Omanagement is denied – unless the user is specifically granted Omanagement DCOM rights.

After DCOM permission validation, Orchestrator 2012 checks user security permissions against the Orchestration Database, and then the objects for which the user has permissions appear in the Runbook Designer.

Note

Orchestrator 2012 does not validate permissions on runbooks invoked by other runbooks. If a user has permission to execute a runbook and this runbook invokes another runbook that the user has no permissions for, both runbooks will be executed.

2.3.2.3 Orchestration database securitySecurity to the orchestration database is implemented through database roles in SQL Server 2008 or later. Table 3 lists the roles that are created in the orchestration database and the permissions granted to each.Table 3: Orchestrator 2012 Database Roles

Role Permission ObjectMicrosoft.SystemCenter.Orchestrator.Operators

SELECT [Microsoft.SystemCenter.Orchestrator.Runtime].[Jobs], [Microsoft.SystemCenter.Orchestrator.Runtime].[RunbookInstances], [Microsoft.SystemCenter.Orchestrator.Runtime].[RunbookInstanceParameters],[Microsoft.SystemCenter.Orchestrator.Runtime].[RunbookServers], [Microsoft.SystemCenter.Orchestrator.Runtime].[ActivityInstances], [Microsoft.SystemCenter.Orchestrator.Runtime].[ActivityInstanceData], [Microsoft.SystemCenter.Orchestrator.Runtime].[Events], [Microsoft.SystemCenter.Orchestrator.Statistics].[Statistics]

Microsoft.SystemCenter.Orchestrator.Operators

EXECUTE [Microsoft.SystemCenter.Orchestrator].[GetSecurityToken],[Microsoft.SystemCenter.Orchestrator].[AccessCheck],[Microsoft.SystemCenter.Orchestrator].[ComputeAuthorizationCache], [Microsoft.SystemCenter.Orchestrator.Statistics.Internal].[GetStatisticsSummary], [Microsoft.SystemCenter.Orchestrator.Runtime].[CreateJob],[Microsoft.SystemCenter.Orchestrator.Runtime].[CancelJob]

Microsoft.SystemCenter.Orchestrator.Runtime

SELECT All tables,dbo.[POLICIES_VIEW], dbo.[POLICY_REQUEST_HISTORY]

Microsoft.SystemCenter.Orchestrator.Runtime

INSERT dbo.[OBJECT_AUDIT]

Page 16

Role Permission ObjectMicrosoft.SystemCenter.Orchestrator.Runtime

INSERT, UPDATE

dbo.[OBJECTS], dbo.[ACTIONSERVERS], dbo.[POLICYINSTANCES], dbo.[OBJECTINSTANCES], dbo.[OBJECTINSTANCEDATA]

Microsoft.SystemCenter. Orchestrator.Runtime

INSERT, DELETE

dbo.[COUNTERINSTANCES], dbo.[POLICYRETURNDATA]


UPDATE dbo.[POLICY_PUBLISH_QUEUE]


CONTROL [ORCHESTRATOR_ASYM_KEY], [ORCHESTRATOR_SYM_KEY]


EXECUTE dbo.sp_insertevent, dbo.sp_PublishPolicy, dbo.sp_UnpublishPolicy, dbo.sp_UnpublishPolicyRequest, dbo.fn_GetPolicyInstanceStatus, dbo.fn_NumFailedInstancesPerServer, dbo.fn_NumInstancesPerServer, dbo.fn_NumRunningInstancesPerServer, [Microsoft.SystemCenter.Orchestrator.Cryptography].[Encrypt], [Microsoft.SystemCenter.Orchestrator.Cryptography].[Decrypt], [Microsoft.SystemCenter.Orchestrator.Internal].[RethrowError]

Microsoft.SystemCenter.Orchestrator.Admins

SELECT, INSERT, UPDATE, DELETE, ALTER, CREATE TABLE

SCHEMA::dbo


REFERENCES

dbo.[OBJECTS]


SELECT dbo.[POLICIES_VIEW], GRANT SELECT ON dbo.[POLICY_REQUEST_HISTORY]


CONTROL [ORCHESTRATOR_ASYM_KEY], [ORCHESTRATOR_SYM_KEY]


EXECUTE [Microsoft.SystemCenter.Orchestrator.Cryptography].[CreateOrchestratorKeys], [Microsoft.SystemCenter.Orchestrator.Cryptography].[DropOrchestratorKeys], [Microsoft.SystemCenter.Orchestrator.Cryptography].[Encrypt], [Microsoft.SystemCenter.Orchestrator.Cryptography].[Decrypt], [Microsoft.SystemCenter.Orchestrator.Internal].[RethrowError], dbo.sp_CustomLogCleanup, dbo.sp_GetLogEntriesForDelete_FilterByDays, dbo.sp_GetLogEntriesForDelete_FilterByEntries, dbo.sp_GetLogEntriesForDelete_FilterByEntriesAndDays, dbo.sp_insertevent, dbo.sp_PublishPolicy, dbo.sp_UnpublishPolicy, dbo.sp_UnpublishPolicyRequest, dbo.fn_GetPolicyInstanceStatus, dbo.fn_NumFailedInstancesPerServer, dbo.fn_NumInstancesPerServer,

Page 17

Role Permission Objectdbo.fn_NumRunningInstancesPerServer, [Microsoft.SystemCenter.Orchestrator.Internal].AddUserToRole, [Microsoft.SystemCenter.Orchestrator].[SetPermissions], [Microsoft.SystemCenter.Orchestrator.Internal].[SetProductInfo]

These roles are configured and populated with the required members as presented in Table 4 during the installation process, so there is typically no requirement to work directly with them.Table 4: Orchestrator 2012 Database Roles Membership

Page 18

Account Database Role Management Service


Member of Orchestrator Admins Group


Orchestrator Runbook Service Account


Orchestrator Runbook Server Monitor Service Account


Orchestrator Web Service User Account

Microsoft.SystemCenter. Orchestrator.Operators

Member of Orchestrator Users Group

Microsoft.SystemCenter. Orchestrator.Operators

By default Orchestrator 2012 connections to SQL Server are not secure. The exception to this is when Orchestrator stores or retrieves sensitive data. In this case, Orchestrator creates a secure connection to SQL Server with a self-signed certificate. This certificate does not provide strong security and is susceptible to man-in-the-middle attacks. If more secure SQL server connections are required, Secure Sockets Layer (SSL) connections to SQL Server can be implemented. For further details about configuring SSL on SQL Server refer to “Enable Encrypted Connections to the Database Engine (SQL Server Configuration Manager)” available at http://msdn.microsoft.com/en-au/library/ms191192.aspx.

2.3.2.4 Data EncryptionOrchestrator 2012 provides a code set of encryption and decryption services that are used to generate Orchestrator 2012 platform-encrypted data. These services are used to secure data flagged for encryption in the Orchestrator database as well as decrypt the data to plain-text so it can be used as part of a runbook. These core encryption services are managed by the Orchestrator database and management server. Rights to these services are granted through membership in the Orchestrator Users group or the Orchestrator System group.

Note

Orchestrator 2012 runbooks could contain data encrypted by an external encryption service and used as runbook published data. Orchestrator 2012 would not handle data from such an external system any differently than any other piece of data.

Table 5 details encryption used by the Orchestrator 2012 feature areas:Table 5: Orchestrator 2012 Data Encryption Areas

Feature area

Description

Runbook activities

Any property masked out when one types in the field is an encrypted property. This would include passwords on the Security Credentials tab but can include other properties as well.

Options menu

The Options menu is used to store credentials and other information used to configure integration packs. Properties of connection settings can contain encrypted properties.

Page 19

http://msdn.microsoft.com/en-au/library/ms191192.aspx

Feature area

Description

Variables Variables that have the Encrypted Variable checkbox selected will be encrypted.

Note

Encrypted variables are intended to be used via subscription in properties that require an encrypted value such as a password used in a runbook activity. If an encrypted variable is subscribed to in a non-encrypted field the encrypted value will be provided. The plain-text value is only available when used in an encrypted property.

Orchestrator has a core cryptographic service whose design is based on the Advanced Encryption Standard (AES) using SQL Server cell-level encryption. As such, all encryption and decryption is performed centrally by SQL Server. Encryption keys are centrally managed by SQL Server. Both the SQL Server Service Master Key and the Orchestrator Database Master Key are required to encrypt and decrypt data.

Orchestrator uses cryptography in both the Run Time and Design Time experiences. Runbook authors interact with runbook activities in the Runbook Designer and often these activities will interact with external systems to "discover" property grids, list values, and other properties. Likewise, when a runbook is tested in the Runbook Tester the encrypted data provided in protected fields needs to be decrypted so it can be passed to the target system. Finally, the Runbook Servers need to be able to decrypt encrypted data to allow runbooks to interact with external systems. As such, the database cryptographic services need to be accessed from the Runbook Servers, Runbook Designer and Runbook Tester.

Runbook Servers access the database directly. As such they directly access the crypto services provided by SQL Server. Run Time access to the crypto services provided by SQL Server are limited to members of the Orchestrator System group.

Runbook Designers and the Runbook Tester access the database indirectly through the management server. The management server offers a new service that services requests for encryption/decryption from the Runbook Designer and Runbook Tester. The management server passes through the security context of the runbook author and these credentials are used to access the crypto services. Design Time access to the crypto services provided by SQL Server are limited to members of the Orchestrator Users group.

Access to encrypted data from Orchestrator is managed by the Orchestrator Users group and the Orchestrator Systems group. Members of these two security groups essentially have rich administrative access to Orchestrator including rights to access the core cryptographic services as well as decrypt data stored encrypted in the database.

Introduced in Orchestrator 2012, encrypted variables allow runbook designers to more securely use variables to provide sensitive data to runbook activities. Encrypted variables are used exactly like standard global variables; that is, by means of a subscription. If activity fields that are subscribed to these variables get republished, the variable contents can be exposed on the data bus. Because of this, it is recommended that encrypted variables should be subscribed to only by fields that are not republished.

Page 20

However, if encrypted data must be published on the data bus in order to be sent to another system (for example, a product that runs on a different server), it is recommended to use only secure channel to that product. For example, HP Service Manager supports a secure mode for connection, and products with web interfaces typically allow using the SSL connection (using the Hyper Text Transfer Protocol Secure (HTTPS) protocol).

2.3.2.5 Web server securityFor production Orchestrator 2012 Web Service and Orchestration Console deployments, HTTPS protocol is recommended to secure communication and prevent malformed requests from a man-in-the-middle attack. For further details on securing Orchestrator web server and the Orchestration Console, refer to “How to Configure the Orchestrator Web Server to use HTTPS” available at http://technet.microsoft.com/en-us/library/hh529160.aspx.

By default Orchestrator Web Service calls are not logged. This applies to requests made with the Orchestration Console as well as the Orchestration Integration Toolkit (OIT). The result is that a user can start a job and pass parameters into a runbook with no record of who started the job.

To record all requests to the Orchestrator Web Service, the audit trail logging has to be enabled. For further details about enabling audit trail, refer to “Audit Trail” available at http://technet.microsoft.com/en-us/library/hh488400.aspx.

2.3.3 Network RequirementsCommunication between Orchestrator 2012 components installed on different computers occurs over Transmission Control Protocol/Internet Protocol (TCP/IP). Table 6 presents TCP/IP ports used by Orchestrator 2012.Table 6: Orchestrator 2012 TCP/IP Ports

Source Target Default Port

Configurable

Notes

Runbook Designer

Management Server

135,1024-65535

YES The Runbook Designer communicates with the management server over DCOM. By default, DCOM communicates over port 135 and dynamically allocates a port between 1024 and 65535. For further details about configuring DCOM for a specific port range, refer to “Configuring Microsoft Distributed Transaction Coordinator (DTC) to work through a firewall” available at http://go.microsoft.com/fwlink/p/?LinkID=229219.

Management ServerRunbook ServerWeb Service


1433 YES Specified during the SQL Server installation.

Client Browser

Orchestrator REST-based web service

81 YES Specified during Orchestrator installation. Both ports must be accessible for the Orchestration Console.

Page 21

http://go.microsoft.com/fwlink/p/?LinkID=229219

http://go.microsoft.com/fwlink/p/?LinkID=229219



Source Target Default Port

Configurable

Notes


82

Activities Various targeted computers depending on activity

For information about individual integration packs, refer to ”Integration Packs for System Center 2012 – Orchestrator” available at http://technet.microsoft.com/en-us/library/hh295851.aspx.

2.3.3.1 Windows FirewallWindows Firewall with Advanced Security is enabled by default on all Windows 2008 R2 computers, and blocks all incoming traffic unless it is a response to a request by the host or it is specifically allowed by a firewall rule to allow the traffic.

When Orchestrator Runbook Designer or a Runbook Server is deployed in the environment segmented by firewalls, the following Orchestrator 2012 services should be added to Windows Firewall configuration on the Management Server computer to allow the Runbook Designer and the Runbook Server to communicate with the Management server:Table 7: Orchestrator 2012 Services Added to Windows Firewall


Operating System

File Location

Orchestrator Management Service

64-bit %Program Files%\Microsoft System Center 2012\Orchestrator\Management Server\ManagementService.exe

32-bit %Program Files (x86)%\Microsoft System Center 2012\Orchestrator\Management Server\ManagementService.exe

Orchestrator Remoting Service

64-bit %SystemRoot%\SysWOW64\OrchestratorRemotingService.exe

32-bit %SystemRoot%\System32\OrchestratorRemotingService.exe

Additionally, for some activities such as the Monitoring Activities, if the target computer is outside the firewall, the following firewall rules have to be enabled to allow Windows Management Interface (WMI) communication:

▪ Windows Management Instrumentation (Async-In)▪ Windows Management Instrumentation (DCOM-In)▪ Windows Management Instrumentation (WMI-In)

Page 22

2.4 Backup and Recovery ConsiderationsWith the various roles available on Orchestrator Servers, the process of backing it should be planned carefully. Different roles have different needs and different options on what to back up and how to back it up.

2.4.1 Management ServerThe Management Server does not persist any data. Runbooks and their settings are stored entirely in the Orchestrator database and accessed by the Management Server as required. Management server has a settings.dat file that includes configuration details to connect to the Orchestrator database. This file can be backed up with standard file backups.

2.4.2 Orchestrator DatabaseThis database contains all runbook definitions etc. and is the heart of the Orchestrator installation. If the database is unavailable no Orchestrator 2012 related activity can continue until such a time as it becomes available again. In addition to the regular database backup, the service master key has to be backed up and stored in a secure off-site location (for further details refer to “BACKUP SERVICE MASTER KEY (Transact-SQL)” available at http://msdn.microsoft.com/en-us/library/ms190337.aspx.

In case database server fails, following steps should be carried out:

Reinstall new Operating system (or image in case of virtualization) and restore the encryption keys before setup. Use the same Computer name as original server.

Restore the necessary databases to the server. Restore the service master key.

2.4.3 Monitoring PlanAs any other enterprise application, Orchestrator should also be monitored for any events or performance problems. We recommend monitoring using a monitoring tool such as System Center 2012 Operations Manager.

The Monitoring Pack for System Center 2012 Orchestrator implements discovery and health monitoring of the following Orchestrator roles and components:

Orchestrator Management Server Health. This scenario checks Orchestrator Management servers for service status and critical log file events.

Orchestrator Runbook Server Health. This scenario checks Orchestrator Runbook servers for service status and critical log file events.

Orchestrator Web Components Health. This scenario implements health roll-up for monitored IIS 7 components that represent the Orchestrator Web Components

2.4.4 Services and Event log monitoringOrchestrator consists of several services that need to be monitored for successful operations. Service manager also depends on other products such as SQL Server to function. These services should be running on all times and are given below:

Orchestrator Management ServerPage 23

http://msdn.microsoft.com/en-us/library/ms190337.aspx

o System Center Managemento System Center Data Accesso System Center Configuration

Orchestrator Runbook Server Orchestrator Database Server

o SQL Server

2.4.5 Performance monitoringOrchestrator consumes different types of resources to operation. Although it is important to monitor CPU and memory on, the most important factor that determines performance is disk subsystem performance.

In order to have a healthy deployment following performance counters should be monitored on SQL servers:

LogicalDisk(*)\% Idle Time: If 0 for too long, indicates a disk performance problem LogicalDisk(*)\Avg. Disk Queue Length: < 2 (for each physical disk in the array for

RAID 0+1) LogicalDisk(*)\Avg Disk sec/Read: < 20 ms (preferably < 10 ms) LogicalDisk(*)\Avg Disk sec/Write: < 20 ms (preferably < 10 ms) MSSQL:Buffer Manger\Buffer cache hit ratio: Should be as close to 100% as

possible. < 80% indicates that the computer running SQL Server needs more memory.

2.4.6 Antivirus exceptionsAntivirus solutions running on service manager can be a source of performance or operations problems. In order to minimize the impact following files/folders can be excluded from antivirus scanning. Please refer to product documentation for further information:

Database Serverso SQL database and log folders (*.mdf,*.ldf,*.ndf)o SQL Server backup files (*.bak,*.trn)o Full-text catalog files

Default instance: Program Files\Microsoft SQL Server\MSSQL\FTDATA

Named instance: Program Files\Microsoft SQL Server\MSSQL$instancename\FTDATA

o Trace Fileso SQL audit fileso SQL query files (*.sql)o The directory that holds Analysis Services datao Analysis Services backup files

Following processes should be excluded from virus scanning:

For SQL Server 2012o %ProgramFiles%\Microsoft SQL Server\MSSQL11.<Instance Name>\

MSSQL\Binn\SQLServr.exeo %ProgramFiles%\Microsoft SQL Server\MSSQL11.<Instance Name>\

Reporting Services\ReportServer\Bin\ReportingServicesService.exe

Page 24

o %ProgramFiles%\Microsoft SQL Server\MSSQL11.<Instance Name>\OLAP\Bin\MSMDSrv.exe

For SQL Server 2008 R2o %ProgramFiles%\Microsoft SQL Server\MSSQL10_50.<Instance Name>\

MSSQL\Binn\SQLServr.exeo %ProgramFiles%\Microsoft SQL Server\MSSQL10_50.<Instance Name>\

Reporting Services\ReportServer\Bin\ReportingServicesService.exeo %ProgramFiles%\Microsoft SQL Server\MSSQL10_50.<Instance Name>\

OLAP\Bin\MSMDSrv.exe

2.5 Scalability ConsiderationsThis section describes the Orchestrator 2012 processes that influence performance in a production environment. The sizing and scalability of the Orchestrator 2012 solution depends on processes that occur during runtime, in the web service, and during authoring. While many authoring activities occur outside a production environment, considerations for setting up a production environment to test performance should also include variances, for example, whether special requests by an Orchestrator administrator are supported.This section describes the Orchestrator 2012 processes that influence performance in a production environment. The focus of this procedure lies in identifying processes that occur during runtime, in the web service, and during authoring. While many authoring activities occur outside a production environment, considerations for setting up a production environment to test performance should also include variances, for example, whether special requests by an Orchestrator 2012 administrator are supported.

2.5.1 Runbooks ScalabilityDespite the variance in their design and complexity, runbooks have a simple structure. They perform three operations: they run activities, manage published data, and perform branch logic. The following sections provide more details about these operations.

2.5.1.1 ActivitiesRunbook activities contain two types of code: platform code and domain code. Platform code is built on a framework that is shared between all runbooks. Platform code manages Orchestrator 2012 processes. Domain code refers to the code in a runbook activity that manages processes outside Orchestrator 2012. For example, the Invoke Web Service activity contains platform code to handle processing in Orchestrator 2012, such as publishing data, and domain code specific to invoking a web service.

There is little processing variability between runbooks with activities that run similar platform code. Domain code depends on latency issues external to Orchestrator. Potentially, domain code varies greatly between activities. To understand the domain code dependencies and their impact on runbook performance, the performance of individual activities have to be profiled and used as a baseline for the production environment.

Note

If a runbook contains an activity that references the .NET libraries, the first reference to the .NET libraries takes additional time to initialise. This delay can be as much as 30 seconds. All remaining activities that reference the .NET libraries

Page 25

run immediately. This delay can also occur when a runbook is started on a computer without Internet access, because then Windows cannot verify the Microsoft Authenticode signature for the .NET libraries, and this causes a delay during the initialisation of the activity.

The solution to removing the delay is to deactivate generatePublisherEvidence in PolicyModule.exe or to create a profile for the service account. For further details about removing the delay refer to “How to Configure Runbook Servers to Optimize Performance of .NET Activities“ available at http://technet.microsoft.com/en-us/library/hh674231.aspx.

For each event that a runbook receives, it executes the specified tasks. Events might arrive at different rates. Both events and tasks can vary in terms of resource consumption and length. Thus in addition to a number of runbook definitions, the activities generated by them might influence the number of runbooks that can be simultaneously executed by a Runbook Server. Those factors include:

▪ Anticipated amount of events to be processed for each use case or runbook definition (average, maximum, spike/rush hour patterns).

▪ Amount of required operations/tasks after the event was received per runbook definition.

▪ Resources required by those operations or tasks. With rare exceptions, this factor cannot be estimated during the planning stage, and must be verified during the testing stage for runbook in the specific implementation.

Event rate and number of operations per event together define the total number of activities that a runbook will create over a period of time:

Number Of Activities (Per Runbook Definition Per Day) = [Event Rate (Per Day)] X [Number Of Operations Or Tasks (Per Runbook)]

To simplify planning, especially for a larger number of runbook definitions, they can be grouped by the load level in the following way:

Load level

Average Event Rate Per Runbook (Taken

From External Systems)

Approximate Number Of

Activities Per Runbook Definition

With ~ 20 Tasks

Requires Orchestrator

Database Connections

Configuration?

SQL Server Internal

Parameters Should Be Tweaked?

Per min Per day Per dayLow 1 or less 1500 or less 30000 or less Optional NoMedium around

53000 – 10000

150000 Optional Optional

High 10 – 15 15000 – 22000

400000 Yes Yes

Very high 20 – 40 30000 - 60000

600000 or more Avoid this situation if possible; follow high level recommendations otherwise

When one or more runbook definitions are identified as very high load level, Microsoft recommends to review their design and try to split them into two or more runbook definitions, wherever possible. A very high load level might cause policy delays.

Depending on the level of uncertainty with respect to event load, it’s possible to assess the estimated values similarly to how it was done in the workload estimation.

Page 26


In most cases, low and medium load levels (except when objects are used in the way that generates high utilisation of some specific system resources) are supported with default Orchestrator 2012 configuration, though they might benefit from optimisation. High and very high load levels require careful testing and optimisation of Runbook Server, database connectivity and database server.

2.5.1.2 Published DataRunbooks in Orchestrator 2012 share data between activities. Every activity consumes Published Data that earlier runbook activities created. How an activity uses the published data depends on the domain code. All runbook activities publish a minimum set of run-time parameters called Common Published Data. Domain code can, but is not required to, publish data. The Published Data that the domain code creates is called Activity-Specific Published Data. The data that an activity produces can contain data elements that are single or multi-valued. For example, every activity produces a single record of single-value Common Published Data. Domain code can produce multiple records of single and multi-value data.

Publishing data to the orchestration database is a resource-intensive activity. Runbook performance depends on the amount of data that each activity publishes and the performance and resiliency of the computer that hosts the orchestration database. As part of planning performance requirements, Microsoft recommends to consider the amount of published data required for runbooks and the performance of the computer that hosts the orchestration database.

2.5.1.3 BranchingRunbook activities create a branch if an activity requires data to pass at the same time for two or more activities. When a runbook starts, processing consists of a single thread. When this thread encounters a branch, a thread is created for each branch. Each thread references the published data from all previous activities along the thread. The total number of threads in a runbook depends on the number of branches used in a runbook. Multi-threaded runbooks require more processing power than single threaded runbooks.

As part of assessing runbook performance requirements, Microsoft recommends to consider the number of branches planned in a runbook. Runbooks with lots of branches require more processing power on the Runbook Servers than runbooks that contain no branches.

2.5.1.4 Service Manager ConnectorThe Orchestrator Web Service supports the System Center 2012 Service Manager connector. Service Manager 2012 targets IT customers who serve approximately 50,000 users. Service Manager request-management scenarios assume that each user submits one request per month. This produces a request volume of 2,500 requests per day (200 requests/hour or approximately three requests every minute). Service Manager 2012 uses the Orchestrator Web Service to update the status of activities, requiring support for a like number of status requests. Service Manager 2012 connector also runs published runbooks discovery. The response time to discover any given runbook folder depends on the number of runbooks in the folder.

2.5.2 Runbook Servers ScalabilityWhen considering the Runbook Server workload, the following parameters should be taken into account:

Page 27

▪ Number of runbooks that will run on the Runbook Server and number of activities per runbook.

▪ Hardware, networking and other resources used by Orchestrator 2012 to perform different tasks and operations or used by third-party products that are installed on the same machine.

▪ Database server hardware and software configuration.

By default, a Runbook Server can run 50 Runbooks simultaneously. The physical computer resources and the complexity of the Runbook limit the actual number of runbooks that a Runbook Server can manage.

2.5.2.1 Number of Runbook DefinitionsAssessing the number of runbook definitions allows finding the approximate number of Runbook Servers required for the implementation. If the approximate number of runbook definitions is not yet known, the number of known use cases can be used instead. The number doesn’t have to be exact, and in both cases this number should be padded by a buffer which corresponds to the level of uncertainty about the implementation volume:

Number Of Runbook Definitions = [Number Of Known Runbooks / Use Cases] X [Factor Of Uncertainty]

Any organisation may define an appropriate level of uncertainty for their implementation, or decide to plan for a specific number of runbook definitions using Table 8:Table 8: Orchestrator 2012 Runbooks Planning Uncertainty Factors

Level of uncertainty

Description Factor of Uncertainty

Low Runbooks are already mostly planned/designed 1 – 2Medium Some runbooks were planned, but more runbooks

might be introduced in later stages3 – 4

High Runbooks are not planned yet; only the use cases are known to some extent

5 – 10

Infinite Nothing is planned yet Plan for 50 runbooks

In most cases the results should be rounded up to the larger integer. But depending on other factors, the number can be rounded to lower numbers as well:

▪ If the number of runbook definitions is not significantly higher than the border value. For example, for 51 runbook definitions, it can be assumed a minimal requirement of one Runbook Server.

▪ If some runbooks do not contain monitors and will only run periodically (e.g. be invoked manually).

Based on the number of runbook definitions, the initial number of required Runbook Servers can be planned as:

Number Of Required Runbook Servers = Round Up To Integer (Number Of Runbook Definitions / 501)

Note

The calculated number does not include availability, load balancing, performance and other considerations. It merely represents the minimal required number of Runbook Servers. Depending on the runbooks design, the hardware and

1 Maximum number of simultaneously running runbooks per Runbook Server.Page 28

database, the maximum number of simultaneously running runbooks per Runbook Server can be below or above 50. Microsoft recommends running performance monitoring on the Runbook Servers to create a realistic baseline.

These factors should be taken into consideration when estimating the workload and planning the capacity of a Runbook Server.

2.5.2.2 Scaling Up vs. Scaling OutIn general the decision between scaling-up and scaling-out needs to be assessed for each of the following reasons:

▪ During the initial workload planning▪ To avoid loading resources to their full capacity▪ To allow some room for occasional spikes▪ To plan for growth over time

Scaling up a Runbook Server can resolve some potential hardware bottlenecks and speed up execution of runbooks. The improvements that can be achieved are limited and during execution the runbook may use significantly more resources. In those cases scaling out (adding Runbook Servers) will be better.

Table 9 presents typical Runbook Server performance bottlenecks, and recommendations for selecting scaling-up or scaling-out approach.Table 9: Runbook Server Scaling Approach

Bottleneck

When each solution should be used

Scale-Up Scale-OutCentral Processing Unit (CPU)

If a single core CPU is used, or the processors are outdated or low capacity.May also help in situations where issues are related to specific runbook (validated that stopping the runbook frees CPU resources).

If the number of runbook definitions causes high (especially Kernel) CPU usage, and reducing the number of Runbook by a certain percentage eliminates the issue.

Random Access Memory (RAM)

If the memory is shared with third-party applications and some runbooks take more resources to run.

If each runbook requires a lot of memory to run and working memory set for these runbook exceeds the Runbook Server memory.

2.5.3 Orchestrator 2012 Performance Tuning▪ Changing the configuration for each of the factors listed in

requires appropriate testing in an environment that is as close as possible to the production environment; the configuration should not be changed unless:

▪ Orchestrator 2012 performance problems are related to the number of running runbook definitions.

▪ The total number of runbooks exceeds the current limit by just a few, so it seems unnecessary to deploy another Runbook Server (e.g. the plan is to deploy 51 runbooks per a Runbook Server).

▪ There are other limitations (e.g. 3rd party product licensing, hardware, etc.) that require the deployment of larger number of runbooks per a Runbook Server.

Page 29

Table 10: Orchestrator 2012 Performance TuningFactor Likelihood Symptoms Configurability

Orchestrator configuration parameter (Maximum Running Runbooks)

High 50 (or other static number) runbook definitions are published and run without any problem (and the same number of PolicyModule.exe processes can be seen in Task Manager). However, when publishing more runbook definitions, they do not run.

Maximum running policies parameter is configurable. For further details refer to “How to Configure Runbook Throttling” available at http://technet.microsoft.com/en-us/library/hh420378.aspx.

Database Connections

High Runbook execution is slow. There seems to be big gaps between the end time of one Activity and the start time of the following Activity.

Maximum number of connections is configurable.

Internal Policy Module Logic

High Runbook execution is slow. But the runbook itself contains a number of tasks that start and complete relatively quickly.

Thread-regulating mechanism of Policy Module can be optimised.

CPU Medium Runbook execution is slow. Further inspection shows that the entire system is slow and unresponsive. Performance Monitor or Task Manager show high CPU usage, which if analysed appears to be a high Kernel CPU time.In the most severe cases, when CPU usage hits 100%, the Runbook Server sends an Orchestrator Platform Event notifying the user of the issue.This problem is especially visible on virtual machines or machines with low capacity CPUs.

Execution of additional runbooks requires creation of more threads, which causes more context switches between them, and as a result higher kernel CPU usage. This cannot be configured. Allocating more virtual CPUs or adding more CPUs will help to balance the CPU usage vs. number of runbooks.

2.5.4 Resource Availability RequirementsEach runbook is unique with respect to the resources it requires to run. Identifying the required resources should be done during the runbook testing. Most typical Orchestrator 2012 resource availability requirements are presented in Table 11.Table 11: Orchestrator 2012 Resource Availability Requirements

Orchestrator 2012

Component

Resource Priority What it affects Recommended value

Runbook Server CPU Very High ▪ Number of runbook definitions that can be published on the machine.

▪ Number of activities that can run concurrently.

▪ Runbook performance (Event Throughput, Event Process Time).

Quad Core CPU or better

RAM High ▪ Number of runbook definitions that can be published on the machine.

▪ Number of activities that

At least 4GB

Page 30



Orchestrator 2012

Component

Resource Priority What it affects Recommended value

can run concurrently.Hard Disk Low ▪ Runbook performance if

runbook activities perform file system operations

Small Computer System Interface (SCSI) 10K

Network Very High ▪ Runbook performance At least 10 Mbps

Database Network Very High ▪ Runbook performance At least 10 Mbps

2.5.5 Database SizingOrchestrator 2012 database sizing and performance is another key factor of performance of Orchestrator 2012. Runbook Servers, Management Server, Orchestration Web Service, and Orchestration Console are all dependent on the database for their operation.

An Orchestrator 2012 database contains two types of data:

▪ Configuration data. In general, configuration data is of little concern when considered in the context of database growth since the storage requirements for this type of data are small relative to the minimum system requirements for Orchestrator 2012.

▪ Log data. Orchestrator 2012 creates a variety of types of log data, all of which can be viewed and managed in the Runbook Designer. Additionally, the Runbook Designer provides a Log Purge feature that allows log data to be trimmed. Orchestrator 2012 log data can be purged per runbook or for all runbooks. The logs can be purged "on demand" or on a scheduled basis. By default the Log Purge job runs at 1:00 AM (local time on the Management Server) every day. It purges all but the last 500 log entries per runbook. Hence if an Orchestrator 2012 deployment has 20 runbooks the default Log Purge job would keep the last 500 log entries per runbook for a maximum of 1000 log entries for the database for all runbooks.Audit History logs are a special case since these logs are not purged with the Log Purge job. The audit history feature tracks the changes made to a runbook in the Runbook Designer and cannot be deleted. The only way to delete Audit History logs is to delete the runbook associated with them. However, data volumes for the Audit History logs for a given runbook are generally low and even a large deployment of Orchestrator will not require more storage than the minimum system requirements.

When runbooks run on a Runbook Server they log data to the Orchestrator database. Every activity in a runbook logs data to the Orchestrator database. There are three logging levels:

▪ Default. This logging level logs allows Orchestrator 2012 to log very basic data for each activity and cannot be disabled.

▪ Common Published Data. This logging level allows Orchestrator 2012 to log an expanded set of log data including Activity Name, Activity Type, Activity ID, Activity End Time, Activity Duration, Previous Activity, Previous Activity Name, and Time Published Data.

Page 31

▪ Activity-specific published data. This logging level allows Orchestrator 2012 to log data that is specific to a particular activity. This logging is intended for debugging and non-production use and can greatly increase the amount of logged data.

Database growth can be estimated by counting the number of activities in a runbook. When counting it is important to take into consideration that features like "looping" can cause a single activity to run multiple times and each runbook activity execution should be counted individually.

To estimate Orchestrator database growth, the calculated number of activities should be multiplied by the log record size as presented in Table 12. The result will be the number of bytes added to the SQL Server database files per invocation of the runbook. Table 12: Orchestrator 2012 Log Record Size

Logging Configuration DB Growth (bytes) per Activity Instance

Performance Factor

Recommended for Production

Default logging 524 1 YesLogging of common published data

6082 2.5 Limited use with planning

Logging of activity-specific published data

6082 + data logged by activity

Greater than 2.5 No

Page 32

2.6 High Availability ConsiderationsThis section describes the Orchestrator 2012 deployment options to improve High Availability.

2.6.1 Management serverAn Orchestrator 2012 deployment is limited to one Management Server. A Management Server does not have to be available for Runbook Servers or runbooks to function. If the Management Server is not available:

▪ Runbook Designer cannot be used to publish runbooks or start, monitor, or stop runbooks.

▪ Orchestration Console can be used to start, monitor, and stop runbooks.

2.6.2 Orchestrator databaseAn Orchestrator 2012 database is hosted on SQL Server 2008 or later. For high availability, Microsoft recommends deploying the Orchestrator database on a SQL Server cluster with a minimum of two nodes.

2.6.3 Orchestrator Web ServiceThe Orchestrator Web Service must be installed on a server that is running IIS. Orchestrator Web Service is required to start, monitor, or stop runbooks using Orchestration Console. It does not have to be available for Runbook Servers or runbooks to function. Microsoft recommends installing Orchestrator Web Service on multiple IIS servers configured for load balancing to provide high availability and additional capacity.

2.6.4 Orchestration ConsoleThe Orchestration Console must be installed on a server that is running Internet IIS. Orchestration Console can be used to start, monitor, or stop runbooks. Microsoft recommends installing Orchestration Console on multiple IIS servers configured for load balancing to provide high availability and additional capacity.

2.6.5 Runbook ServersRunbook Servers are required for runbooks to function. Runbook Servers are not designed to run on a computer configured as a cluster node. For high availability, Microsoft recommends to deploy at least two Runbook Servers. If the primary Runbook Server for a runbook is unavailable, the runbook can run on another server.

Page 33

3 Orchestrator 2012 Architecture Patterns

This section presents typical design patterns for Orchestrator 2012 deployments.

3.1.1 Design Pattern #1 – Single Server Orchestrator 2012 Infrastructure

This Orchestrator 2012 design pattern is for the basic Orchestrator 2012 deployment. All Orchestrator 2012 components are deployed on a single physical or on a virtual server. This server hosts the Orchestrator Database, Management Server, Runbook Server, Runbook Designer and Orchestration Console. This design pattern presented in Figure 1, though fully functional, does not provide any high availability and therefore, it is not recommended for production but for PoC or Development environments only.

Figure 1: Single Server Orchestrator 2012 Infrastructure

3.1.2 Design Pattern #2 – High Availability Orchestrator 2012 Infrastructure

A High Availability (HA) Orchestrator 2012 design pattern presented in Figure 2 is the most common and widely used one. Critical features of Orchestrator 2012 are configured to enable HA namely, the Orchestrator Database, Runbook Servers and the Orchestrator Web Service.

Page 34

At a minimum, two Runbook Servers are deployed for failover and load balancing. The number of Runbook Servers can be scaled out to satisfy additional business and process requirements.

The Orchestrator Database where configuration information, runbooks, and logs are stored is configured on a two node SQL Server cluster.

The operator uses the Orchestration Console and the Orchestrator Web Service. The Orchestration Console and the Orchestrator Web Service depend on the performance of the orchestration database and the IIS server that hosts the Orchestrator Web Service. Orchestrator Web Service is installed on load balanced servers running IIS to provide HA and additional capacity.

Figure 2: High Availability Orchestrator 2012 Infrastructure

3.1.3 Design Pattern #3 – High Availability and Multi-Site Orchestrator 2012 Infrastructure

The Orchestrator design pattern presented in Figure 3 is essentially an expansion of Design Pattern #2 specifically intended for managing devices and/or integrating systems from the second data centre where slow network and security is a factor.

On a slow network where latency is greater than 30ms and packet loss is certain, configuring a Runbook Server across a Wide Area Network (WAN) for the purpose of managing devices and/or integrating with systems is not recommended. The best approach is to configure a dedicated Orchestrator infrastructure in each data centre and build runbooks that pass data between the Orchestrator Web Services. The following guidelines provide options in an Orchestrator deployment to improve availability (and performance):

Page 35

Component Planning ConsiderationManagement Server Orchestrator is limited to one Management Server

The Management Server is not required for the Runbook Servers or Runbooks to function. No HA planning is required.

If the management server is not available, you cannot connect the Runbook Designer to publish runbooks.

Orchestrator Database

For high availability, you can deploy the Orchestrator database on a Microsoft SQL Server cluster with a minimum of two nodes.

Orchestrator Web Service

The Orchestrator web service must be installed on a server that is running Internet Information Services (IIS).

The Orchestrator web service does not have to be available for runbook servers or runbooks to function. If the Orchestrator web service is not available, you cannot run the Orchestration web console to start, monitor, or stop runbooks.

You can install the web service on multiple IIS servers configured for load balancing to provide high availability and additional capacity.

Runbook Server Configure at least two Runbook Servers for HA. If the Primary Runbook Server for a Runbook is unavailable, the next Standby Runbook Server will then run the Runbook.

Runbook Servers do not support computers that are configured as a Cluster Node

Runbooks For additional high availability multiple Runbook servers can be deployed. Runbook servers can run independently of Management servers.

By default, each Runbook server is configured to simultaneously run a maximum of 50 runbooks.

Consider the resource requirements of the Runbooks on a particular server. If required, you can change the default 50 Runbook number by using the Runbook Server Runbook Throttling tool.

If the server has a number of runbooks with high resource requirements, you might run fewer runbooks simultaneously on the runbook server.

If they are simple runbooks with minimal requirements, you might consider increasing the number of simultaneously run runbooks.

Page 36

Figure 3: High Availability and Multi-Site Orchestrator 2012 Infrastructure

3.1.4 Design DecisionTypically you will need to think about the following:

How many datacenter will you be integrating with? Does your network meet the required latency value? How many Runbooks will Orchestrator be processing? Will Orchestrator be integrating with external systems (SCOM, SCSM, 3rd-Party)?

Page 37

How many event is Orchestrator expected to process? Will the Runbooks require Failover capability?

Page 38

Header Corp Name MS (External Corporation Name)

4 Evaluate Solution Requirements

The following summarises guidance for determining your solution requirements as it applies to Orchestrator.

Task Planning Considerations1. Define the Scope of the Project

Identify the business requirements Define the processes you want to automate and identify all

applications, services, services, servers and manual processes associated with the tasks you want to perform

Prioritize requirements based on customer’s business impact

2. Identify Tasks Map the processes you intend to automate to individual steps. Create a swimlane diagram to simplify the task of authoring Runbooks.

3. Define Individual Workloads

Determine execution frequency of the process you want to automate.

A Runbook that runs one time per day uses significantly fewer resource than a continuous running Runbook. Consider both the workload on the Orchestrator system and the automated process.

Consider how much logging of Published Data is required in each Runbooks. As logging increases, network traffic and load on the Orchestration Database server increases.

4. Determine Total Jobs Running

Calculate the total number of jobs that could be running at any point in time. Consider the maximum workload into account in your system design.

5. Identify Required Integration Packs

Determine the Integration Packs required for the automation process.

If no out-of-the-box integration pack is available, determine integration surface(s) that the external system offers i.e. Command Line Interface, PowerShell, web service, WMI, SSH, database, API, scripts etc..

6. Determine the Security Model

Determine if the Runbooks servers and resources will be located in more than one Active Directory forest. Is there a cross-domain trust?

Are there Operations Manager gateways that require certificates?

Review the current security requirements for your environment to identify permission and certificate requirements

7. Design Runbook Server requirements

Do the Runbook Server require to be across Wide Area Network (WAN) links and trust bounderies? If so, determine gateway server placement in relation to the Orchestrator Database and Runbook Servers.

Consider the a fast connection between the Runbook Server to the Orchestration Database with very low latency

Page 1


Task Planning Considerations(Recommended: <10ms but the requirement is <30ms)

8. Fault tolerance Determine the level of fault tolerance for your Orchestrator Deployment. Depending on your requirements, you can design your Orchestrator environment to be highly available in the case of a single failure such as SQL cluster, multiple Runbook Servers and Error Handling and Redundancy design in Runbooks.

9. Resource requirements

Determine the requirements for your Orchestrator deployment, and any additional load that increased requirements on processes impacted by automation create. Do you have adequate runbook servers for the number of runbooks that can be running at a given time? Is the Orchestrator database the appropriate size to handle all requests and log Published Data?

10. Service and operations requirements

Identify all requirements for your environment. Include any data consolidation strategies and requirements for cross-management group, data-retention, data-warehouse size, or fault-tolerance.

11. Network Determine if additional bandwidth is required to support the increased traffic the runbook servers and the Orchestrator database generate. Do you have to change any network port settings to accommodate the Orchestrator web service?

12. Authoring Determine where and how authoring of runbooks is carried out. Authoring of runbooks typically occurs on computers isolated from production. However, your business requirements might include the requirement to author runbooks when they were not planned.

13. Test Environment If you are authoring in isolation from your production environment, identify the necessary resources to build and test new runbooks.

14. Pre-Production Environment

It is prudent to deploy high impact runbooks in a pre-production environment before introducing the runbook into a production environment. Pre-production environments should closely approximate the full-scale production environment.

Page 2


5 Appendix A – Sample Use Cases

The following use cases are illustrated in this section:

Sample Use Cases DescriptionAlert triage and remediation

Automating tier 1 actions to monitoring alerts

Maintenance tasks and advanced patching

Cluster patching Stop/start servers or services in the right sequence Automate SQL Server maintenance tasks Maintenance windows

Service catalog and change management, provisioning and Private Cloud integration

Server and VM provisioning => foundation of a private cloud

Integration with a service catalog

Dynamic resource allocation

e.g. depending on load or calendar

Optimize processes through cross-technology integration

Integrating monitoring and ticketing in the different automated processes in a consistent manner

Migration and interoperability between different solutions (e.g. new ticketing system)

Line of business scenarios

New employee onboarding, password resets, file transfers, etc.

5.1 Use Case 1: Alert Remediation5.1.1 Free Space RemediationThis runbook waits for new free space issues, and dynamically reaches for the affected machine and disk to try to resolve the issue. Only if this fails, a ticket is created.

Page 3


Page 4


5.1.2 Detect and repair secure channelThis runbook detects a secure channel issue, and tries to reset it using the NETDOM command, possibly having to start the server using iLO if needed. If anything fails, a ticker is created.

Page 5


5.2 Use Case 2: Maintenance5.2.1 Advanced patching (patching machines in order,

with pre- and post- checks)First runbook, handling the patching for a specific machine and a specific patch, disabling monitoring and checking services are fine after patching through Configuration Manager:

Page 6


Main runbook, calling the child patching runbook for a series of machines in the right order:

Page 7


5.2.2 SQL Server maintenance tasksThis runbook takes a server name as a parameter, discovers the databases hosted on this server, and for each of them determines the state of indexes. If a reorganization is needed, the runbook executes it right away (with notification of the action and its outcomes). If a rebuild is needed, a change request is created in the appropriate change management system. The scripts used for the reorganization and rebuild could very well be scripts your SQL administrators already use today.

This second runbook executes the rebuild of an index, once it has been approved.

Page 8


5.3 Use Case 3: Change management and provisioning

Tying into change management and provisioning processes can be done in many different manners. Runbooks can be called externally (web service or command line tools), or could be waiting for a status in a change request from the ITSM solution. The first example below shows these different approaches, and the rest of the examples for Use Case 3 highlight how Service Manager and Orchestrator can be tied together to simplify these scenarios.

5.3.1 Provisioning : Provision virtual machinesHere is a runbook handling the VM creation. It would take the VM Name, memory, CPU, etc. parameters, as well as any criteria you would like to expose. See how the actual application deployments could be linked after the VM creation, if your processes and/or tools require it.

The following runbook calls the previous one, once a change request has been approved in an ITSM solution:

Page 9


5.3.2 Provisioning: Application deployment

This is another form to surface another provisioning scenario. The form would look like this in the Service Manager web portal:

Here is the runbook being called (with or without approval in Service Manager)

Page 10


Child runbook called by the previous runbook:

5.3.3 Administration: Put machines in Operations Manager maintenance mode

Here also, the Service Catalog integration between Service Manager and Orchestrator enables a dynamic and very simple implementation. The form in Service Manager service

Page 11


catalog lists the available servers, and the runbook will be called seamlessly when submitting the form:

Remember that this is very flexible. A runbook could be waiting for a file to be dropped in a folder, with a list of servers to put in maintenance mode. It would just call the previous runbook when such a file is detected and parsed.

Page 12


5.4 Use Case 4: Cross-Technology IntegrationThis section is highlighted using a connection between System Center Operations Manager and IBM NetCool (with NetCool being the “manager of managers”). A similar logic would be used with another manager of managers like BMC Event Manager or ProactiveNet, or for ticketing (creating tickets from monitoring alerts, in BMC Remedy or another tool for example).

5.4.1 Connecting System Center Operations Manager to IBM NetCool (simple example)

This simple example would just wait for alerts in Operations Manager, create a new alert in NetCool (dynamically filling the field) and “tag” the original alert in Operations Manager (updates a custom field with the ID of the NetCool alert that was just created)

The reason for the “tagging” is highlighted in this second runbook : When an alert is closed in NetCool, you might want to close the corresponding original alert in Operations Manager, and the “tagging” enables that mapping):

5.4.2 Connecting System Center Operations Manager to IBM NetCool (advanced example)

The same logic is used here, but with a temporary database to handle alert storm, and also with redundancy when trying to “forward” alerts to NetCool:

Page 13


Page 14


5.5 Use Case 5: Dynamic Resource Allocation5.5.1 Web Farm demo: Scaling out a service to handle

additional loadMethod 1 : A single runbook detects the condition (too many connections, or a performance issue in a synthetic transaction), and then provisions a new virtual machines and adds it to the application’s load balancer.

Method 2 : The same process is now split in two runbooks.

The first runbook detects the condition abd creates a change request:

The second runbook waits for the change request to be approved and actually does the work:

Page 15


Page 16


5.6 Use Case 6: Line of business and other5.6.1 New hire onboardingThis runbook monitors new users provisioned in Active Directory (in this case also triggering the synchronization from Forefront Identify Manager as the identity management solution - Your process may requires integrating into another identity solution or just wait for new users in Active Directory)

This is the runbook being called when a new “contractor” user has been detected:

Page 17


Page 18

Documents

Datacenter Services Datacenter Automation Design … · Web viewDesign Architecture Datacenter Automation Table of Contents 1Introduction3 1.1Background3