Configuring Service Manager for Performance and Scale - MMS2013 Presentation

For more information: www.concurrency.com

http://www.concurrency.com/

Configuring Service Manager for Performance and ScaleNathan LasnoskiInfrastructure Architect, Microsoft MVPConcurrency

SD-B312For more information: www.concurrency.com


Experiences with Service Manager and OrchestratorVisibilityUnderstand what you deliver, how its configured, and how you support it

… One configuration database for 20,000+ servers, 1000+ SQL databases, and work items

Process ImprovementOptimize or re-shape processes for enterprise organizations

… 1,000,000+ work instances, each of which saved on average of 15 minutes of IT time

Automation (Client or Datacenter)Take manual processes and replace them with automation

… 200,000+ user and asset automations, each of which saving an hour of time



What does performance mean?Console OperationsIncidents, problems, and changes open and close quicklyRead operations provide content quickly to IT users

Portal PerformanceNo or minimal “please wait” on the portalSubmissions complete without failure

Automation PerformanceAutomation execution time. Application deployment in 1 vs. 5 minutes makes a difference

Report PerformanceReport data is delivered quickly and user expectations are setFor more information: www.concurrency.com


What does scale mean?Massive User CountsCustomers with 200,000+ Users

Huge DatacentersDatacenters with 20,000+ Servers

GeographyGeographically dispersed users“Follow-the-sun”

Quantity of RequestsMillions of incidents, requests, and changes



Agenda

Architecture for “Large” Scale Deployments (50,000+)SQL Server Configuration TipsUser Experience OptimizationWorkflow and Connector OptimizationData Warehouse Optimization



Deployment Architecture for Large Scale



Assumptions for Large Scale DeploymentsGeneral AssumptionsGeographically dispersed IT analysts and end usersConstant availability without large maintenance windowsFailover scenarios for alternate siteNeed for offloading functions to scale as needs arise

General AssumptionsLarge scale of users (50,000+ users)Large scale of work items (ex: millions of incidents per year)Large scale of configuration items (hundreds of thousands of computers, software, servers, etc.)



Hardware Recommendations for Large ScaleManagement Servers4 – 8 core16 GB RAM(console and workflow servers)

SQL Servers16 – 24 core32+ GB RAMHigh performance disk

DW SQL Servers16 – 24 core32+ GB RAMHigh performance disk



Hardware Recommendations for Large ScaleSQL Servers and DW SQL Servers8 – 16 core32+ GB RAMHigh performance disk

SharePoint and Web Content Servers4 - 8 core16 – 32 GB RAM80 GB HD

Orchestrator Management and Runbook Servers4 – 8 core16 – 32 GB RAMHigh performance disk



Example Architecture for 50,000+ UsersManagement ServersDedicated workflow mgmt. serverDedicated console servers with load balancerDedicated mgmt. servers for Orchestrator

SQLClustering for all componentsAlwaysOn for operations databases

OrchestratorRunbook Servers

OrchestratorManagement

SM Management Servers

SM Connection Servers

SM Portal

SharePoint / WCS

SM Orchestrator Server

SM ORCH Target Server

Service Manager DBSQL 2012

Datawarehouse DBSQL 2012

Windows Server 201224 – 48 vCPUs, 32 – 192 GB RAM

80 GB OS, 600 GB Data


60 GB OS, 1 TB+ Data2-4TB of disk for 1095 days

DR: RTO 24 hours

Windows Server 20122 vCPUs, 8 GB RAM

80 GB OSScale number as

workloads demandRTO: 30 min?


60 GB OSScale as workload demands

RTO: 15 min

Windows Server 20124 vCPUs, 8 – 16 GB RAM

60 GB OSDR: RTO 24 hours

Windows Server 20124 – 8 vCPUs, 8 - 16 GB RAM

60 GB OS80-100 concurrent sessions/server

DR: RTO 30 minutes


First is failover for PrimaryScale as workloads demand

RTO: Globally Coninuously Available?


60 GB OSRTO: Not clear


First is failover for PrimaryScale as workloads demand

RTO: Globally Coninuously Available?


80 GB OSScale number as

workloads demandRTO: 30 min?

Service Manager DBSQL 2012


80 GB OS, 600 GB Data



RTO: 15 min



RTO: 15 min

Active DirectorySCOM, SCCM,Other Systems



SQL Server Configuration



SQL Server ConfigurationTempDBHigh performance LUN, split IO to log volumeMultiple TempDBs (1 per 2 cores), but only one log, normalized sizeTempDB performance is critical to Service Manager performance

ServiceManager DatabaseHigh performance LUN, split IO to log volumeTransaction log set to BASIC by defaultTest performance of your disk:“sqlio -kW -t2 –s30 -o1 -frandom -b64 -BH -LS c:\Testfile.dat “

RAM AllocationConfigure for 2 GB less than the total RAM on the SQL server



SQL Server ConfigurationSQL BrokerDrastic performance difference with workflowsValidate SQL Broker set to 1 vs. 0

Checking SQL BrokerSELECT is_broker_enabled FROM sys.databases WHERE name = 'ServiceManager‘

Setting SQL Broker (change window only)ALTER DATABASE ServiceManager SET SINGLE_USER WITH ROLLBACK IMMEDIATE ALTER DATABASE ServiceManager SET ENABLE_BROKERALTER DATABASE ServiceManager SET MULTI_USER



Max Degree of ParallelismDefines parallel processing rules in SQLDefault is “0” and allows a single query to use all processor coresBetter results in our environments with “1” to “4”.

Learn more and set as required. Test, as your experience may vary.http://msdn.microsoft.com/en-us/library/ms181007(v=SQL.105).aspx

SELECT name, value FROM sys.configurations WHERE name = ‘max degree of parallelism’

Validate Read Committed Snapshot Isolation (RCSI)Validating isolation levelsSELECT name, is_read_committed_snapshot_on FROMsys.databases where name = 'ServiceManager‘

ALTER DATABASE ServiceManager SET READ_COMMITTED_SNAPSHOT ON


http://msdn.microsoft.com/en-us/library/ms181007(v=SQL.105).aspx

http://msdn.microsoft.com/en-us/library/ms181007(v=SQL.105).aspx


SQL Performance for ServiceManager DBRAM Usage Page File % Usage should be less than 1%Memory – Available Mbytes should not be below 100mb (indicates starved OS)

Disk PerformanceAve. Disk Sec/Read and /Write should be less than 20 msec typically (some spikes ok)Ave. Disk Queue Length can indicate disk IO issues, though less valuable than Ave. Disk Sec/Read

CPU PerformanceValidate SQL service and CPU usage.

Change Grooming SettingsMinimize Retention Settings – ideally less than a week or twoCreate custom grooming rules



SQL Logs and Recovery ModelCheck your log size and expansionDoes your log write, clear, and have expansion space?

Check your recovery modelUse SIMPLE, or configure a transaction log backupSuggestion is to use SIMPLE



High AvailabilitySQL 2012 AlwaysOn High availability with AlwaysOnhttp://blogs.technet.com/b/babulalghule/archive/2013/02/17/how-to-install-service-manager-2012-sp1-with-a-sql-2012-alwayson-availability-groups.aspx


http://blogs.technet.com/b/babulalghule/archive/2013/02/17/how-to-install-service-manager-2012-sp1-with-a-sql-2012-alwayson-availability-groups.aspx




DEMO: Checking your SQL Settings



User Experience Optimization



Pick the Right User ExperienceSelf-Service PortalEnd user self-service interactions or IT interactionsService Requests are excellent for “low / no training”, interactions

ConsoleHigh bandwidth, low latency scenariosRemoteApp necessary for high bandwidth scenariosLess than 100 msec or less150 – 200 msec has 40% degradation

Web Consoles (GridPro and Cireson)Excellent for low bandwidth and/or latency scenariosSome match with console experience (GridPro)



Console OptimizationConsole OptimizationSlower performance when maximizedApply SP1 for console memory leak

Minimize Quantity of ViewsDelete views not needed. More views slows console load time.

Use Search vs. ViewsUse searching vs. views to find data quickly

Configure the Global Operators GroupLimits uses selectable when assigning work items (drastic performance improvement)



View OptimizationBasic ViewsBring back only information from one classThe fastest view to create

Type Projections / Combination ClassesCombine classes to bring back information that includes relationshipsUse the smallest projection possibleDownload type projections for Incident, Problem, Change, and Service Request

Advanced ClassesDo not use the (Advanced) class in views. Use a type projection. The (Advanced) class can be used in searches that return small quantitieshttp://blogs.technet.com/b/servicemanager/archive/2010/12/02/faq-why-is-my-custom-incident-view-so-slow.aspxhttp://blogs.technet.com/b/servicemanager/archive/2011/09/19/new-change-request-type-projections-management-pack.aspx

http://blogs.technet.com/b/servicemanager/archive/2010/12/02/faq-why-is-my-custom-incident-view-so-slow.aspx

http://blogs.technet.com/b/servicemanager/archive/2010/12/02/faq-why-is-my-custom-incident-view-so-slow.aspx

http://blogs.technet.com/b/servicemanager/archive/2011/09/19/new-change-request-type-projections-management-pack.aspx

http://blogs.technet.com/b/servicemanager/archive/2011/09/19/new-change-request-type-projections-management-pack.aspx

Scoped User Roles vs. Non-ScopedScopedScoped user roles facilitate filtering based on groups and queuesMinimize use of scoped user roles due to additional table join in the database



Portal PerformancePortal and IconsMore icons means longer load timeConsider using a “start page” to point to Service Offerings

Scope Request OfferingsUse user roles to scope access to service offerings and request offeringsThe more service offerings and request offerings, the longer the load

Disable App Pool Recycling for SharePoint and SSPDefault is nightly recycling which causes slow initial performanceMake sure not to set recycling to high memory usagehttp://technet.microsoft.com/en-us/library/cc753179(v=WS.10).aspx


http://technet.microsoft.com/en-us/library/cc753179(v=WS.10).aspx

http://technet.microsoft.com/en-us/library/cc753179(v=WS.10).aspx


Portal PerformanceUse Searching in Request OfferingsDon’t increase the size of the query resultsUse pre-search filters to allow searching of hundreds of thousands of CIs in secondshttp://blog.concurrency.com/infrastructure/service-manager-request-query-result-filtering/

Use Known InformationUse previous questions and default values to filter queriesUse known information (such as user name and relationship to computer)Avoid MP ENUM questions as they cannot be used in post-selection filters


http://blog.concurrency.com/infrastructure/service-manager-request-query-result-filtering/

http://blog.concurrency.com/infrastructure/service-manager-request-query-result-filtering/


DEMO: Pre-Search Filters



Server and Workflow Optimization



Dedicated ServersUse a Dedicated Workflow ServerAlways the first server in the management group by default

Use a Dedicated Orchestrator Target ServerAny server in the management groupUsed for any Orchestrator interactionUse a dedicated account for Orchestrator automations



Workflow TweaksImplied Permissions WorkflowConsider disabling implied permissionsUtilize direct permissions with un-scoped operator role

New Priority CalculationDisable or use priority calculation ruleDisable Incident_Adjust_PriorityAndResolutionTime_Custom_Rule.Add if using SLOs

First Assigned RelationshipDisable “WorkItem_SetFirstAssignedTo_RelationshipAdd_Rule” if not used

SLO ApplicationLimit to what needed, as SLO application is performance heavyOrchestrator can apply SLOs based on complex rule matrixes.

Group Calculation IntervalMinimize Group Calculation IntervalDefault is every 30 seconds, 6000000 = 10 minutes

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\System Center\2010\Common\GroupCalcPollingIntervalMilliseconds



Troubleshooting Performance DelaysHow to find a problematic workflowUse queries in blog post:http://blogs.technet.com/b/servicemanager/archive/2013/01/14/troubleshooting-workflow-performance-and-delays.aspx

Setting the WatermarkArtificially moving the watermark forward… only last resort


http://blogs.technet.com/b/servicemanager/archive/2013/01/14/troubleshooting-workflow-performance-and-delays.aspx




DEMO: SM Checking Workflows



OrchestratorWhen to Use OrchestratorComplex workflows, ex. Incident categorization, routing, notificationsUse vs. many individual Service Manager workflowsHigh capacity automation performancePerformance distribution of workflow processes

Incident CategorizationMonitor Object or “Get” initiationComplex routing and translation



OrchestratorUsing Orchestrator with NotificationsComplex logic and offloading notificationshttp://blog.concurrency.com/infrastructure/scsm-notifications-with-orchestrator-roll-up/


http://blog.concurrency.com/infrastructure/scsm-notifications-with-orchestrator-roll-up/

http://blog.concurrency.com/infrastructure/scsm-notifications-with-orchestrator-roll-up/


OrchestratorRunbook Initiation Option 1Service Manager workflow initiates runbook activityEasier and out of box

Runbook Initiation Option 2Orchestrator initiator runbook to find active runbooksDrastically faster than Service Manager OOB initiation (about 4 times faster!!)



DEMO: Using Orchestrator for Workflow



Connector ConfigurationConstrain SyncOnly sync those groups, users, and configuration items you need

Null is BadDo not sync “null” values on connectors. Connectors fight with each other.



Connector ConfigurationDCM SynchronizationDisable DCM workflowhttp://blogs.technet.com/b/mihai/archive/2012/11/30/configuration-manager-connector-s-dcm-rule-can-cause-massive-performance-issues-in-service-manager.aspx

Using the DNS Trick for Active DirectoryDNS trick (fix in UR2)

Sync Custom VariablesSync variables from Active Directory not covered by connector, or fasterhttp://blog.concurrency.com/featured-post/how-to-sync-other-properties-from-active-directory-to-service-manager-using-orchestrator/ (password last set variable)


http://blogs.technet.com/b/mihai/archive/2012/11/30/configuration-manager-connector-s-dcm-rule-can-cause-massive-performance-issues-in-service-manager.aspx



http://blog.concurrency.com/featured-post/how-to-sync-other-properties-from-active-directory-to-service-manager-using-orchestrator/




Workflow / Connector SchedulingOption 1: Configure in XML or PowerShellCan cause workflows to run more efficiently

Option 2: Configure Orchestrator to Turn Jobs on and offConfigure Orchestrator job with schedule to turn on and off / start / end



Batch SizeTweak the batch sizeCan cause workflows to run more efficiently

<Rule ID="CIListRule" Enabled="true" Target="OMConnectorLibrary!Microsoft.EnterpriseManagement.LinkingFramework.OpsMgrConnector.SyncWorkflowTarget" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100"> <Category>Maintenance</Category> <DataSources> <DataSource ID="DS1" TypeID="Subscriptions!Microsoft.SystemCenter.CmdbInstanceSubscription.DataSourceModule"> <Subscription> <InstanceSubscription Type="$MPElement[Name='OMConnectorLibrary!Microsoft.EnterpriseManagement.LinkingFramework.OpsMgrConnector.OpsMgrCIs']$"> <UpdateInstance /> </InstanceSubscription>  <PollingIntervalInSeconds>10</PollingIntervalInSeconds> <BatchSize>1000</BatchSize> </Subscription> </DataSource> </DataSources>



Related Orchestrator Tips



Orchestrator Topologies and Service ManagerOrchestratorUse a dedicated SQL serverUse dedicated runbook serversUse dedicated runbook and management servers for specialized functions

Service ManagerTarget specific Service Manager server, not used for consoles or workflows

HardwareRunbook servers: 4 core, 16 – 32 GB RAMManagement servers: 4 core, 16 – 32 GB RAMSQL servers: 4 core, 16 – 32 GB RAM



Orchestrator Topologies and Service ManagerRunbook DesignUse smaller, more easily segmented runbooksDO NOT use “activity specific” logging in production (is substantially slower by 3 to 1 ratio)

Runbook ExecutionUse dedicated runbook serversUse dedicated runbook servers for specialized functions

Clear the LogsAutomatically clear logs, unless the cycle is to rapid, causing it to lock up the systemManually clear the logs or use Orchestrator job:http://blog.concurrency.com/infrastructure/virtualization/manually-clearing-orchestrator-logs/ For more information: www.concurrency.com

http://blog.concurrency.com/infrastructure/virtualization/manually-clearing-orchestrator-logs/




Orchestrator Topologies and Service ManagerBeware of the “Licensing Issue”Make sure to upgrade to SP1, beware of “license” issue in earlier Orchestrator builds

Hardware and LoggingMake sure your Orchestrator servers are on great hardware, especially disk hardwareDo not configure your logging improperly (use either simple or full with transaction log backups)



DEMO: Manually Clearing the Logs



Data Warehouse



DW and SQL PerformanceScalability and SQL Sizing ImportanceUse a dedicated SQL serverAllocate enough RAM and don’t starve the OSUnderstand analysis services impact and potentially offloadBeware cube processing impact

Running Jobs ManuallyHow to execute a job to correct issuesOrder of events when updating the DW

Snapshots in CMDBUtilize snapshot driven CMDB reports for accessing non-DW data without performance impactProvide reports in self-service driven model, such as through Reporting Services + SSP



Configure Transform Batch SizeConfigure the Batch Size of the TransformAdjust to larger batch

The transform module default batch size is 50,000 items. The batch size can be adjusted by inserting rows into the DWRepository.ETL.Configuration table as follows:insert into DWRepository.ETL.Configuration( ConfigurationFilter, ConfigurationPath, ConfiguredValueType, ConfiguredValue)values ( 'etl.Transform', 'BatchSize', 'Int32', '100000') For more information: www.concurrency.com


Questions!



Top Tweeted QuestionsQuestions…Why is Service Manager so cool?



Evaluation

Complete your session evaluations today and enter to win prizes daily. Provide your feedback at a CommNet kiosk or log on at www.2013mms.com.Upon submission you will receive instant notification if you have won a prize. Prize pickup is at the Information Desk located in Attendee Services in the Mandalay Bay Foyer. Entry details can be found on the MMS website.

We want to hear from you!



Resources

http://channel9.msdn.com/Events

Access MMS Online to view session recordings after the event.



© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Technology

Configuring Service Manager for Performance and Scale - MMS2013 Presentation