run book automation with hp operations orchestration

RUN BOOK AUTOMATION

WITH HP OPERATIONS ORCHESTRATION

04.10.2011

Stefan Zahrer

Agenda

Why introduce Run Book Automation

Planning

Tool selection and appraisal

Experience so far

RBA Design Framework

Summary

30.09.2011 2 Titel der Präsentation und Name des Redners

WHY INTRODUCE RUN BOOK AUTOMATION

RBA implementation project objective

1. Main objective: Improve RUN efficiency in Infrastructure Operations – reduce cost

2. Develop a RBA platform for all global data centers

3. Establish a RBA service

Quality criteria

Improve provisioning speed and quality

Make automation available to all services

Ensure reusability of flows and low maintenance effort

Quickly deliver financial benefits

Oh, almost forgot: Make this happen in a infrastructure that was

build over the last 10 years without automation in mind…


Vision: Fully automated provisioning – no hands-on tasks

Rapid Provisioning

High Standardization

Up-to-date Documentation

Reliable Quality – No Human Errors

Low Process Costs

No Workload Buildup

Less Operational Management


A

u

t

o

m

a

t

i

o

n

Manual Provisioning

Industrialized Provisioning

Fixed Service Offering

Self-service Portal with service request dialog

Detailed execution logs

Reliable lead time and quality

PLANNING

Planning


POC for

tool selection

RBA Design Framework Iterations

Pilot

Virtual Windows Server

Establish RBA Service

2010 2011 2012

Other Flows and Automations

(HP SM Interface, etc.)

Management

Decision

More Automations in

Munich

Automations for

global

Datacenters

Re

vie

w

Automations in the making

Installation of Managed virtual Linux Server

Installation of Oracle Databases

Promote AD Member Server to Domain

Controller

Automation for HVD data store administration

(later reuse for Netapp Storage London)

TOOL SELECTION AND APPRAISAL

HP OO wins in proof of concept

3 tools in POC

5 automation tasks to

demonstrate capabilities

Only HP OO can complete all 5

tasks within a set time

9

Category HP OO B C

Accomplishment of mandatory workflows ++ ++ +

Accomplishment of optional workflow ++ o o

Skills of personnel (Number of personnel) ++ ++ +

Number of pre-assembled connectors and potential

workflow development effort +++ + o

Overall quality of results ++ ++ +

Pricing + + -

HP OO tool appraisal after nine month of flow

customizing


In general very happy with HP OO tool!!!

Easy to use

Flow customizing more efficient than classical scripting

Program control in graphical flows is easy to understand

HP OO Central Web Portal

More functionality for OO Central user input dialog

Calendar for date selection, set default values, radio button, etc.

Fix one or two bugs OO Central

HP OO Studio features to improve

Flow synchronization and staging is complex

Replace file system based repositories with database repository with staging

functionality

Coordination of shared flow customizing is difficult

Improve check-in/check-out functionality

Feature to synchronize system accounts between repositories

More transparency on flow permissions, better handling

Non-functional

Further development internet community and knowledge base

Optimize support reaction times

Experienced HP OO experts in short supply

Accelerator Packs and Integrations are like coding examples but no simple drag&drop

EXPERIENCE SO FAR

People

People like their jobs – sometimes not readily offering information/support

For automation we need to now how things are done

but also why they are done this way – understanding

both takes time

Automation requires cross-service collaboration

Initial concern about automation going berserk

In the beginning people were critical – my job is to difficult for automation

Automation without peoples support may be possible but effort explodes

30.09.2011 12 Stefan Zahrer

Technology

Infrastructure is complex – very, very complex indeed

Ask for picture of the architecture – discover it’s in

people’s heads

Work on production systems as early as possible – lab

systems are not the same

Architecture must be designed with automation in mind

Automation can be driver for standardization


Run Book Automation is organizational change for all

IT functions


Automation Benefits

High Standardization

Fixed Service Offering

Fast and reliable

provisioning

Infrastructure Operations

Design overarching

processes and consistent

infrastructure

Cross-service collaboration

Work on architecture, not in

run operations

Share responsibility for

automated service delivery

Outsourcing Provider

Collaborate in automation

design, build and release

Include automation in

continuous service

improvement process

Support operators to

become experts with cross-

service knowledge

Application Development

Identical processes for all

projects

Develop Self-Service

provisioning know-how

Standardize architecture to

a higher level

Technical Infrastructure

Standardization on all

levels (Storage, Server,

OS, DB, …)

Administration designed for

automation (tool-based

decisions, etc.)

Offshoring

More advanced

administration tasks

Better understanding of

infrastructure required

Participation in automation

design

Documentation

Documentation in

automation tool

Consistent and reliable

Focus on technical

administration processes

Develop

People

Develop

Business

Model

Develop

Standards Develop Qualification

RBA DESIGN FRAMEWORK

The Gordian Knot – how to ensure reusability and

maintainability

HP OO flows execute functions in infrastructure systems based on organizational processes

Process sequence and technical operations are closely interwoven

What happens if things were to change?

In the infrastructure: new software version, feature changes, new systems, …

In the organization: Changes to processes, new CMDB attributes, …


Automation Flow

Automation Flow

Automation Flow

Automation Flow

Infrastructure

Major topics of RBA Design Framework


Flow Authoring

Flow Architecture

Flow Call Relation-

ships

Naming Conventions

Authorization Model

ITIL Integration

Change Management

Incident Management

CMDB Maintenance

Downtime Management

Freeze times, veto, …

Flow Staging and Deployment

Flow Repositories

Flow Release Process

Private Database

Advanced logging / reporting

Documentation of user input

Infrastructure Topology Model

Infrastructure Interface (II)

Action Plan (AP)

Optional: Subcontroller (SC)

Controller (CO)

User Interface (UI)

Utility

(U

T)

Transactional Data

Persistence Table

Key Value Store

Logical

Logging

Server Name

Table

Master Data Infrastructure Topology

Model

Select and Start User-

Interface-Flow in HP OO

Central

Input in User-Interface-

Flow of HP OO

System Call to I3S

(Create/Save Change)

System Call to I3S to

verify Change

Relevance

Change Relevant

Flag set?

Yes

Input Risk- and Impact-Parameters

Execution Time?

Start immediately after aproval

No

Submit Change

Input Parameters for Downtime

Request

Downtime Window

Input Parameters for

Time is agreed (Start

Time/Finish Time)

Fetch Downtimes via I3S for first

task

Time is agreed

Possibility that

downtime window

is not available

Writing of Variables in Private

Database

End of User Interface Flow

Aproval Process

First Automated Task (Status assigned)

I3S creates XML-File

Scheduled File Check picking up XML-File

Start of Scheduling Flow

- Acceptance of I3S Task

- Searching for Downtime of every CI in I3S

- Scheduling of Controller Flow for every CI at defined

Downtime or execute immediately

Start

Start of scheduled Controller Flow

Check if respective Change is canceled

No

Canceled? YesAbort

Report in Private Database

CI still exists in I3S

Change?

No

Yes

Check if running in

Downtime Window

Mode (Flow Input)

Check if Downtime is

still valid

Yes

No

Yes

Read Input Values from Private

Database

Do Work on CI

Exited with Error?

Update Data in Private DB

Check for last CI

Document in I3S

no

Create Incidentyes

Close Task

yes

Exit

no

SUMMARY

Summary

Tools are (finally) ready for large style automation

Vendors starting to include interfaces for automation

in their products

Automation of existing infrastructure is possible

to large extend

Long time risk in using tools without

architecture model in place

Implementation effort is significant

Financial benefit/ROI depends on number of runs


Questions


BACKUP

Flow Architecture – separates process control and

system control



Action Plan (AP)


Controller (CO)

User Interface (UI)

Utilit

y (

UT

)

Creates GUI for users

• Populates selection lists

• Verifies user input

• Checks user permissions

Controls processing order

of tasks

Executes a single function on

one infrastructure system

Combines functions to execute

a complex task on one system

Combines tasks on different

systems to reduce complexity

of Controllers

Characteristics of flow types


Flow Type Complexity Reusability Modify and Test for

change to

Modify/Test

Effort

User Interface Very High Very Low - User input

- Infrastructure topology

- Organizational Process

Can be tested without affecting

infrastructure

Medium

Controller High Very Low Organizational Process High

Subcontroller Medium Low Technical Process High

Action Plan Medium High Infrastructure systems and

applications

Low

Infrastructure Interface Low High Infrastructure systems and

applications

Low

Utility Low High New HP OO versions only Low

Rules for flow calls limit complexity




Controller (CO)

User Interface (UI)

Utilit

y (

UT

)

Action Plan (AP)

Execute Configuration Changes


Execute Configuration Changes

Action Plan (AP)

Infrastructure Systems Private DB

Logging, etc.

Scheduler Persistence

Table

• Gather information

• Create i3S Changes

• Logging

Private Database enhances HP OO standard features

Reporting of success/failure

Detailed reporting for services

Documentation of user input

Handover variables for scheduled runs


Transactional Data

Persistence Table Flow run specific variables

Key Value Store Global Constants

Logical Logging Basis for result reporting

IP-NameTable Server Name Mgmt.

more to come…

Master Data

Infrastructure

Topology Model


Dependencies in infrastructure (Locations, Networks, AD Entities, SW Packages, order, preferences, etc.)

Supports guided user input

Reduces flow complexity

Infrastructure Topology Model stores information in a

structured easy to maintain form

Main purposes

Enhance User Input flows

Content of list boxes, default selections

Limit selections to valid combinations

Dynamic selections in Controller flows based on

System architecture, location, usage, …

Preferences (e.g. prefer one ESX Cluster

whenever possible)

Control software installation flows

Software versions/editions

Installation order

Required and optional packages

Staging data from DEV to PRD


RBA_Types

ID

Type

Description

int

varchar(512)

varchar(512)

<pk>

RBA_Entities

ID

Type_ID

Name

Description

int

int

varchar(512)

varchar(512)

<pk>

<fk>

RBA_Entities_RBA_Entities

ID

Parent_Entity_ID

Child_Entity_ID Description

int

int

int varchar(512)

<pk>

<fk2>

<fk1>

RBA_Entities_Extended_Attributes

ID

Entity_ID

Name Value

Description

int

int

varchar(512) varchar(512)

varchar(512)

<pk>

<fk>

RBA_Entities_RBA_Entities_Extended_Attributes

ID

Entitiy_Entity_ID Name

Value

Description

int

int varchar(512)

varchar(512)

varchar(512)

<pk>

<fk>

RBA_Entities_RBA_Entities_RBA_Entities

ID

Entity_ID

Entity_Entity_ID

Description

int

int

int

varchar(512)

<pk>

<fk1>

<fk2>

RBA_Entities_RBA_Entities_RBA_Entities_Extended_Attributes

ID

Entitiy_Entitiy_Entity_ID Name

Value

Description

int

int varchar(512)

varchar(512)

varchar(512)

<pk>

<fk>

No “hard encoded” values in flows

Reduce flow complexity – data sets defined by SQL queries

Less need to change to flows – edit database or SQL statement only

Database changes are immediately available in all affected flows

Execution Flows User Interface Flows

Private DB

Interaction between flows and Private DB


Dialog Step 1 Dialog Step 2 Summary

Step

SQL SQL SQL

HP OO

Scheduler

Controller Flow


Persistence

Table

Logical

Logging

Table

SQL

Dependencies virt. Windows Installation – simple view


Location

Name

Hosting

Site

Vmware

Cluster

Windows

Server OS

Version

Windows

Server

Packages

Windows

Server

Role

AD

Domains

Type

Vmware

Network

Label

Sites Usage

Cost

Center

Windows

Server

Add.

Groups

Network

Segments

Vmware

Data-

center

Windows

SW Depot

Vmware

vCenter

Windows

Server OS

Edition

Windows

Server

Image

Network

DNS

Domains

Vmware

Resource

Pool

Vmware

Datastore

Suffix

Hostname

Prefix

Owner

Group

Vmware

Datastore

Exclude

Backup

Environ-

ment

Job Mgmt

Server

USER EXPERIENCE AND REPORTING

Most input is selected

from list boxes

Comments support user

Input dialog with list boxes – supports the user input

and avoids typos


Step by Step

Input

Step 1 Step 2 Step 5: Input Summary

Only valid combinations can be selected – less room

for errors


Step 1 Step 2

Limited

selections

Infrastructure

Topology Model

Cost Center and description

from database

Detailed logical reporting of all changes to

infrastructure


UUID = 7a18cc03-5cba-4d2b-9eab-1b594f3b38bc

Timestamp RunID ChangeID Requester CI Status Flow Comment

7/21/2011 2:27:50 PM -30 C1234567 n059307 Start CO.windows.vm.create Start of VM Create

7/21/2011 2:27:50 PM -30 C1234567 n059307 Start AP.infoblox.getNewHostname

7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Success AP.infoblox.getNewHostname

7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start AP.infoblox.getIpByName

7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Success AP.infoblox.getIpByName

7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Success AP.dtm.addDownTimeWindow

7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start SC.vmware4.vm.deployHost Start of VM Deploy

7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.vm.getDetails

7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 VM not found AP.vmware4.vm.getDetails

7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.customizationspec.modify

7/21/2011 2:27:52 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.customizationspec.modify

7/21/2011 2:27:52 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.datastore.getLargest

7/21/2011 2:27:54 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.datastore.getLargest

7/21/2011 2:27:54 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.vm.deploy

7/21/2011 2:27:55 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.vm.deploy

7/21/2011 2:27:55 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.task.waitUntilFinished

7/21/2011 2:46:34 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.task.waitUntilFinished

7/21/2011 2:46:34 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.vm.setCpuAndMemorySize

7/21/2011 2:46:49 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.vm.setCpuAndMemorySize

7/21/2011 2:46:49 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.vm.setPowerState

7/21/2011 2:46:59 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.vm.setPowerState

7/21/2011 3:05:18 PM -30 C1234567 n059307 mucs801126 Failure SC.vmware4.vm.deployHost Windows Deployment failed!

7/21/2011 3:05:18 PM -30 C1234567 n059307 mucs801126 Failure CO.windows.vm.create ServerDeploy failed!

HP OO executes functions on infrastructure

components


HP OO i3S CMDB

DPPM

Infoblox

Network

Networker

Backup

$Univers

Job Mgmt AD

Directory Service

DTM OVO

Monitoring

SCOM

Monitoring

i3S Incident

i3S Change

SCCM

VMware

vSphere

Netapp

Storage & Fabric

Windows / Linux

OS

HP SA

HP OO flows

Reuse for other datacenters requires standards

Similar infrastructure systems and architecture

Management and administration of systems designed to support automation

Common ITIL management tools and processes

planned

Status flow customizing


Topic Scope

Released for Munich DC

Installation of managed virtual Windows servers

Users: IT Operations

mucs- and mums-servers in most common networks

Not covered: Add new server to DTM Tool

Add new server to Networker DB

Reserve IP Address and IP Name for mucs/-mums-Servers

Users: dedicated Team members

Permissions currently limited to four networks/VLANs

Create/revert/delete VMware Windows snapshots

Users: BA Service

Limited usage, make available to DIPRO/AO.NET?

Planned

Installation of virt. W2S Servers

Users: DMG, AI

Installation of Managed virtual Linux Server

For SAP virtualization 50 – 70 Linux installations required from November

Users: Linux Service, SAP Basis

Installation of Oracle Databases

Users: Oracle Service

Promote AD Member Server to Domain Controller

Users: AD Service

Automation for HVD data store administration (later reuse for Netapp Storage London)

Users: Storage Service, Windows Service

Automation of simple changes to $Universe backup jobs

Users: Job Mgmt. Service, SAP Basis

THANK YOU

Documents

run book automation with hp operations orchestration