DevOps and The Service Desk

Don’t Let The Developers Hijack The Discussion!

Randy SteinbergRSteinberg@concurrency.com

CHICAGOLAND

LOCAL CHAPTERConcurrency

The Issue

Here You

Go!Great!

Dev

Ops

This Is A Billion Dollar Problem!

• Major Healthcare Deployment Failure – HHS Secretary Resigns

• Financial Firm $457M Trading Error Riles Investors

• Customer Order Failures Result In IPO Withdrawal

• Airline IT Woes - Lost Customers, Airport Fines, Stock Drops

2.4%

• Federal Disaster - High volume of background check

transaction volumes halts gun sales

What is the return on

investment of an IT

solution that cannot be

deployed and operated at

acceptable cost and risk?

Amount negative equal to

your development

investment plus lost

opportunity costs and

possible fines and penalties.

When does the business

actually see and realize

value from IT?

At the point the…

…SERVICE is delivered!

Scary Things People Say About DevOps

• “With DevOps we can deploy up to 15 times a week…”

• “You can bypass your IT organization…”

• “Infrastructure engineers often have little incentive to change anything - they are

used to being ignored until something goes wrong”

• “Just co-locate dev and ops desks and place them under the same management”

• “It’s all about continuous delivery of services…”

DevOps Defined

• A cultural and professional movement that stresses communication, collaboration, and integration between software developers, quality assurance and IT operations professionals with these perspectives:

– A service perspective (people, process, technology, data)

– An agile perspective (continuous delivery, faster deployment)

– A risk perspective (safety, security, reliability)

DevOps Operating Principles

• Flow – increase the flow (speed) of work from left(Dev) to

right (Ops)

• Feedback – continuously improve, work together (extend

operations into development) and recognize that mistakes are

more expensive to deal with once in production

• Continuous Experimentation and Learning – create a culture

that fosters experimentation, and quickly learns from failures

DevOps Lingo

• Continuous Integration – requirement that developers

integrate code into a shared repository on a daily basis

• Continuous Delivery – make sure that software is ALWAYS in a

releasable state throughout it’s lifecycle - not to be confused

with…

• Continuous Deployment – every change that passes

automated tests gets automatically deployed to production

Key DevOps Benefits� Allows IT to respond effectively to accelerated demand for quality software products and services

� Recognizes the need for cultural and organizational improvements to communicate and

collaborate across IT engineering silos

� Supports and leverages industry best practices (IT Service Management, Agile, Lean)

� Encourages use of automation wherever appropriate

� Shifts labor costs from unplanned, low value activities to productive value activities

� Claimed Benefits*:

� Code shipped 30 times faster

� Deployments completed 8,000 times faster

� 50% fewer failures

� Failures restored 12 times faster*2013 State Of DevOps Report – Puppet Labs

Melding Together

Dev Ops

Extend delivery to production

Extend operations into

development

Ensure delivered solutions

can be operated at

acceptable cost and risk

Offer IT support services in a

manner that can easily be

consumed at acceptable

speed and cost

Ensuring solutions can be operated at

acceptable cost and risk

“Let’s put kiosks in shopping malls to sell our products!”

Utility – Fit For Purpose Warranty – Fit For Use

“Is there enough space and power?”“Can the floor handle the weight of the kiosk?”“Does it pass fire and safety inspection?”“Is there a pathway to move kiosk into position?”“Who unpacks and disposes of the shipping crate?”“Do malls restrict when you can install the kiosks?”“What repair service will we use to fix them?”“What does customer do if kiosk is not working?”“Who approves content to be displayed on kiosk?”“Do we have adequate skills to support them?”“Can support resources quickly get on site if needed?”“etc….”

Working Together – A Win-Win Story

Offer IT support services in a manner easily

consumed at acceptable speed and cost

� Lean Operations

� Hybrid Cloud

� Virtualization

� Containerization – Converged Infrastructure

� Declarative Configuration Management

� Software Defined Infrastructure/Networks/Data Center

Organization Challenges

Developer View Of Operations Architecture As Viewed By

Development and Operations

Here

You

Go!

Great!

My

Service

Levels!

Out Of

My Way!

Neither Of

You Truly

Understand

IT!

Operations View Of Development

A Typical IT Organization

IT

Development

CIO

IT OperationsNetwork

Management

Server

Management

Storage

ManagementEtc….

No one is accountable for service – just capability

What organization is

best positioned to work

across all these silos?

Extending The Service Desk May Be The Easiest

Approach To All This Organizationally

IT

DevelopmentIT Operations

IT

Architecture

and

Operational

Planning

Service Desk

Developer View Of IT

We are king – we own the CIO relationship

We are closest to the business and know what’s best for them

No functionality = No solution

What they say about themselves

What they say about ArchitectureImportant to us – for Data and Applications

Could care less about rest of enterprise – they should be using

what we develop

Gets in the way and stifles our creativity

What they say about OperationsSlooooowwwww!!!

We’ll talk to them when we’re ready to go live (usually 1-2 weeks before cutover)

We’ve already built solutions for important things like backup, job scheduling and monitoring

“They’ll get it done – they always do…”

Operations View Of ITWe’re always dumped on at the last minute

We are closest to how things work and know what’s best

for everyone

We’re the ones that make it all work…

What they say about themselves

What they say about ArchitectureThey speak gobbilty-gook…

Why bother? – Just put the tools in…

What the hell is an architecture anyway?

Best job in IT – all theory and don’t have to produce anything….

What they say about Development

They want to install what???

Why weren’t we told about this earlier?

These guys have no clue how to make things really work

“There’s no budget for this…”

Architect View Of IT

No one in IT appreciates what we do

Most in IT don’t understand important concepts like layering,

abstraction, normalization…

What they say about themselves

What they say about Development

Developers don’t listen to us…

They think we’re just a tick mark on the QA checklist

Reuse is not in their vocabulary…

What they say about Operations

Never heard of us…

These guys are so tactical we can’t relate…

Break In The Service Delivery ChainTypical Developer/Architect Concerns...

...Typical Things They Wish To Avoid Until Later

Design Build Test Convert

Rollout Operate Support Monitor

The job is done when thesystem is built

The job ain't done 'til wecan run this thing safely

Architect

Dev and Ops – Working Together Throughout

The Entire Development Lifecycle!

Build, Test And

Deploy IT Service And

Support Infrastructure

Validating That IT And

Business

Requirements Can Be

Met

Execute IT Service And

Support Activities To

Support Users and

Customers

Review IT Service And

Management Metrics

To Proactively Identify

Opportunities For

Improving IT Services

Design Service And

Support Organization,

Processes,

Technologies, Data

And Reporting

Infrastructure

Determine Service

Scope, Offerings,

Service Metrics,

Service Level,

Requirements,

Sourcing And Delivery

Strategy

Service

Improvement

(Act)

Service

Operation

(Check)

Service

Transition

(Do)

Service

Strategy

(Plan)

Service

Design

(Do)

DEV

OPS

Requirements Design Build & Test Deploy Maintain & Support

So What Happens When You Put

Development and Operations Together?

We’re in

conceptual data

design and we

just did a logical

database

architecture…

…so how much

storage you need

in GB? What size

processor? How

about network

traffic volumes?

SLA objectives?

Service Management Is Now Critical!!

1111

Customers or Business End Users

IT Services

Applications Applications Operations

External Suppliers

Servers Networks

1111

Developers

IT Service Management “Bridge”(Coordinates delivery of services across IT technology silos)

External SuppliersExternal SuppliersExternal SuppliersExternal Suppliers

Common Delivery Tools

Service Catalog

Support Services

Let’s build a bridge

to our developers!

Common Delivery Processes

Service-Based Organization

Ops needs to

provide

development

with standard

services easily

consumed

DevOps – Examples Of ServicesService Catalog/Portfolio Examples

� Service Desk� Service Monitoring� Incident Response � Problem Control� Request Fulfillment� Backup/Restore Mgt� Job Schedule Mgt� Clock Mgt� Startup/Shutdown Mgt� File Transfer and Control Mgt� Archive Mgt� Data Entry Support� Report Distribution� Request Fulfillment

� Release Planning and Packaging � Service Deployment and

Decommission� Site Preparation Support � Service Validation and Testing

Support� Training Support� Organizational Change Support � Knowledge Management� Test Lab Management� Security Testing

� Service Catalog Mgt� Service Level Mgt� Supplier Mgt� Operational Planning and

Consulting� Solution Planning and

Development� Development Support

Operations� Capacity Mgt� Availability Mgt� IT Service Continuity Mgt� Process Design Support

� Lease and License Mgt� Service Audit and Reporting� Process Mgt� Risk Mgt and Audit Support� Security Design and

Consulting� Enterprise Architecture Mgt

and Research� IT Financial Mgt� IT Project Mgt� Change Control � Configuration and Asset Mgt

Service

Operations

Services

Service

Transition

Services

Service

Design

Services

Control

Services

Developers Work With Operations To Pick

From Sets Of Services

Leverage Existing

Support Services

Enhance Existing

Support Services

Purchase New Tools

And Support

Solutions

Support Service Cannot Be

Addressed By Any Vendor or

Tooling Solution

Almost None - But

Must Ensure

Standards Are

Being Adhered To

May Have To Build

Additional Scripts,

Log Files And Event

Handlers To Better

Leverage Existing

Infrastructure

Varies -

Depending

On Service

Solution Scope

May Require Design,

Build And Development

Efforts To Provide Needed

Services

Reuse Work To Do Spend Money Complexity

Provide Support

Services Via An

Outside Vendor

Varies - But May

Need To Provide

Input For Service

Requirements

Outsource It

Design

Decision

Design

Impact

DevOps - Impact On OperationsChange Management

Slow!!! – Most changes average 8-15 business days (mostly waiting for approvals!)

New Focus:

� Quickly determine risk and then increase or decrease flow – e.g. if the risk is low, let the

Change Manager approve it – utilize CAB only for higher risk changes

� Aggressively drive Standard Changes – if the change doesn’t involve human hands and can

be automated, then risk is drastically lower

� Virtualization, Declarative Configuration Management, and Software Defined

Infrastructure is a 2 edge sword – changes to virtualized/software defined components is

much lower risk while changes to core components (e.g. Hypervisors, infrastructure as code

elements) carries much higher risk than normal

DevOps - Impact On OperationsRelease Management

New Focus:

� Automate Push Of Code From QA To Production

� Release Units = Total Solution Units – no longer dealing with different timetables and

releases by technology silo

DevOps - Impact On OperationsConfiguration Management

New Focus:

� Automate discovery and tracking of configuration changes – this is getting too complex to

do manually and too critical to ignore

� Tie configurations to the applications and services they support – can’t effectively manage

applications and services when you don’t know the pieces and parts

� Software naming and configuration management is critical – especially with agile practices

for continuous integration, release and deployment

DevOps Impact On Operations

Consider leveraging suppliers to provide

common support services

Need to ensure that suppliers do not

adversely introduce changes into the

production environment

The role of IT is changing to focus more on

service integration versus engineering and

building everything that may be needed!

Supplier Management

Dev and Ops – Working Together

Build, Test And

Deploy IT Service And

Support Infrastructure

Validating That IT And

Business

Requirements Can Be

Met

Execute IT Service And

Support Activities To

Support Users and

Customers

Review IT Service And

Management Metrics

To Proactively Identify

Opportunities For

Improving IT Services

Design Service And

Support Organization,

Processes,

Technologies, Data

And Reporting

Infrastructure

Determine Service

Scope, Offerings,

Service Metrics,

Service Level,

Requirements,

Sourcing And Delivery

Strategy

Service

Improvement

(Act)

Service

Operation

(Check)

Service

Transition

(Do)

Service

Strategy

(Plan)

Service

Design

(Do)

DEV

OPS

Requirements Design Build & Test Deploy Maintain & Support

More than just pushing code into production!

Strategy Phase – Considerations

Solution Model

Business Volumes

Specialized Support Needs

Implementation Timeframes

High Level Functional Requirements

Development Organization

Working Standards

Sourcing Partners

Solution Architecture

Overall Enterprise Fit

Standards To Be Used

Frameworks To Be Used

Opportunities For Reuse

High Level Service Requirements

Support Capabilities Fit

Hosting Partners

Delivery Strategy

Timeframes

Costs:

- One Time

- Ongoing

Development Considerations Architecture Considerations ITSM Consideration

Strategy Design OperationTransition Service Improvement

Strategy Phase – Step By Step

Develop Service

Descriptions and Features

Identify Key Service

Assets

Conduct Service Business

Impact Assessment

Identify Funding Strategy

Analyze Market,

Customers and Needs

Identify Service

Stakeholders

Identify Service

Ownership and Steering

Identify Service Value

And Objectives

Identify Service Critical

Success Factors

Develop Business

Demand Forecasts

Develop High Level

Service Requirements

Develop Service Model(s)Identify Standards and

Methods

Define Service Units and

Packages

Assess Service Delivery

Capabilities

Identify ConstraintsProcure and Assign

Service Resources

Identify Organizational

Change Strategy

Develop High Level Work

Plan and Schedule

Conduct Service Risk

Analysis

Identify Sourcing Strategy

and Partners

Develop Preliminary Cost

Estimates

Develop Service Pricing

and Cost Recovery

Strategy

Identify Service

Development Team

Prepare Service

Specification Charter

Develop Service Portfolio

Entry

Develop And Present

Business Case

Conduct Service Strategy

Review Gate

Identify Service Products

and Outcomes

Develop Perspective,

Position, Plans and Patterns

Strategy Phase - Deliverables

� Support Strategy

� High Level Solution Architecture

� Service Requirements

� Service Demand Forecasts/Estimates

� Cost Estimates

� Process Assessment

� Technology Assessment

� Organizational Assessment

� Training Strategy

� Working Standards

� Sourcing Partners

� Strategic Tooling Architecture

� High Level Deployment Strategy

� High Level Project Plans

Design Phase – Considerations

Application Designs

Software Package Customizations

Testing Requirements

Component Configurations

Procurement Requirements

Operational Detailed Requirements

(i.e.):

- Required Alerts

- Backup Requirements

- Solution Configurations

Data Architecture

Functional Architecture

Technical Architecture (i.e.)

- Application

- Middleware/Messaging

- Network

- Transaction

- Database

Operational Architecture

Data Models/Data Design

Availability Designs

Capacity Designs

Service Continuity Designs

Security Designs

Deployment Designs/Strategy

Process Designs

Support Roles/Responsibilities

Operational Control Tasks

Facilities Designs

Monitoring Designs

Procurement Requirements

Development Considerations Architecture Considerations ITSM Considerations

Strategy Design OperationTransitionService

Improvement

Design Phase – Step By Step

Design Service Delivery

Agreements

Design Service Availability

Design Physical Site and

Environmental Elements

Design Service

Communications and

Training Elements

Implement Design

Knowledge Bases and Tools

Select Design Approaches

and Strategies

Assemble Design Team,

Plans and Schedules

Design Business and

Management Process

Workflows

Design Service Data and

Information Elements

Develop Detailed Design

Requirements

Design Application ElementsDesign Request Fulfillment

Models

Design Supplier Interfaces Design Networking ElementsDesign Service Delivery

Organization

Design Service Technical

Assets and Configurations

Design Service CapacityDesign Management Tooling

Elements

Design Service

Documentation Elements

Design Monitors, Events,

Alerts and Thresholds

Design Service Operability

Elements

Design Security

Design Service Continuity

Elements

Design Service Billing and

Chargeback

Procure Service Delivery

Suppliers and Partners

Design Service Testing

Strategies and Plans

Design High Level Transition

Strategies and Schedules

Develop Service Design

Package

Conduct Service Design

Review Gate

Design Quality

Measurements and

Indicators

Design Phase - Deliverables

� Technical Architectures

� Data Architectures

� Service Descriptions/Catalogs

� Designed Processes

� Designed Operational Procedures

� Designed Tooling Infrastructure

� Designed Roles/Responsibilities

� Designed Measurements/Reports

� Designed Monitoring

� SLA/OLA Agreements

� Availability Designs

� Capacity Estimates

� Security Designs

� Physical Facility Designs

� Service Continuity Designs

� Supplier Integration

� Component Configurations

� Testing Requirements

� Detailed Transition Plans

� Detailed Training Plans

Transition Phase - Considerations

Application Coding

Package Implementation

Migration/Code Libraries

Rollout Schedules/Volumes

Maintenance Needs

Testing and Validation

User Acceptance

Support Call Lists

Early Life Support

Validated Architecture

Ensured Integration With Rest Of

Enterprise

Updated Architecture Documentation

Migrations/Releases

Operational Testing

Site Surveys/Preparation

User/Support Training

Operational Integration:

- Service Desk

- Technical Support

- Operation Control

- Facilities Management

- Application Management

Development Considerations Architecture Considerations ITSM Considerations

Strategy Design OperationTransitionService

Improvement

Transition Phase – Step By Step

Build Service Quality Metrics

and Reporting

Document Service

Configurations and

Knowledge

Conduct Service Operational

Testing

Finalize Service Cost Models

and Budgets

Conduct Transition Risk

Assessment

Establish Transition

Governance

Assemble Transition Team,

Plans and Schedules

Finalize Service Acceptance

Criteria

Build Physical Site

Infrastructure Elements

Establish Service Release

Policies

Build Service Application and

Utility Elements

Build Service Management

Warranty Elements

Configure Service For

Production Operation

Finalize Internal and External

Support Agreements

Build Service Continuity

Elements

Implement Service Delivery

and Support Organization

Conduct Service Component

Testing

Conduct Service Availability

and Security Testing

Conduct Service User

Acceptance Testing

Conduct Service Capacity

and Performance Testing

Conduct Service Quality and

Compliance Reviews

Conduct Service System

Integration Testing

Conduct Service and

Support Training and

Communications

Deploy Service Elements

Develop Detailed Transition

Plans

Evaluate Service and Known

Errors

Implement Service Desk and

Early Life Support

Conduct Service Transition

Review Gate

Update Service Catalog and

Portfolio

Build Service Billing and

Chargeback Elements

Transition Phase - Deliverables� Detailed Transition Plans

� Operational Scripts

� Job Schedules

� Run Books

� Solution Documentation

� Configured Security

� Configured Components

� Operational Test Results

� Trained Users

� Trained Support Staff

� Solution Known Errors

� Installed Monitoring Agents

� Installed Processes

� Assigned Support Staff

� Installed Management Tools

� Site Surveys

� Facility Build Outs

� Installed Releases

� Early Life Support

� Installed Applications

� User Acceptance Results

� Post Implementation Reviews

Operations Phase - Considerations

None – at this point Application

Management takes over for ongoing

application maintenance and support

None. Delivering Services

Ongoing Support

Monitoring/Event Management

Maintenance

- Applications

- IT Infrastructure

Execution of planned operational

control tasks

Incident,/Problem Management

Request Fulfillment

Access Management

Development Considerations Architecture Considerations ITSM Considerations

Strategy Design OperationTransitionService

Improvement

Operations Phase - Deliverables� Incident Logs

� Problem Logs

� Operational Results

� Fulfilled Requests

� Escalated Alerts

� Shift Turnover Reports

� Escalated Operational Issues

� Delivered Services

� Executed Operational Controls

� Workforce Management

� Service Issues Log

Improvement Phase - Considerations

None – At This Point Application

Management Takes Over For

Ongoing Application Improvement

Review Of Service And Operational

Results To Identify Further

Architecture Improvement

Opportunities

Maintain Architecture While

Improvements Are Being Made

Collecting Service Measurements

Service Quality Reporting

Review Of Service And Operational

Results To Identify Further

Opportunities For Improvements

Development Considerations Architecture Considerations ITSM Considerations

Strategy Design OperationTransitionService

Improvement

Improvement Phase - Deliverables

� CSI Register

� Service Measurements

� Process Measurements

� Technology Measurements

� Service Quality Reports

� Service Dashboards

� Service Improvement Recommendations

� Service Improvement Plans

� Service Reviews

ITSMLib• Service Operation

• Service Transition

• Service Design

• Service Strategy

• Continual Service Improvement

• ITSM Implementation

• ITSM Organization

• ITSM Tooling Technologies

• ITSM Governance

• Tools and Aids

• Operations Management

• ITSM Metrics and Reporting

• ITSM Books and Articles

• Games and Simulations Write RandyASteinberg@gmail.com for free access

Rsteinberg@concurrency.com