Network Management KPI's

TECTDCT-3297

2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-1032 Cisco Public

Network Management

KPIs and ITIL BRKNMS-1032


Topics

Introduction

ITIL

Operational KPIs

QA Checks

3

Introduction

4


The Big Three it Takes to Be Successful

People

Process Tools

5


Network Management Lifecycle

6


Network Management Lifecycle

7


CSFs, KPIs, and Metrics

One or More Metrics Contribute to a KPI

One or More KPIs Contribute to a CSF

From a Hierarchical Perspective,

These Three Terms Are

Complementary to Each Other

8


ITIL Service Management Framework

9

ITIL Overview

10


ITIL ProcessDefinitions and Goals Incident Management

Incident: Any event that causes an interruption or reduction to the quality of an IT service

Restore normal service operation as quickly as possible and minimize the adverse impact on business operations

Problem Management

Problem: unknown underlying root cause of one or more incidents

1- Minimize the adverse impact of Incidents and Problems.

2- To prevent recurrence of Incidents related to these errors

Change Management

Change: Addition, modification, or removal of approved hardware, network, software, application, etc.

Ensure that standardized methods & procedures are used to minimize the number & impact of Change-related Incidents

Reduce the Risk of Changes

Event Management

Event: any detectable or discernible occurrence that has significance to the management of the IT infrastructure.

Proactive detection of anomalies

Proactive detection of element health

Solution validation

11


Incident Management Charter

Objectives

Restore normal service operation as quickly as possible and minimize the adverse impact on business operations

Definition

Any event that causes an interruption or reduction to the quality of an IT service

KPIs

Restoration within SLA

Incident Notification

Incident Resolution

Time to Isolate

Restore Service as Quickly as Possible 12


Incident Tickets

4966

4013

4730

1573

1036

1442

0

1000

2000

3000

4000

5000

6000

7000

Feb March April May June July

Incident Tickets

13


Sample Possible Weekly Incident Reporting

0

5

10

15

20

25

30

35

40

45

50

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8

Tickets by Severity

4-Low

3-Medium

2-High

1-Critical

0

5

10

15

20

25

30

35

40

45

50


Total Tickets by Week

0

5

10

15

20

25

30

35

40

45

50


Tickets by Severity

4-Low

3-Medium

2-High

1-Critical

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Week 1Week 2

Week 3Week 4

Week 5Week 6

Week 7Week 8

% Proactive vs Reactive

Reactive

Proactive

14


Problem Management Charter

Definition

Unknown underlying root cause of one or more incidents

Objectives

Minimize the adverse impact of Incidents and Problems

Prevent recurrence of Incidents related to these errors

KPIs

Permanent solutions

Known Error Articles

Proactive resolutions

Permanent Solutions to Reduce the Number of Incidents 15


Problem Management Open vs. Close Trend

11

20

14

1211

17

12 12

14

21

16

8

2

5

12

65 5

0

5

10

15

20

25


Open Close Solution

16


Problem Management Reactive vs. Proactive

9

18

12 12

11

14

2 2 2

0 0

3

0

2

4

6

8

10

12

14

16

18

20

Feb March April May Jun July

Reactive Proactive

17


Change / Release Management Charter Change is the only constant. http://en.wikiquote.org/wiki/Heraclitus

Definition

Addition, modification, or removal of approved hardware, network, software, application, etc.

Objectives

Standardized methods and procedures

Minimize the impact of Change-related Incidents

KPIs

Impact of Change on Incidents

Fewer changes backed out, cancelled

Predictable results Risk Management / Reducing the Risk of Change 18


Change Management Ticket Trend

60

48

58

67

61

76 75

65

41

59

82

45

78

5351

58 57

42

5856

78

0

10

20

30

40

50

60

70

80

90

Jan Feb Mar April May June July

Submitted Close Current Open

19


Other Change Metrics

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Jan Feb Mar Apr May Jun Jul Aug

% Backed Out Changes

# Backed out Changes Succecssful Changes

0

20

40

60

80

100

120

140


# Backed out Changes 20 26 30 34 24 26 24 30

Succecssful Changes 70 95 120 140 90 85 76 34

2 1 3 0 1 5 2

9

56

68

120

87 90 85

76

34


Changes Ouside MW

Changes Outside MW Changes Within MW

2 1 3 0 1 5

2

9

56

68

120

87 90

85

76

34

0

20

40

60

80

100

120

140


Changes Outside MW Changes Within MW Linear (Changes Outside MW) Linear (Changes Within MW)

20


ITIL Workflow

21

Operational KPIs


What KPIs are Important

23


SLAs, Goals, and Informational KPIs

SLAs are required

- SLAs usually have $s attached

- Customers Are Watching SLAs

Goals are Internal

- KPIs are used internally and externally

- Shared with customers to show Continual Improvement

Informational KPIs

- Used for internal measurement and continual improvement

Discussion / Q&A welcome

24


Tickets Per Device Trend

11.16

8.1

9.38

3.03

1.962.3

0

2

4

6

8

10

12

14


Tickets Per Device Trend

25


Sample Measurements

0

5

10

15

20

25


Top 5 Alarms

Call In

Port Down

Device unreachable

Port Error

High Utilization

0

2

4

6

8

10

12

Tickets by Location

P4

P3

P2

P1

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

JanFeb

MarApr

MayJun

JulAug

Axis

Tit

le

% of Duplicates

Real Tickets

Duplicates

Interface WAN Utilization

Speed Ave Util Peak % of time >

90%

Chicago 3.08 89% 100% 89%

New York 3.08 78% 95% 5%

Atlanta 3.08 56% 86% 0%

Denver 1.54 68% 91% 3%

Charlotte 3.08 93% 100% 100%

Greensboro 1.54 56% 79% 0%

High Point 1.54 5% 15% 0%

Winston-Salem 1.54 45% 87% 0%

Boston 3.08 73% 98% 19%

26


Tickets by Categories

27

2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-1032 Cisco Public 28

13

12

11

10

09

0

20000

40000

60000

80000

100000

120000

140000

Alarms by Week and Type

13

12

11

10

09


Number of Devices Generating X+ Tickets

56

47

99

16

712

0

20

40

60

80

100

120


Devices

29


Closure Code Count by Type

30


Top Targeted Addresses

31

0 50 100 150 200 250 300

10.1.3.234

10.34.2.321

10.21.5.214

10.1.3.45

10.1.3.23

10.34.2.26

Number of Attacks

Number of Attacks


Executive Dashboards Widget Inc Executive Dashboard

Metric Goal Jan Feb Mar Health

P1 na 6 4 6

MTTN (Min) 15 12 9 16

% Notified W/ Goal 95% 100% 100% 95%

MTTRestore (Min) 120 5 23 22

MTTR (Hrs) 4 0.5 0.65 1.1

P2 na 8 5 7

MTTN (Min) 15 20 13 12

% Notified W/ Goal 95% 85% 100% 100%

MTTRestore (Min) 240 134 164 154

MTTR (Hrs) 8 3.3 4.2 3.5

Total Tickets na 412 312 563

% Proactive 85 89 91 90

Calls Offered na 353 401 392

Average HT


Technology Specific KPIs

UCS / Servers

CPU Utilization

Memory Utilization

I/O speed

Drive Space

Interface Utilization

TX/RX Packet Drops

Fiber Channel Buffer to Buffer Tran

Short list of some Technology Specific KPIs

Security

Top Blocked Attack Signatures

Top Blocked Attack Sensor

Top Source Blocked Attacks

Top Destination Blocked Attacks

IPS Signature Caegories

Top Fired Signatures

Top Denied Packets

Top Denied Source Addresses

Top Source Address Failed Attempts

33


Technology Specific KPIs

Optical

Interface Light levels

Routing and Switching

CRC errors

Interface utilization

Packet drops (QOS)

B3 Error Bursts

Buffers

Short list of some Technology Specific KPIs

Voice / Video (TP)

Phone registrations

Device load / weight

Jitter / Latency (IPSLA)

% uptime of Key Services

34


KPI Process

Identify where source data to be used for KPI is collected or stored.

Retrieve data from source

Store data in a format similar to its source for current and historical analysis and reporting.

Transform the data into more human readable formats.

Merge the data with business metadata collected from additional sources.

Perform Calculations and summaries on the data, storing the calculated results as appropriate.

Format KPIs for output.

Generate Report and Store.

35


Data Sources

Log files

Data Stores

Streams

Reference DBs

Geo-IP

DLCM

Business Lookups

36


PDF

High Level Information Flow

Data

Sources

Hadoop/

Hive Cluster Flume

Kettle

Kettle /

Sqoop

ODS

Oracle/

Columnar JDBC

Pentaho

Reporting

Fusion

Charts /

Liferay

Java

AP

I

PDF

Excel

HTML

Flash

Reports via

Email or

Portal

Pentaho

Analytics

R Project

(Statistical)

Map Info

(Spatial)

JD

BC

JD

BC

Geo-Spatial

37


Pentaho Data

Integration

PNOC Data Sources & Targets (Present & Future)

Remedy

ANA

FNMT

DLCM

InfoVista

PDF

Excel

HTML

Flash

Reports via

Email or Portal Pentaho Data

Integration

Geo-Spatial

PNOC

Operational

Datastore

Pentaho BI

Suite

Reports

Analytics Portals/

Dashboards

Purpose

Built Data

Marts

Prime NCS

Prime Optical

Net Forensics

Mars Observer

& More

NCCM

38


Measurement Value Chain: Data to KPI

Identify

Source of

Data

Import All data

to PODS

PNOC

Operational

Datastore

Clean up

dates and

values

Merge Raw

Data with

Business

Information

Perform

Calculations &

Summaries

Format for

Output

Generate KPI

Report

39

QA Checks


Questions Around QA Checks

Should we do QA checks?

What should we QA?

How Much should we QA?

How do we use QA Data?

What does the data mean?


41


Number of Tickets Completed

Engineer Wk 1 Wk 2 Wk 3 Wk 4 Wk 5 Wk 6 Wk 7 Wk 8

Joe 10 11 10 12 13 12 11 13

Sunil 8 8 9 9 8 5 10 9

Amanda 12 11 10 12 13 10 10 11

John 8 8 9 8 10 11 10 10

Brian 10 11 10 12 11 13 10 11

Carlos 15 14 15 16 14 10 10 10

42


Ticket QA Check %s

Engineer March April May June

Joe 60% 75% 90% 95%

Sunil 100% 90% 95% 95%

Amanda 70% 75% 80% 85%

John 40% 50% 60% 55%

Brian 65% 75% 90% 95%

Carlos 75% 90% 95% 100%

43


QA Facts to Remember

What you QA can and should change

The True meaning comes from multiple sources

Select the most complicated or critical operations

QA Data can be used for multiple reasons

Sample tickets dont QA all data unless its Automated


44


Sample QA List

Was the ticket documented correctly per process

Did Engineer isolate issue quickly (Within Goal, Training Opportunity)

Did Engineer fill in Resolution time field (Specific New Process Tracking)

Did Engineer Follow up with customer per process

Was proper escalation process followed

Where technical errors made in the change

Did Engineer respond appropriately to the Incident / Customer

45


How do we use QA data

The main reason for QA is internal understanding

This allows for targeted training with the team

Engineer specific coaching as needed

Can be used as part of a motivational incentive program

Can create healthy competition between engineers / teams

Used as an internal metric for quality of work

QA Checks can improve team performance

Using QA checks with customers

Overall QA can show quality of work and transparency ( be careful)

The fact that customers know that you do QA internally is good

QA checks WILL improve quality and Customer Satisfaction

46


Summary

Know What the KPIs Mean

Incident Management

Problem Management

Change Management

Most Important thing about KPIs?

Metrics, KPIs, & CSFs

Who do QA Checks Help?

How to Present KPIs


47


Complete Your Online

Session Evaluation Give us your feedback and you

could win fabulous prizes.

Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through

our wireless network to access our

portal) or visit one of the Internet

stations throughout the Convention

Center.

Dont forget to activate your Cisco Live Virtual account for access to

all session material, communities, and

on-demand and live activities throughout

the year. Activate your account at the

Cisco booth in the World of Solutions or visit

www.ciscolive.com.

48


Final Thoughts

Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042

Come see demos of many key solutions and products in the main Cisco booth 2924

Visit www.ciscoLive365.com after the event for updated PDFs, on-demand session videos, networking, and more!

Follow Cisco Live! using social media:

Facebook: https://www.facebook.com/ciscoliveus

Twitter: https://twitter.com/#!/CiscoLive

LinkedIn Group: http://linkd.in/CiscoLI

49


Other NMS Sessions If you like this one, you might like some of these:

Source: Placeholder for Notes is 18 points

50

Session Title

Mo

nd

ay

BRKNMS-2515 A Difference-Maker for Operating Cisco Collaboration Services Proactive Problem Management

BRKNMS-2658 Securely Managing Your Networks with SNMPv3

BRKNMS-3043

Advanced Performance Measurement for Critical IP Traffic with Cisco IOS IP Service Level

Agreements

BRKNMS-1037 Cisco Unified Communications Management Suite Methodology and Best Practices

BRKNMS-3021 Advanced Cisco IOS Device Instrumentation

CCSNMS-1903

Customer Presentation: Learn How Human Kinetics Uses a New Cisco Solution to Dramatically

Improve Network and Security Operations

PSONMS-1038 Cisco Prime for Enterprise Innovative Network Management

Tu

esd

ay

BRKNMS-1032 Network Management KPI's (You are Here)

BRKNMS-3132 Advanced NetFlow

BRKNMS-2035 Ten Cool LMS Tricks to Better Manage Your Network

BRKNMS-3021 Advanced Cisco IOS Device Instrumentation

BRKNMS-2640 Advanced DHCP and DNS Deployments


Other NMS Sessions If you like this one, you might like some of these:

51

Session Title W

ed

nesd

ay

BRKNMS-2512 Moving Cloud from hype to realty: Delivering Cloud using management architecture

BRKNMS-2031 SYSLOG Design, Methodology and Best Practices

PSONMS-1038 Cisco Prime for Enterprise Innovative Network Management

BRKNMS-2514

Performance Assurance: An integrated Approach to Managing Service Delivery in Enterprise

Networks

BRKNMS-2513 Best practices for cloud automation with Cisco Intelligent Automation software

BRKNMS-2202 Ethernet OAM Technical Overview and Deployment Scenarios

BRKNMS-2030 Onboard Automation with Cisco IOS Embedded Event Manager

Th

urs

da

y

BRKNMS-1036 Simplify Network Management and Reduce Operational Costs with Cisco Prime

BRKNMS-2511 Cisco Videoscape Management Solution

BRKNMS-2031 SYSLOG Design, Methodology and Best Practices

BRKNMS-1035 The NOC at CiscoLive


Visit the Cisco Store for Related Titles

http://theciscostores.com


Who Am I? Charles Spaugh

Assurance Service Operations

Manage the PNOC NMS Platform

Work closely with the Service Assurance NOC

Have managed multiple customer facing operation teams for Cisco ROS and Assurance Services

Worked as a client server and Cisco Voice engineer

- 15+ years of Operations, Design, and Implementation

9+ Years in law enforcement

[email protected]

Assurance Services Within Advanced Services

54


HUGEdata

High-speed cost-effective MPP database/analytics system

Processes, stores, and analyzes large amounts of data/reasonable cost/

short learning curve.

Supports: ANSI SQL 92, Views, DML statements (insert, update, delete)

Commodity hardware and operating systems

High security onsite, hosted, or cloud

Supports third-party BI, stats, and ETL

Standard database interfaces: ODBC, JDBC and native connections

Much lower total cost of ownership and expansion

Embedded R for statistics, regression, etc

55


Pentaho Data Integration (Kettle)

Thanks to its multi-threaded and massive parallel processing architecture, Pentaho Data Integration can complete the most complex data

transformations in seconds. Dynamic clustering capability of this

architecture further enhances its performance and provides clients an

enterprise-class data integration platform that fits the needs of today's

modern and big data requirements.

56


Pentaho Reporting

Pentaho has broad connectivity to a diverse set of relational and non-relational data sources. Through direct access or by using Pentaho Data

Integration for a full transformation, cleansing and preparation of data,

Pentaho Reporting seamlessly integrates and reports on multi-source

environments including:

Data warehouses, data marts, and operational data stores

All popular proprietary, open source, analytical, and NoSQL databases

Unstructured and semi-structure sources

Hadoop data via Hive and Hbase

Enterprise ERP or CRM application data

57


Flume

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. Its main

goal is to deliver data from applications to Hadoops HDFS. It has a simple and flexible architecture based on streaming data flows. It is robust

and fault tolerant with tunable reliability mechanisms and many failover

and recovery mechanisms. The system is centrally managed and allows

for intelligent dynamic management. It uses a simple extensible data

model that allows for online analytic applications.

58


Hadoop

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers

using a simple programming model. It is designed to scale up from single

servers to thousands of machines, each offering local computation and

storage. Rather than rely on hardware to deliver high-avaiability, the

library itself is designed to detect and handle failures at the application

layer, so delivering a highly-availabile service on top of a cluster of

computers, each of which may be prone to failures.

59


Hive

is a framework for running applications on large cluster built of commodity hardware. The Hadoop framework transparently provides applications

both reliability and data motion. Hadoop implements a computational

paradigm named Map/Reduce, where the application is divided into many

small fragments of work, each of which may be executed or reexecuted

on any node in the cluster. In addition, it provides a distributed file system

(HDFS) that stores data on the compute nodes, providing very high

aggregate bandwidth across the cluster. Both Map/Reduce and the

distributed file system are designed so that node failures are automatically

handled by the framework.

60


Sqoop

Sqoop (SQL-to-Hadoop) is a straightforward command-line tool with the following capabilities:

Imports individual tables or entire databases to files in HDFS

Generates Java classes to allow you to interact with your imported data

Provides the ability to import from SQL databases straight into your Hive data warehouse

61


InfiniDB

InfiniDB Enterprise delivers a future-proof data management infrastructure with the ability to scale from a single database node to a multi-node, MPP

deployment. With the power of InfiniDB Enterprise you have the ability to

handle anything from a small data mart to a large scale, highly available

data warehousing platform. Start with the InfiniDB column database and

youll never outgrow your database engine.

62


Fusion Charts

FusionCharts v3

Stunning charts for Web and Enterprise

FusionWidgets v3

Data visualization widgets for dashboards

FusionMaps v3

Interactive & data-driven flash maps

PowerCharts v3

Interactive charts for specialized domains

63


R Project

Revolution Analytics delivers advanced analytics software at half the cost of existing solutions. By building on open source Rthe worlds most powerful statistics softwarewith innovations in big data analysis, integration and user experience, Revolution Analytics meets the demands

and requirements of modern data-driven businesses.

64


MapInfo

Location Intelligence products unleash a new dimension of your data, revealing trends and patterns previously hidden in spreadsheets and

graphs. By understanding the spatial and geographic relationships of your

business intelligence, you can identify new customers, expand your

markets and make key decisions with confidence.

65

Documents

Network Management KPI's