40

IT Problems & Problem Management

Embed Size (px)

Citation preview

Page 1: IT Problems & Problem Management
Page 2: IT Problems & Problem Management

Session Date & Time• Date: Wednesday, June 29, 2016

Time: 1100-1145Location: Tuscany Ballroom

Page 3: IT Problems & Problem Management

• Bill Alderson responds to Information Technology high visibility, high stakes technical problems. Network outage, slowness, slow applications or disasters affecting government and commercial Information Technology Enterprise environments. ABC News told the story of how Bill and his team helped restore communications at the Pentagon immediately following 911. Bill assisted with six deployments to Iraq and Afghanistan requested by Army G2, Joint Chiefs and US Central Command diagnosing Biometrics and others critical systems. One of his missions is to help executives and technologists see both technical and leadership root causes that can be obviated through common sense best practices.

Page 4: IT Problems & Problem Management

Bill Alderson Infographic Bio

Page 5: IT Problems & Problem Management

• Deep packet analysis remains essential for definitive irrefutable diagnosis and optimization of complex systems. Bill demonstrates the tools, techniques and methods used to annotate complex technology findings so that technologists, managers, executives and vendors can agree on root cause. Once the problem is identified and agreed upon the true pinpoint mitigation can begin. The days of shotgun style "forklift wholesale upgrades" on everything have passed. We must optimize existing assets allowing them to perform well.

Bill has proven ability to optimize large scale networks and applications from experience in analyzing the Pentagon immediately following 911, analysis of Biometrics applications across Iraq and Afghanistan, numerous optimizations of Joint Chiefs of Staff and OSD network analysis. Experience from analysis of the largest 100 commercial enterprise networks such as Stock Exchanges, Financial, Insurance and Healthcare institutions will be demonstrated with annotated examples for CIO, Executives and top level technologists.

Page 6: IT Problems & Problem Management

IT Critical Problem ResolutionTechnology and Psychology

[email protected]

Page 7: IT Problems & Problem Management

“Swiss Army Knife” Portfolio of Tools

Select Well.Avoid SpendingOnly on “Suites”

All-in-one-toolsAlthough easier to “buy”

don’t solve many problems.They leave you “broke and broken”

with a gold plated toolset.

Page 8: IT Problems & Problem Management

Optimization Troubleshooting Phases

Page 9: IT Problems & Problem Management

Preparation & Setup

Page 10: IT Problems & Problem Management

Analysis & Iteration

Page 11: IT Problems & Problem Management

Reporting & Presentation

Page 12: IT Problems & Problem Management

Problem Management

Page 13: IT Problems & Problem Management

Down - Intermittent - Slow

Page 14: IT Problems & Problem Management

Technical vs. Leadership Root Cause

Page 15: IT Problems & Problem Management

The Needle

Page 16: IT Problems & Problem Management

The Environment

Page 17: IT Problems & Problem Management

Packet Traces

Page 18: IT Problems & Problem Management

Store Every Packet? Who’s can and is going to analyze them and when?

Page 19: IT Problems & Problem Management

Finding The Stack With The Problem

Page 20: IT Problems & Problem Management

Finding The Needle

Page 21: IT Problems & Problem Management

Measured at the Server

Fast TCP connect time. Fast Ack from F5 does not show true client response time which is why Apalytics provided Internet Monitoring.

1.4 second Get response is very slow which is why detailed platform and application analysis was performed.

The 2nd & 3rd Gets were fast at 1 millisecond proving some commands are fast.

Page 22: IT Problems & Problem Management

CF Longest Requests

1,958,266ms = ~32 minutes from one request391,692ms = ~7 minutes

Page 23: IT Problems & Problem Management

Page Analysis from the Internet DNS does not play a role in slowness. Connection time varies and at time approaches 200 milliseconds which can be at the platform, internet, network, load balancer or firewalls. Connection delay analysis will require multiple capture points to definitively pinpoint and should be considered when multi-point capture test points can be configured at the Security Tap devices. But that is not material for improvement of this application at this timeFirst byte time is the most concerning issue in the infrastructure. Last byte time is also a concern as it appears that platform TCP/IP stack services are slow to move data out onto the wire after the first byte has started. It may also be that platform improvements may improve both response times and output speed. Page load time is a composite of all elements of the page that must come together to provide the user with the visual page and the main context of the query. This too is concerning, but it is caused by the slowness of the individual components of the page as they add serially to the response time which are represented in the main concerns. An example of the total page would be small visual images and data making up the user interface view (i.e., logos) that are not part of a computational or lookup, but rather a static image that should be served rapidly by the server.

Page 24: IT Problems & Problem Management

Network Intrinsic Application Analysis

Page 25: IT Problems & Problem Management

Multi-tier Analysis

Page 26: IT Problems & Problem Management

Multi-Tier Identification

Page 27: IT Problems & Problem Management

Application Monitoring Design Phase

Page 28: IT Problems & Problem Management

Multi-tier Macro vs. Micro

Event

Process

Net-Ser-Tr-Sw-Q

Security Auth

User ClickClientNetworkWebSvrNetworkAppSrvNetworkSQLSvrNetworkAppSvrNetworkMainframeNetworkAppSvrNetworkWebSvrNetworkClientUser Display UpdateMacro Response

Time

Micro Response Time

Page 29: IT Problems & Problem Management

HTTP Post from client

Web1 Middlewa

re 155ms

HTTP / SQL Multi-tier 1

Page 30: IT Problems & Problem Management

Back to clientWith HTTP

SQL Calls completeQuery and returns Rows to Web1

SQL Calls finish .497SQL Call start -.231

SQL Resp Time =.266

Web1 Middleware

12ms

HTTP / SQL Multi-tier 2

Page 31: IT Problems & Problem Management

Logon A is 72 milliseconds…

Logon B is 420 milliseconds!

Oracle Logon Slow

Page 32: IT Problems & Problem Management

Micro-Analysis Phase

Web App I/F #1&2 SQL TransLogger MF#1 MF#2 Time Breakdown

Page 33: IT Problems & Problem Management

TCP Satellite Retrans 3.5 Seconds

Page 34: IT Problems & Problem Management

Processing Analysis

Page 35: IT Problems & Problem Management

Packet Loss Analysis

Page 36: IT Problems & Problem Management

Citrix Session Abort Signature “Chernobyl Packet”

The packet that evidenced a problem on a Citrix server. This pattern was used as a signature on the Infinistream Sniffers to find these problems until they were remediated.

Prior to this users were stuck in this cycle for hours.

Page 37: IT Problems & Problem Management

Citrix User Filer Access Error Details

Page 38: IT Problems & Problem Management

Blind vs. Pinpoint Upgrades

Blind Upgrade = Shotgun Approach = Forklift Upgrade

Page 39: IT Problems & Problem Management

Root Cause Optimization

Definitive Root Cause Analysis Pinpoint Cause Measure ROI PotentialPinpoint Purchases Validate & Prove ROI Award Innovation

OptimizationRoot

Cause Analysis

Page 40: IT Problems & Problem Management

IT Critical Problem ResolutionTechnology and Psychology

[email protected]