Enterprise monitoring of Oracle on z/VM and Linux · •Discovery is an essential component of a Monitoring solution •Methods of Discovery: •Discovery via dedicated discovery

Mike Sine, IBM

[email protected]

Ben Stern

[email protected]

April 2016

Enterprise monitoring of Oracle on z/VM and Linux

mailto:[email protected]


2 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices

Special Notices and Trademarks

Special Notices

This presentation reflects the IBM Advanced Technical Skills organizations’ understanding of the technical topic. It was produced and reviewed by the members of the IBM Advanced Technical Skills organization. This document is presented “As-Is” and IBM does not assume responsibility for the statements expressed herein. It reflects the opinions of the IBM Advanced Technical Skills organization. These opinions are based on the author’s experiences. If you have questions about the contents of this document, please contact the author at [email protected] .

Trademarks

The following are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both.

IBM, the IBM logo, Candle, DB2, developerWorks, iSeries, Passport Advantage, pSeries, Redbooks, Tivoli Enterprise Console, WebSphere, z/OS, xSeries, zSeries, System z, z/VM.

A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml.

NetView, Tivoli and TME are registered trademarks and TME Enterprise is a trademark of Tivoli Systems, Inc. in the United States and/or other countries.

Microsoft, Windows, Windows NT, Internet Explorer, and the Windows logo are registered trademarks of Microsoft Corporation in the United States and/or other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

UNIX is a registered trademark in the United States and other countries licensed exclusively through The Open Group.

Intel and Pentium are registered trademarks and MMX, Pentium II Xeon and Pentium III Xeon are trademarks of Intel Corporation in the United States and/or other countries.

Other company, product and service names may be trademarks or service marks of others.



AGENDA

Introduction

Challenges Introduced by Virtualization and Cloud

Approaches to Managing Applications in z/VM and Linux

• Understanding the Application

• Synthetics

• End User Experience

• Change

• Analytics

• Transaction Tracking

• Middleware/Oracle

• z/VM and Linux

The nature & requirements of Applications are

evolving…

– The era of Cloud - requires digital engagement, rapid

delivery of client facing applications, and new

consumption models with focus on business outcomes

and end user experience.

– Hybrid Applications - Clients in most industries are

rapidly moving to models where their applications are

hosted in a mixture of On-premises infrastructure and

Cloud Infrastructure (public and private, IaaS and PaaS)

The nature of “managing the application” is evolving…

– End User Experience - No longer about managing the

infrastructure; rather managing the application to ensure

the optimal end user experience

– Analytics - clients are looking for analytics to solve

problems even faster

– Hybrid Management - As with Hybrid Applications,

management solutions are increasingly hybrid – hybrid

technologies and hybrid delivery models such as on-

premise and SaaS 4

Application Management – Driving Forces for Change

5

APM Video on YouTube

Reducing & Preventing Outages and Slowdowns for the 24/7 Hybrid Application World

Application Performance Management can help

Diagnose for quicker problem resolution

Identifyperformance issues beforehand

Isolate where the problem is occurring

1 2 3

Improve availability by 60-90%Reduce outages by 50% or more Diagnose problems 90% faster

Today’s Hybrid Application

End users

Devices

MiddlewareWeb Servers,

App Servers, …

BackendDataBases, Z Backends, …

Cloud Workloads

On Premises Workloads

Application Resourceson IaaS, PaaS (VMs,

containers, Cloud Foundry runtimes, etc)

Meeting the new Demands

Application Performance Management (APM)

https://www.youtube.com/watch?v=XBs_3JYxAcY

Systems & Hypervisors, Network, Storage

Middleware, Databases, Services

Applications: Cloud, mobile, Traditional

Topology:Tracks and displays

dependencies

Single Dashboardw/visibility to all

Diagnostics:(Deep Dive):

Traces/displays methods and timing to get to line of code causing delays

End User Monitoring:(Synthetic & Real)

Measures availability & response time from the user perspective, across geographical locations

Transaction Tracking:Gathers /displays response time across components so end-to end timing is understood

Reporting/Analytics:Metric history and trendingMetrics and logs analysisDynamic baselinesPredictive analysis

Resource Monitoring:Collects Metrics,

Propagates status

7

APM Capabilities Overview

Challenges Introduced by Virtualization and Cloud

• Heterogeneous Virtualized environments (most customers have at least 2 hypervisors)

– System Z (z/OS)

– System z (z/VM)

– VMware

– Microsoft Virtual Server/Hyper-V

• Business Applications span physical and virtualized environments

• Requires performance management insight into the application• Response Time Monitoring

• Deep Dive analysis of key applications and middleware

• Isolating which component of an application is slow

• Requires performance management insights into virtualized environments

• Overall resource utilization of servers?

• Resources allocated per VM?

• Resource utilization per VM, and how can I optimize it?

• How do I know if my application is performing slowly due to the hypervisor?

• Are the Storage or Network a bottleneck for my virtualized environment?

• Containers introduce a new level of complexity

• Dynamic Environments• Dynamic provisioning/de-provisioning

• Capacity on-demand

• Automated workload deployments such as PureApp

• Self-service requests for new VMs and applications/middleware

• Management tools must stay in sync with provisioned resources

• Images must be compliant with Security and Regulatory requirements

– IBM Power Systems

– Citrix

– Xen

– KVM

Challenges Introduced by Virtualization/Cloud

Approaches to Managing Applications in z/VM and Linux

• How do you know what’s deployed? • Build Monitoring into the Provisioning/Orchestration process• Discovery is an essential component of a Monitoring solution• Methods of Discovery:

• Discovery via dedicated discovery tool: Can discover OS’s, Network components, applications, middleware, and more importantly, discover the relationships between the components.

• Pros: Very deep discover into configuration details, drift, etc. Agentless discovery• Cons: Scheduled discovery mechanism

• Discovery via APM Transaction Tracking• Pros: Automatically discover transaction paths as application topologies

change. Real-time discovery• Cons: Only discovers application/middleware components. Additional

mechanisms needed for Network and Storage

• Ideally, combine real-time discovery of APM Transaction Tracking with a scheduled discovery tool

Understanding the Application

• Synthetic monitoring is essential

• Synthetic Monitoring Provides:• The ability to quickly determine whether performance problems are caused

by the network.

• Provides the ability to prove that SLAs are being met. Response Time Monitoring can’t do that unless there is constant workload

• Not just synthetics of the UI, but understanding the response time of APIs

• Imagine the following scenarios:• Application performance is great within the LAN, but across the entire WAN

• Performance is good in all locations except for one• Performance is bad

everywhere, but only the login is slow

• Performance is bad and all steps that require DB access are slow

Synthetic Monitoring

• If the application is performing well…the individual components are not nearly as important

• Understand end user experience from multiple perspectives• What is the overall performance

• What are individual users experiencing

• What is the experience at specific geographic locations

• Performance is bad everywhere, but only the login is slow

• Performance is bad and all steps that require DB access are slow

End User Experience

• 80% of problems are introduced by Change• Change can be topology changes, network changes, middleware

configuration changes, or changes in the workload• Showing change events in context with performance data provides

insight• As organizations move to DevOps, change events may be code pushes

Code Push

Change Event

Change…in the context of the Application

• Deploy infrastructure monitoring to monitor:• OS’s• Middleware• Hypervisors• Storage• Network• Etc.

• Leverage the best of Agent-based and Agentless monitoring• Leverage Analytics

• Defining static thresholds in a dynamic environment is difficult to maintain• Multi-Variant analytics provide unique insight into the environment

DB Cache Hit RatioResponse Time

Approach to Monitoring in Today’s World

MQ

DataPower

App Server

HTTP Server

Database Server

• Discover and monitor complex applications

• Understand the application topology

• Quickly isolate bottlenecks in the application

• Drill down into the suspect component to debug

End to End Transaction Tracking

• Monitor the Oracle database with low overhead

• Use remote monitoring for constrained servers

• Oracle system tables contain a wealth of performance data

• Monitor key resources such as:

• Tablespaces

• Cache Hit %

• Contention

• Archive Logs

• Rollback Segments

• Databases

• Instances

• Top SQL Statements

Oracle Monitoring

• SGA

• PGA

• RAC

• ASM

• Ensure that the monitoring tools are RAC aware

• Provide visibility into OS and Oracle metrics in-context. Frequently, disk latency or other OS metrics will impact Oracle

• Be able to easily navigate from the Oracle metrics to the underlying OS or hypervisor

• Provide consistent metrics across personas (Dev, IT Ops, Lines of Business, Oracle SME, etc.)

• Need access to both real-time and historical data

• Leverage Analytics

Oracle Monitoring

z/VM and Linux

Infrastructure Monitoring and Management


File Level backup and recovery for

Linux Virtual Machines

Simple, intuitive, graphical z/VM

management tool

Tivoli Storage Manager

Wave for z/VM

IBM Infrastructure Suite

Operations Manager for z/VM

OMEGAMON XE on z/VM and Linux

Performance monitoring of

z/VM and Linux guest

Facilitate automated operations,

take action based on events

Backup and restore full z/VM

environment

Backup and Restore Manager

for z/VM

Linux on System z

z/VM


AGENDA

Introduction

Monitoring Requirements

• Virtual Linux and z/VM performance considerations

• Don’t forget the hardware

• Integration from hardware – systems – applications Persistent

historical views

Enterprise Management


OMEGAMON XE on z/VM and Linux agents with ITM example

MonitoringInfrastructure

There are 3 types of agentsThere is one z/VM agent per z/VM LPARThere are two options for Linux

Linux smart agent running on Linux Guest,

Or, “agentless” option via SNMP

TEMS Hub

TEPS

TEPS DB

TEMS Data (Built in operational DB)

Warehouse

TEP Browser or Desktop

z/VM/CP

PTKCMD

(Guest)

Linux

(Guest)

MQ

TEMA

VM

TEMA

Linux

(Guest)

ORACLE

TEMA

Linux

(Guest)

Linux

TEMA

pHyp xHyp

AIX on

Power

Linux on

System x

AIX

TEMA

Linux

TEMA

Smart Agent or

SNMP

Highly scalable across the entire Enterprise

Standard / Best Practice Data Sources Availability Monitoring:

Process Monitoring

Windows Service Monitoring

Functionality Test WMI

Windows Performance Monitor (Perfmon)

Windows Event Log

SNMP versions 1, 2, and 3

Script

Windows/Unix/Linux

Java™ Management Extensions (JMX)

Common Information Model (CIM),

Best Practice Library contains several Agents Built with the Agent Builder

Creating installation packages for agents

Agentless monitoring - ITM Agent Builder

Develop Customized Agent

Solutions in Minutes!

2424

Custom Monitoring with the Agent Builder

• Create local and remote monitoring agents• Agent Builder supports many industry standard monitoring interfaces• Extremely high quality by leveraging the same data collectors using in IBM’s

out of the box agentsMonitoring

Server

Agent Builder

Based Agent

Agent-less: SNMP

V1, V2C, and V3

Agent-less:

JMX

Log

FileScripts

WMI,

Perfmon,

Event Log

Availability

Agent-less:

WMI, Perfmon,

Event Log

Agent-less: CIM

Agent-less:

JDBC

Agent-less:

HTTP/HTTPS

Agent-less:

ICMP

Agent-less:

SSH/RXA


Tips—Overall Health of Your System

At a quick glance you can see the

%CPU usage, what your

overcommit ratio is, the number of

users in a wait state, and paging rates of all your z/VM systems


Resource Constraint Analysis (Waits)


Do not ignore the hardware!

Just because Linux resources are virtual, do not ignore the

hardware!

• Hardware is another potential layer of shared resources.

• LPAR weight, CPU sharing, LPAR load, and other attributes need

to be monitored for overall system performance.

• The measurement should include the entire CEC and not just the

LPAR hosting z/VM.


LPAR Utilization Workspace


Processor by LPAR name workspace


CPC workspace


History On-Demand with Persistent Historical Views

This makes it easier to see anomalies, or

match spikes. Capturing performance data

as a base line is a must:

General history data – business as usual.

Detailed raw monitor data prior to and

following any major changes.

Ability to review attributes of a past

incident through the enterprise view!

On-Demand through the Portal or Batch


On-Demand: Persistent Historical Views


On-Demand: Persistent Historical Views


Max and Avg CPU example:


Avg Linux Memory breakdown example:

IBM Wave for z/VMIBM Wave for z/VM provides the graphical interface that simplifies and helps to automate the management of z/VM and Linux on z System virtual servers.

A simple, intuitive graphical tool providing management, provisioning, and automation for a z/VM environment, supporting Linux virtual

servers.

Allows delegation of administrative

capabilities to the appropriate teams

Simplifies and Automates tasks

Provisions virtual resources

(Guests, Network, Storage)

Supports advanced z/VM

capabilities such as Single System

Image and Live Guest Relocation

Monitors and manages virtual

servers and resources from a single

graphical interface

36

Operations Manager for z/VM

Operations Manager

for z/VM

Increase productivity

Authorized users to view and interact with monitored virtual machines without logging onto them

Multiple users view/interact with a virtual machine simultaneously

Improve system availability

Monitor virtual machines and processes

Take automated actions based on console messages

Reduce problems due to operator error

Service Virtual

Machine being

monitored

Service Virtual

Machine being

monitored

Console monitoring

Console monitoring

Take action

• View & interact

with consoles

• View spool files

Automation Routine activities done more effectively with

minimal operations staff Schedule tasks to occur on a regular basis

Schedule tasks

Monitor page and spool usage

Respond to system events

(user state changes)

Integration Fulfill take action requests from performance monitoring

products (e.g. OMEGAMON XE on z/VM and Linux) Send alerts to email, central event management systems

(e.g. Netcool\OMNIbus), etc.

Idle monitor

37

38

Using Backup and Restore Manager with Spectrum Protect

z/VMBackup and Restore

Manager

Other guest

FBA or ECKD

DASD

Spectrum

Server

Spectrum

Client CMS minidisk

and SFS files

dirA/file1.ext

dirB/file2.ext

dirC/file3.ext

FN FT FM

FN FT FM

FN FT FM

Linux

Combine the solutions for file

level and volume level recoverySpectrum

Server

Spectrum

Client

Linux

38

Backup and Restore Solution for z/VM and Linux


Thank You

MerciGrazie

Gracias

Obrigado

Danke

Japanese

English

French

Russian

German

Italian

Spanish

Brazilian Portuguese

Arabic

Traditional Chinese

Simplified Chinese

Tamil

Thai

KoreanHindi

Documents

Enterprise monitoring of Oracle on z/VM and Linux · •Discovery is an essential component of a Monitoring solution •Methods of Discovery: •Discovery via dedicated discovery