View
243
Download
0
Category
Preview:
Citation preview
Mike Sine, IBM
sine@us.ibm.com
Ben Stern
bstern@us.ibm.com
April 2016
Enterprise monitoring of Oracle on z/VM and Linux
2 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
Special Notices and Trademarks
Special Notices
This presentation reflects the IBM Advanced Technical Skills organizations’ understanding of the technical topic. It was produced and reviewed by the members of the IBM Advanced Technical Skills organization. This document is presented “As-Is” and IBM does not assume responsibility for the statements expressed herein. It reflects the opinions of the IBM Advanced Technical Skills organization. These opinions are based on the author’s experiences. If you have questions about the contents of this document, please contact the author at sine@us.ibm.com .
Trademarks
The following are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both.
IBM, the IBM logo, Candle, DB2, developerWorks, iSeries, Passport Advantage, pSeries, Redbooks, Tivoli Enterprise Console, WebSphere, z/OS, xSeries, zSeries, System z, z/VM.
A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml.
NetView, Tivoli and TME are registered trademarks and TME Enterprise is a trademark of Tivoli Systems, Inc. in the United States and/or other countries.
Microsoft, Windows, Windows NT, Internet Explorer, and the Windows logo are registered trademarks of Microsoft Corporation in the United States and/or other countries.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
UNIX is a registered trademark in the United States and other countries licensed exclusively through The Open Group.
Intel and Pentium are registered trademarks and MMX, Pentium II Xeon and Pentium III Xeon are trademarks of Intel Corporation in the United States and/or other countries.
Other company, product and service names may be trademarks or service marks of others.
3 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
AGENDA
Introduction
Challenges Introduced by Virtualization and Cloud
Approaches to Managing Applications in z/VM and Linux
• Understanding the Application
• Synthetics
• End User Experience
• Change
• Analytics
• Transaction Tracking
• Middleware/Oracle
• z/VM and Linux
The nature & requirements of Applications are
evolving…
– The era of Cloud - requires digital engagement, rapid
delivery of client facing applications, and new
consumption models with focus on business outcomes
and end user experience.
– Hybrid Applications - Clients in most industries are
rapidly moving to models where their applications are
hosted in a mixture of On-premises infrastructure and
Cloud Infrastructure (public and private, IaaS and PaaS)
The nature of “managing the application” is evolving…
– End User Experience - No longer about managing the
infrastructure; rather managing the application to ensure
the optimal end user experience
– Analytics - clients are looking for analytics to solve
problems even faster
– Hybrid Management - As with Hybrid Applications,
management solutions are increasingly hybrid – hybrid
technologies and hybrid delivery models such as on-
premise and SaaS 4
Application Management – Driving Forces for Change
5
APM Video on YouTube
Reducing & Preventing Outages and Slowdowns for the 24/7 Hybrid Application World
Application Performance Management can help
Diagnose for quicker problem resolution
Identifyperformance issues beforehand
Isolate where the problem is occurring
1 2 3
Improve availability by 60-90%Reduce outages by 50% or more Diagnose problems 90% faster
Today’s Hybrid Application
End users
Devices
MiddlewareWeb Servers,
App Servers, …
BackendDataBases, Z Backends, …
Cloud Workloads
On Premises Workloads
Application Resourceson IaaS, PaaS (VMs,
containers, Cloud Foundry runtimes, etc)
Meeting the new Demands
Application Performance Management (APM)
Systems & Hypervisors, Network, Storage
Middleware, Databases, Services
Applications: Cloud, mobile, Traditional
Topology:Tracks and displays
dependencies
Single Dashboardw/visibility to all
Diagnostics:(Deep Dive):
Traces/displays methods and timing to get to line of code causing delays
End User Monitoring:(Synthetic & Real)
Measures availability & response time from the user perspective, across geographical locations
Transaction Tracking:Gathers /displays response time across components so end-to end timing is understood
Reporting/Analytics:Metric history and trendingMetrics and logs analysisDynamic baselinesPredictive analysis
Resource Monitoring:Collects Metrics,
Propagates status
7
APM Capabilities Overview
Challenges Introduced by Virtualization and Cloud
• Heterogeneous Virtualized environments (most customers have at least 2 hypervisors)
– System Z (z/OS)
– System z (z/VM)
– VMware
– Microsoft Virtual Server/Hyper-V
• Business Applications span physical and virtualized environments
• Requires performance management insight into the application• Response Time Monitoring
• Deep Dive analysis of key applications and middleware
• Isolating which component of an application is slow
• Requires performance management insights into virtualized environments
• Overall resource utilization of servers?
• Resources allocated per VM?
• Resource utilization per VM, and how can I optimize it?
• How do I know if my application is performing slowly due to the hypervisor?
• Are the Storage or Network a bottleneck for my virtualized environment?
• Containers introduce a new level of complexity
• Dynamic Environments• Dynamic provisioning/de-provisioning
• Capacity on-demand
• Automated workload deployments such as PureApp
• Self-service requests for new VMs and applications/middleware
• Management tools must stay in sync with provisioned resources
• Images must be compliant with Security and Regulatory requirements
– IBM Power Systems
– Citrix
– Xen
– KVM
Challenges Introduced by Virtualization/Cloud
Approaches to Managing Applications in z/VM and Linux
• How do you know what’s deployed? • Build Monitoring into the Provisioning/Orchestration process• Discovery is an essential component of a Monitoring solution• Methods of Discovery:
• Discovery via dedicated discovery tool: Can discover OS’s, Network components, applications, middleware, and more importantly, discover the relationships between the components.
• Pros: Very deep discover into configuration details, drift, etc. Agentless discovery• Cons: Scheduled discovery mechanism
• Discovery via APM Transaction Tracking• Pros: Automatically discover transaction paths as application topologies
change. Real-time discovery• Cons: Only discovers application/middleware components. Additional
mechanisms needed for Network and Storage
• Ideally, combine real-time discovery of APM Transaction Tracking with a scheduled discovery tool
Understanding the Application
• Synthetic monitoring is essential
• Synthetic Monitoring Provides:• The ability to quickly determine whether performance problems are caused
by the network.
• Provides the ability to prove that SLAs are being met. Response Time Monitoring can’t do that unless there is constant workload
• Not just synthetics of the UI, but understanding the response time of APIs
• Imagine the following scenarios:• Application performance is great within the LAN, but across the entire WAN
• Performance is good in all locations except for one• Performance is bad
everywhere, but only the login is slow
• Performance is bad and all steps that require DB access are slow
Synthetic Monitoring
• If the application is performing well…the individual components are not nearly as important
• Understand end user experience from multiple perspectives• What is the overall performance
• What are individual users experiencing
• What is the experience at specific geographic locations
• Performance is bad everywhere, but only the login is slow
• Performance is bad and all steps that require DB access are slow
End User Experience
• 80% of problems are introduced by Change• Change can be topology changes, network changes, middleware
configuration changes, or changes in the workload• Showing change events in context with performance data provides
insight• As organizations move to DevOps, change events may be code pushes
Code Push
Change Event
Change…in the context of the Application
• Deploy infrastructure monitoring to monitor:• OS’s• Middleware• Hypervisors• Storage• Network• Etc.
• Leverage the best of Agent-based and Agentless monitoring• Leverage Analytics
• Defining static thresholds in a dynamic environment is difficult to maintain• Multi-Variant analytics provide unique insight into the environment
DB Cache Hit RatioResponse Time
Approach to Monitoring in Today’s World
MQ
DataPower
App Server
HTTP Server
Database Server
• Discover and monitor complex applications
• Understand the application topology
• Quickly isolate bottlenecks in the application
• Drill down into the suspect component to debug
End to End Transaction Tracking
• Monitor the Oracle database with low overhead
• Use remote monitoring for constrained servers
• Oracle system tables contain a wealth of performance data
• Monitor key resources such as:
• Tablespaces
• Cache Hit %
• Contention
• Archive Logs
• Rollback Segments
• Databases
• Instances
• Top SQL Statements
Oracle Monitoring
• SGA
• PGA
• RAC
• ASM
• Ensure that the monitoring tools are RAC aware
• Provide visibility into OS and Oracle metrics in-context. Frequently, disk latency or other OS metrics will impact Oracle
• Be able to easily navigate from the Oracle metrics to the underlying OS or hypervisor
• Provide consistent metrics across personas (Dev, IT Ops, Lines of Business, Oracle SME, etc.)
• Need access to both real-time and historical data
• Leverage Analytics
Oracle Monitoring
z/VM and Linux
Infrastructure Monitoring and Management
20 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
File Level backup and recovery for
Linux Virtual Machines
Simple, intuitive, graphical z/VM
management tool
Tivoli Storage Manager
Wave for z/VM
IBM Infrastructure Suite
Operations Manager for z/VM
OMEGAMON XE on z/VM and Linux
Performance monitoring of
z/VM and Linux guest
Facilitate automated operations,
take action based on events
Backup and restore full z/VM
environment
Backup and Restore Manager
for z/VM
Linux on System z
z/VM
21 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
AGENDA
Introduction
Monitoring Requirements
• Virtual Linux and z/VM performance considerations
• Don’t forget the hardware
• Integration from hardware – systems – applications Persistent
historical views
Enterprise Management
22 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
OMEGAMON XE on z/VM and Linux agents with ITM example
MonitoringInfrastructure
There are 3 types of agentsThere is one z/VM agent per z/VM LPARThere are two options for Linux
Linux smart agent running on Linux Guest,
Or, “agentless” option via SNMP
TEMS Hub
TEPS
TEPS DB
TEMS Data (Built in operational DB)
Warehouse
TEP Browser or Desktop
z/VM/CP
PTKCMD
(Guest)
Linux
(Guest)
MQ
TEMA
VM
TEMA
Linux
(Guest)
ORACLE
TEMA
Linux
(Guest)
Linux
TEMA
pHyp xHyp
AIX on
Power
Linux on
System x
AIX
TEMA
Linux
TEMA
Smart Agent or
SNMP
Highly scalable across the entire Enterprise
Standard / Best Practice Data Sources Availability Monitoring:
Process Monitoring
Windows Service Monitoring
Functionality Test WMI
Windows Performance Monitor (Perfmon)
Windows Event Log
SNMP versions 1, 2, and 3
Script
Windows/Unix/Linux
Java™ Management Extensions (JMX)
Common Information Model (CIM),
Best Practice Library contains several Agents Built with the Agent Builder
Creating installation packages for agents
Agentless monitoring - ITM Agent Builder
Develop Customized Agent
Solutions in Minutes!
2424
Custom Monitoring with the Agent Builder
• Create local and remote monitoring agents• Agent Builder supports many industry standard monitoring interfaces• Extremely high quality by leveraging the same data collectors using in IBM’s
out of the box agentsMonitoring
Server
Agent Builder
Based Agent
Agent-less: SNMP
V1, V2C, and V3
Agent-less:
JMX
Log
FileScripts
WMI,
Perfmon,
Event Log
Availability
Agent-less:
WMI, Perfmon,
Event Log
Agent-less: CIM
Agent-less:
JDBC
Agent-less:
HTTP/HTTPS
Agent-less:
ICMP
Agent-less:
SSH/RXA
25 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
Tips—Overall Health of Your System
At a quick glance you can see the
%CPU usage, what your
overcommit ratio is, the number of
users in a wait state, and paging rates of all your z/VM systems
26 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
Resource Constraint Analysis (Waits)
27 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
Do not ignore the hardware!
Just because Linux resources are virtual, do not ignore the
hardware!
• Hardware is another potential layer of shared resources.
• LPAR weight, CPU sharing, LPAR load, and other attributes need
to be monitored for overall system performance.
• The measurement should include the entire CEC and not just the
LPAR hosting z/VM.
28 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
LPAR Utilization Workspace
29 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
Processor by LPAR name workspace
30 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
CPC workspace
31 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
History On-Demand with Persistent Historical Views
This makes it easier to see anomalies, or
match spikes. Capturing performance data
as a base line is a must:
General history data – business as usual.
Detailed raw monitor data prior to and
following any major changes.
Ability to review attributes of a past
incident through the enterprise view!
On-Demand through the Portal or Batch
32 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
On-Demand: Persistent Historical Views
33 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
On-Demand: Persistent Historical Views
34 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
Max and Avg CPU example:
35 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
Avg Linux Memory breakdown example:
IBM Wave for z/VMIBM Wave for z/VM provides the graphical interface that simplifies and helps to automate the management of z/VM and Linux on z System virtual servers.
A simple, intuitive graphical tool providing management, provisioning, and automation for a z/VM environment, supporting Linux virtual
servers.
Allows delegation of administrative
capabilities to the appropriate teams
Simplifies and Automates tasks
Provisions virtual resources
(Guests, Network, Storage)
Supports advanced z/VM
capabilities such as Single System
Image and Live Guest Relocation
Monitors and manages virtual
servers and resources from a single
graphical interface
36
Operations Manager for z/VM
Operations Manager
for z/VM
Increase productivity
Authorized users to view and interact with monitored virtual machines without logging onto them
Multiple users view/interact with a virtual machine simultaneously
Improve system availability
Monitor virtual machines and processes
Take automated actions based on console messages
Reduce problems due to operator error
Service Virtual
Machine being
monitored
Service Virtual
Machine being
monitored
Console monitoring
Console monitoring
Take action
• View & interact
with consoles
• View spool files
Automation Routine activities done more effectively with
minimal operations staff Schedule tasks to occur on a regular basis
Schedule tasks
Monitor page and spool usage
Respond to system events
(user state changes)
Integration Fulfill take action requests from performance monitoring
products (e.g. OMEGAMON XE on z/VM and Linux) Send alerts to email, central event management systems
(e.g. Netcool\OMNIbus), etc.
Idle monitor
37
38
Using Backup and Restore Manager with Spectrum Protect
z/VMBackup and Restore
Manager
Other guest
FBA or ECKD
DASD
Spectrum
Server
Spectrum
Client CMS minidisk
and SFS files
dirA/file1.ext
dirB/file2.ext
dirC/file3.ext
FN FT FM
FN FT FM
FN FT FM
Linux
Combine the solutions for file
level and volume level recoverySpectrum
Server
Spectrum
Client
Linux
38
Backup and Restore Solution for z/VM and Linux
39 © 2013, 2015 IBM CorporationManaging z/VM and Linux Performance Best Practices
Thank You
MerciGrazie
Gracias
Obrigado
Danke
Japanese
English
French
Russian
German
Italian
Spanish
Brazilian Portuguese
Arabic
Traditional Chinese
Simplified Chinese
Tamil
Thai
KoreanHindi
Recommended