Using the IBM Tivoli Universal Agent to Enhance z/OS ... · PDF fileUsing the IBM Tivoli Universal Agent to Enhance ... Using the IBM Tivoli Universal Agent to Enhance z/OS Monitoring

1

IBM Advanced Technical Skills

© 2010 IBM Corporation

Using the IBM Tivoli Universal Agent to Enhance z/OS Monitoring

Mike Bonett, IBM Advanced Technical Skills Gaithersburg, MD

The IBM Tivoli Universal Agent is a hidden treasure that allows monitoring of almost any technology environment to be integrated into the OMEGAMON and IBM Tivoli Monitoring architecture. This presentation introduces the Universal Agent and provides technical examples of how it can expand OMEGAMON monitoring into additional z/OS areas beyond the OMEGAMON and IBM Tivoli Composite Application Management (ITCAM) product functions, and enhance the consolidated view of the environment provided by the IBM Tivoli Enterprise Portal.

2


© 2010 IBM CorporationUsing the IBM Tivoli Universal Agent to Enhance z/OS Monitoring

Trademarks

AF/OPERATOR®

AF/REMOTE®

AIX®

AS/400®

BladeCenter

Candle Management Server®

CandleNet®

CandleNet Portal®

CICS®

CICSPlex®

Database 2

DB2®

Domino®

e-business(logo)®

e(logo)e-business®

e-business (logo)

e-business on demand

eServer

Enterprise Storage Server®

GDPS®

Geographically Dispersed Parallel Sysplex

HACMP

HACMP/6000

IBM®

ibm.com®

IBM TotalStorage Proven

IMS

MQSeries®

MVS

MVS/ESA

MVS/XA

Netfinity®

Netfinity Manager

NetView®

OMEGACENTER®

OMEGAMON®

OMEGAMON II®

OMEGAVIEW®

OMEGAVIEW II®

OS/390®

OS/400®

Parallel Sysplex®

RACF®

Redbooks

S/390®

System/390®

ThinkPad®

Tivoli®

Tivoli (logo)®

Tivoli Enterprise

Tivoli Enterprise Console®

Tivoli Management Environment®

TotalStorage®

VM/ESA®

VTAM®

WebSphere®

z/OS®

z/VM®

zSeries®

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel Inside (logos), MMX and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Those trademarks followed by ® are registered trademarks of IBM in the United States, other countries, or both;

all others are trademarks or common law marks of IBM in the United States, other countries, or both.

These trademarks represent the terms that will be used in the presentation.

3



Agenda

� Universal Agent Overview

� Monitoring Architecture

� Universal Agent Data Providers

� z/OS Specific Examples

The following topics will be covered:

•An overview of the Universal Agent, its purpose, and its functions.

•The monitoring architecture to Universal Agent operates within.

•Data Providers – the means by which information is sent to or retrieved by the Universal Agent.

•Specific examples showing the use of the Universal Agent to monitor z/OS components.

4



OMEGAMONXE & II

forDB2 z/OSDB2plex

OMEGAMONXE For WAS

on OS/390

OMEGAMONXE

Distributed&

Databases

OMEGAMONXEfor

LinuxSystem z

forVM

DB2 PEDB2 PM

BPA

CICS PMCICS PA

ITM

TivoliAccess

Mgr for BI

IMS PMIMS PA

VMPerformance

Toolkit

ITM

WSAM

DB2 WASUSSIMSCICS WebSphere

MQ & WBI

z/OS NetworkPerformance

z/OSStorage

TivoliStorage

Optimizer

OMEGAMONXE & II

forMainframeNetworks

OMEGAMONXE & II

forCICS

CICSPlex

CommandCenter

forCICS

OMEGAMONXE & II

forIMS

IMSPlex

OMEGAMONXE for

WebSphereMQ

on OS/390

OMEGAMONWebSphereIntegration

BrokersConfig Mgr

CommandCenter

MQSeries

Distributed

OMEGAMONXEfor

Storage

OMEGAMONII

for SMS

ITMNPNPM

NPM/IP

IBM TivoliOMEGAMON

XE forDB2

PM or PE

IBM TivoliOMEGAMON

XE forCICS and CICS TG

IBM TivoliOMEGAMON

XE forIMS

IBM TivoliMonitoring

OperatingSystems

Apps

Databases

Virtual Servers

IBM TivoliOMEGAMON

XE forStorage

IBM TivoliOMEGAMON

XE forMessaging

IBM TivoliOMEGAMON

XE forMainframeNetworks

IBM TivoliOMEGAMON

XE forz/OS

OMEGAMONz/OS

ManagementConsole

IBMTivoli

CompositeApplication

Mgmtfor WASon z/OS

ITCAM Transactions

ITCAM SOA

IBM TivoliOMEGAMON

XE forz/VM

& Linux

z/VM & Linux

� Many products

� Many monitored environments

� Both z/OS and distributed

� But… provided agents cannot account for every desired component or data source

ITM/OMEGAMON/ITCAM Monitoring

The IBM Tivoli Monitoring (ITM), OMEGAMON, and IBM Tivoli Composite Application Management (ITCAM) products provide many different agents to monitor and manage both z/OS and distributed components. These agents cover the major operating systems, subsystems, and middleware components, and support the ability to manage the environment through a single logical view – the Tivoli Enterprise Portal (TEP).

Other products besides the ITCAM and OMEGAMON products provide agents to integrate some of their information into the TEP. Examples include IBM Tivoli NetView on z/OS, IBM Tivoli System Automation on z/OS, IBM Tivoli Workload Scheduler, and IBM Tivoli Information/Management.

However, the product agents cannot cover every possible source of data to show in the TEP. Information from custom applications, business related information, other IBM and non-IBM software products, and technology for which no agent currently exists may be beneficial to integrate and use within the logical TEP view.

5



IBM Tivoli Universal Agent

� A “generic agent” provided with IBM Tivoli Monitoring

� Can be configured to monitor many different data sources

� Information can be viewed in both real-time and historical workspaces on the Tivoli Enterprise Portal

� Data can be managed with Tivoli Enterprise Portal monitoring situations and automation policies, just as data from other Tivoli Enterprise Monitoring Agents

� Extends the performance and availability management capabilitiesof IBM Tivoli Monitoring to applications and operating systems not otherwise covered by other IBM Tivoli Monitoring agents

� Further enables a single point of monitoring and management for enterprise resources

The IBM Tivoli Universal Agent is provided to accommodate as many additional sources as possible. It can be configured to monitor and retrieve data from almost any source – and methods of getting data to the Universal Agent will be covered in this presentation.

The data from the Universal Agent can be used just as any other agent data in the TEP. It can be shown in both real time and historical workspaces. It can be used to create custom portal views, situations, and policies. The data can trigger commands to invoke automated actions.

Information can come from many different enterprise locations to the Universal Agent, and allows the performance and availability management capabilities of IBM Tivoli Monitoring to be extended beyond the standard product agents. These functions further support using the TEP as a single point of monitoring and management for enterprise resources.

6



Architecture and Installation

� Supported execution platforms

– Windows, UNIX (AIX, HP-UX, Solaris), Linux (Intel, System z), i5/OS

– See the ITM Installation/Setup Guide for specific supported releases

� Flexible placement within the monitoring infrastructure

– TEPS platform

– TEMS platform

– Dedicated/shared platform

– Multiple agents can be deployed within a TEPS/TEMS environment

HubTEMS

TDW

Out of the box Monitors

ITM, OMEGAMON, ITCAM, Product provided

FileSocketAPISNMP

PostODBCScript HTTP Data Provider

WHProxy

Remote TEMS 1

Remote TEMS 2

AgentlessMonitors

TEPS

CLISOAPODBC

CLISOAPODBC

TEPConsole

SQL Interface3rd Party Reporting

SQL Interface3rd Party ReportingUA

The Universal Agent is provided with the ITM base code. It can be installed on Windows, UNIX, Linux (including Linux on System z), and i5OS platforms. The specific platform releases supported are documented in the current ITM release Installation/Setup Guide.

The agent fits into the TEP-TEMS (Tivoli Enterprise Monitoring Server) – Agent architecture. Once installed, it connects to a TEMS (either Hub or Remote). Placement of the Universal agent is flexible – it can be installed on the TEPS platform, TEMS platform (except on z/OS), or on a separate supported platform by itself or with other agents. Multiple Universal Agents can be installed within an TEPS-TEMS environment; this allows agent monitoring and usage to be separated as desired (e.g. based on location, or types of resources being monitored).

Once the agent is installed and operational, it receives and request data through a variety of ways. The functions that provide this capability are Data Providers. These providers allow a level of “agentless monitoring”, as they can be used to monitor a resource without placing an agent directly on that resource.

7



Data Providers

� Functions with the Universal Agent that provide methods of receiving data

� Allows the most convenient method for sending data to be chosen by the installation

Tivoli Enterprise Monitoring Server

UA IntelligentRemote Agent

Data Provider(s)

Data Source

UniversalAgent

SolutionInstalledComponents

A “Data Provider” is the logical path between the data source and the Universal Agent. There are different types of Data Providers, to provide flexible options when choosing the type of data source to monitor. The combination of the data source, Data Provider, and Universal Agent functions when connected to the TEMS can provide a total monitoring solution for the target resource.

8



Data Provider TypesData SourceData Provider

Consolidates API, Socket, File, and Script data providers (default data provider)

ASFS

TCP/IP sockets (custom data)Socket

SNMP manager ( network discovery, trap monitoring, and MIB data collection, etc.). .

SNMP

Any script or program that sends results to standard output.

Script

TCP/IP sockets (predefined data) Post

ODBC-compliant databases ODBC

Monitors Internet URLs HTTP

Sequential files File

Universal Agent API client API Server

This table describes the Data Providers:

•The API Server receives information for the provided Universal Agent API client. The API client can br installed on any supported UA platform, and provides command line and C programming capabilities to send data to the API Server Data Provider.

•The File Data Provider reads sequential files (usually files that are written to on some regular basis, such as a log file), and captures the desired data.

•The HTTP Data Provider monitors Internet URLs, by querying them on a designated interval and capturing data such as response time and data sizes.

•The ODBC Data Provider queries ODBC-compliant databases and captures the results of the query. This data provided is only support when the Universal Agent is on a Windows platform.

•The Post Data Provider allows data to be sent to the Universal Agent via TCP/IP sockets, in a specific predefined format. Incoming data must match the format.

•The Script Data Provider periodically executes a designated script on the Universal Agent platform, and the output from the script (send to standard output) is captured by the Universal Agent.

•The SNMP data provider turns the Universal Agent into an SNMP manager, to perform network discovery, trap monitoring, MIB data collection, etc.

•The Socket Data Provider is like the Post Data Provider, except the format of the incoming data can be customized as desired.

•The ASFS Data Provider consolidates the API, Socket, File, and Script data providers into a single process. This improves the Universal Agent efficiency when using multiple data providers.

9



Metafiles

After a Data Provider is selected, additional customization is necessary, to determine the specific data source to be used, what (if any) data filtering is needed, and mapping the data from the source to a structure the Universal Agent can recognize and display in the TEP. The Data Provider uses a Metafile to implement this function.

10



Metafiles

� Identifies the specific source of the data

� Maps data coming from the source into a format recognizable by the data provider

– Used to define the data table layout in the Portal Views

� Must be validated before loaded into UA agent

� Loaded at UA startup time via a configuration file, or from command line

A metafile performs the following functions:

•It identifies the specific source of the data. For example, the database to be queried, or the systems from which TCP/IP information will be received.

•It maps data received from the source to a format recognized by the data provider. In the TEP and TEMS environment, agent data is maintained as a set of tables. The Metafile describes how the incoming data is mapped to a table representation. This mapping also defines the layout of the data in the default TEP workspace view for the Universal Agent.

•Since the information in the Metafile must be properly formatted, a validation step is necessary before the Universal Agent can successfully use the Metafile. Once it is validated, it is “loaded” into the Universal Agent, which will then work with the identified source (either receiving data or requesting data) to map it into the tables and workspaces.

•Metafiles can be automatically loaded and activated by the Universal Agent via a configuration file read at agent startup time, or from a command line command on the platform the Universal Agent is running.

11



Metafile Layout

Starts definition of specific data attributesAll//ATTRIBUTES

Summarizes incoming data over an intervalAll//SUMMARY

Defines SQL statement or stored procedure to

retrieve data

ODBC//SQL

Returns confirmation message to data sourceSOCK//CONFIRM

Extracts data from multiple-row recordsFILE//RECORDSET

Defines location and characteristics of collected

data

FILE,ODBC,

SOCK,SCRIPT

//SOURCE

Redirects incoming data to one or more other

attribute groups

all//INTERNAL

Identifies name of attribute groupall//NAME

Provides monitored application name – must be

unique

all//APPL

Identifies SNMP MetafileSNMP//SNMP

PurposeUsed byStatement

A Metafile consists of a series of statements containing various parameters, and in some cases following by one or more sub-statements. Not every statement applies to every Data Provider. This table identifies the order in which the statements must appear, and the Data Provider that the statement is applicable to. The statements are fully documented in the ITM 6.x Universal Agent Users Guide:

•The //SNMP statement identifies a SNMP Data Provider metafile to the agent.

•The //APPL statement gives the set of data mapped by this metafile a name. This name will be used in the TEP navigation tree, under the branch for this agent.

•The //NAME statement gives a name to this specific data group. A Metafile can contain multiple name statements, each one identifying a different group of data. Each name will be used in the TEP Navigation tree, under the branch for the //APPL name.

•The INTERNAL statement is optional, and can be used to manipulate data to other data groups defined in this Metafile. This is used when the incoming data is to be represented in different ways in the portal.

•The //SOURCE statement is used by the Socket, File, ODBC, and Script Data Providers to identify the source of the data:

•The Socket Data Provider uses it to determine which systems data will be received from.

•The File Data Provider uses it to identify the file to be read.

•The ODBC Data Provider uses it to identify the ODBC database to be queried.

•The Script Data Provider uses it to identify the program to be run.

•The //RECORDSET statement is used by the File Data Provider to handle files where the a single logical record spans multiple physical lines in the file.

•The //CONFIRM statement is optionally used by the Socket Data Provider to return a confirmation message to the source that is sending the data.

•The //SQL statement specifics the SQL query to be run against the ODBC database.

•The //SUMMARY statement is used when the data to be reported in based on the summary of the incoming data, and not the data itself. For example, if the data value received are NORTH, SOUTH, EAST, or WEST, and the count for each value received during the interval is what is desired, the SUMMARY statement is used to provide that information.

•The //ATTRIBUTES statement starts the definition of the data layout for this data group. It will be followed by records defining the data attributes name, type, and other characteristics.

12



Metafile Processing

TextEditor

KUMPCON VALIDATE

//APPL…

//NAME…

//SOURCE...

//ATTRIBUTES…

UNIVERSALAGENT

Specifies data characteristics based on application knowledge and monitoring requirements.

Validates definitions and optional loads or refreshes metafile in UA

Definitions can also be loaded via configuration file at agent start, or via command

The process to create and load a Metafile is very straightforward:

1. Any text editor can be used to create the Metafile contents.

2. The KUMPCON VALIDATE command (the syntax differs between Windows and Linux, check the documentation for details) is used to validate the Metafile. It will report any errors in the statements in terms of parameter usage and syntax.

3. If the KUMPCON VALIDATE step is successful, it can optionally load the Metafile into the agent and activate it immediately.

4. Optionally, the Metafile can be loaded via the KUMPCNFG configuration file, which is read by the agent when it starts, or via the KUMPCON IMPORT or KUMPCON REFRESH commands at another time.

13



Existing Integration Packages� Open Process Automation Library

– http://catalog.lotus.com/wps/portal/topal

Before developing a Metafile to monitor a resource, it is wise to check the IBM Tivoli Open Process Automation Library (OPAL) website, at http://catalog.lotus.com/wps/portal/topal . This library contains contributions, from IBM, Business Partners, and Customers, on customized automation solutions built on top of IBM products. The Universal Agent is a popular platform for this work, and it is very possible that a package may already exist for it to monitor a particular resource.

The chart shows a point in time snapshot of the results of searching the OPAL website for “Universal Agent”. 154 results were returned, so there are a lot of pre-built packages that can be downloaded and used without having to develop a Metafile (and any other associated artifacts, such as a monitoring script/program) from scratch.

14



z/OS Integration Considerations

� Universal Agent does not run in z/OS

� Methods of surfacing z/OS data sources to other platforms are necessary

– Network protocols

– File sharing

– Database access (e.g. ODBC, JDBC)

The preceding charts gave an overall introduction to the Universal Agent and its functions. Now, the Universal Agent in the context of z/OS Integration will be considered.

The Universal Agent does not run on z/OS, so another operating system platform must be used to support its functions. If Linux on System z is available, the agent can be installed on that platform, to keep it on System z hardware. If not, another distributed platform must be used.

Since the Universal Agent does not run of z/OS, detailed planning is necessary to identify how z/OS data sources can be surfaced to the agent. Connectivity between z/OS and the Universal Agent platform is required. Providing data source access will depend upon using network protocols, remote file sharing, or remote methods of database access.

15



z/OS Integration Options - Examples

�DB2 Connect or DB2 Enterprise catalogs

z/OS DB2 location as an ODBC source

�Java program uses JDBC to capture information and send via TCP/IP

�ODBC

�SOCKET (via JDBC)

DB2 Database

�Send information to USS file; NFS or SMB

share to UA platform

�Automation product captures messages

and sends via TCP/IP directly or as a trep

�FILE

�SOCKET

�SNMP

Console

Command

Response

�Send information to USS file; NFS or SMB


�Automation product captures messages

and sends via TCP/IP directly or as a trap

�FILE

�SOCKET

�SNMP

SYSLOG

Messages

Send information to USS file; NFS or SMB


�FILELog file

Method UsedTarget UA Data

Provider

z/OS Data

Source

This table illustrates examples of z/OS data sources, and the possible options for accessing the data source via the Universal Agent. The optimal method for a given situation will depend upon a variety of factors, including responsiveness between z/OS and the agent platform, security requirements, expected data volume, etc.

If the information source is a file, The File Data Provider can be used. However, the data must be available in a Unix Systems Services (USS) file (which may require a process to copy it from a sequential or partitioned dataset into USS). Once in USS the file can be access by the agent via file sharing – either Network File System (NFS) or Server Message Block (SMB) sharing from the agent platform to z/OS.

If the information source is the z/OS SYSLOG, the File, Socket, or SNMP Data Providers can be used. the File Data Provider requires the same steps as described earlier, moving the SYSLOG information to a USS file (which can be done by many z/OS automation products). The Socket Data Provider requires an automation function on z/OS to capture the SYSLOG message and create a TCP/IP connection to the agent to send the data. If the automation product has trap generation capabilities it can send the information as a SNMP trap.

If the information is a console command response (usually a command issued periodically to generate information for use in the TEP), the process is the same as described for the z/OS SYSLOG.

If the information is in a DB2 z/OS database, either the ODBC or Socket Data Providers are options. The ODBC option requires a Universal Agent running on Windows, and the use of DB2 connect or DB2 Enterprise Edition, both of which allow a DB2 z/OS database to be cataloged and used as an ODBC data source. The Socket Option can be used if the Agent is not running on a Windows platform, or if ODBC is not possible. A Java program can perform a Java Database Connectivity (JDBC) SQL query to capture the desired data, and then connect to the agent via TCP/IP to send the data.

16



MONITORING SYSLOG INFORMATIONIF MSGID = 'MWB777I' & TEXT=MESSAGE THEN

EXEC(CMD('TESTSKC2 ' MESSAGE) ROUTE(ONE AUTO1))

CONTINUE(Y) NETLOG(Y);

The following sequence of charts gives an example of monitoring SYSLOG information. In this example the MWB777I message is captured and displayed in the portal. The first step is to use automation (in this case IBM Tivoli NetView on z/OS) to capture the message by defining an entry in the NetView automation table.

17






SOCKET TYPE=INIT TCPNAME=TCPIP"

"PIPE NETV SOCKET TYPE=SOCKET SOCKTYPE=STREAM",

"| CORRWAIT 30",

"| VAR STREAMMSG "

/* CONNECT TO THE UNIVERSAL AGENT */

"SOCKET TYPE=CONNECT SOCKID="srvsock “,

ADDRESS="||hostip "PORT="||port

ADDRESS NETVASIS " PIPE LIT /"string"/",

"| NETV SOCKET TYPE=SEND SOCKID="srvsock,

"| CORRWAIT 60 | CONSOLE"

"PIPE NETV SOCKET TYPE=RECV SOCKID="||srvsock,

"| CORRWAIT 60",

"| LOCATE /BNH621I/",

"| STEM INRECS."

"PIPE NETV SOCKET TYPE=SHUTDOWN SOCKID="srvsock,

"| CORRWAIT 10"

"PIPE NETV SOCKET TYPE=CLOSE SOCKID="srvsock,

"| CORRWAIT 10"

When the automation table detects the MWB777I message, it invokes a NetView REXX automation procedure, which extracts the desired message contents and uses the NetView SOCKET functions to send the data to the Universal Agent Socket Provider.

18








"| CORRWAIT 30",

"| VAR STREAMMSG "








"| CORRWAIT 60",


"| STEM INRECS."


"| CORRWAIT 10"


"| CORRWAIT 10"

//APPL ZOSMSG

//NAME MSG01 E 7200

//SOURCE SOCK 10.1.1.124 E




//CONFIRM 'The message has been received'

//ATTRIBUTES ';'

JOBNAME D 8

MSGID D 10

MSGTEXT D 256

The Universal Agent matches the incoming data to a Metafile based upon several options (the system the data is coming from based on the Metafile SOURCE statement, the APPL name that is included in the incoming data, etc.) to determine how the data should be formatted. The selected Metafile indicates the following:

•From the //APPL statement, the application name in the TEP will contain zOSMSG.

•From the //NAME statement, the data sets in the Metafile will be named MSG01, the ‘E’ indicates this is Event data (it comes it based on a state change, and not a regular interval), and the data should be kept for 2 hours (7200 seconds) before it is discarded.

•From the //SOURCE statements, the systems from which data can be received. Any data coming from a system not on this list is ignored and not show in the TEP.

•From the //CONFIRM statement, the message the agent will return to the sender after it receives the data.

•From the //ATTRIBUTES statement, the incoming data fields are delimited by a semi-colon. The lines following the //ATTRIBUTES statement identify each attribute, its type and its maximum size. For example, the first attribute is JOBNAME, the D indicates a Display (character) data type, and 8 is the maximum number of characters in the data.

19








"| CORRWAIT 30",

"| VAR STREAMMSG "








"| CORRWAIT 60",


"| STEM INRECS."


"| CORRWAIT 10"


"| CORRWAIT 10"

//APPL ZOSMSG

//NAME MSG01 E 7200





//CONFIRM 'The message has been received'

//ATTRIBUTES ';'

JOBNAME D 8

MSGID D 10

MSGTEXT D 256

The final result shows the message information in the TEP. Note that under the Universal Agent branch (Universal Data Provider), the //APPLICATION statement name ZOSMSG is used in the navigation tree item; the item name includes the name of the system from which the data is received, and the version of the Metafile (versions can change if the data layout is changed and the Metafile is reloaded). The //NAME statement MSG01 value names the workspace, and provides a default table view of the received data. The columns match the attributes defined in the Metafile. At this point the data can be used in the TEP just as any other data received from product agents.

20



Command Response Monitoring - JES2 Monitor

//APPL JES2

//NAME JDTLS S 86400

//SOURCE SOCK TESTMVS E

//CONFIRM 'JDDETAILS DATA

RECEIVED'

//ATTRIBUTES ';'

SAMPLETIME D 20

RESOURCE D 20

LIMIT C 10000000

USAGE C 10000000

LOW C 10000000

HIGH C 10000000

AVG C 10000000

This example is similar to the preceding one, except it uses information from the output of a command (in this case the JES2 $JDDETAILS command, which shows the usage of JES2 resources). A NetView REXX automation procedure periodically issues this command, captures and formats the output, and sends to the Universal Agent. The Metafile //NAME statement ‘S’ parameter identifies this as sampled data (data received on a regular interval), and the data will be kept for 24 hours (86,400 seconds). The ATTRIBUTES fields now include ‘C’ for Counter (numeric) fields, and the value is the maximum expected value of the field.

21



JES2 Information Monitoring

The results in the portal show some customization that has been performed in the workspace. The upper workspace view has been changed to a bar graph, to pictorially show the relative usage of the JES2 resources.

22



Monitoring IBM HTTP Server on z/OS

� Several Data Sources

– HTTP Server URLs (direct monitoring by Agent)

– HTTP Server Logs

• Access log most likely to use

– z/OS Console Command Responses

• F <WebServer>,APPL=-D STATS

– /Usage/Initial URL (if Service/Usage* directive is active)

– SMF Data

• Type 103 records

This example explores monitoring of the IBM HTTP Server on z/OS. There are several different ways monitoring can be applied:

-The HTTP Data Provider can directly monitor key IBM HTTP Server URLs, to provide response time and data volume information.

-The IBM HTTP Server logs can be monitored, to look at request details or summaries.

-The response to the z/OS F <WebSphere>,APPL=-D STATS commands, issued from a console to the HTTP Server address space, can be captured, as it provides some HTTP Server statistics.

-The /Usage/Initial URL, if activated via the ServiceUsage HTTP Server directive in the httpd.conf configuration file, can be retrieved to get HTTP Server statistics.

-SMF type 103 records can be read to get HTTP Server Statistics.

There are many options, and various factors will determine which one is most suitable for a particular situation.

23



HTTP Server URL monitoring

� Define the URL to the Agent

– KUMPURLS configuration file

– TEPS “Take Action”Command

– Situation

To monitor a URL, no Metafile is needed. The target URL is entered in the KUMPURLS configuration file, or identified via a TEP “Take Action” command to start monitoring. Monitoring URLs create MANAGED_URL and URL_OBJECTS workspaces.

24



URL Workspace

Default Table Customized View

In the MANAGED_URL workspace, the default table provides the URL name, status and timestamp of last status, page title, and response times information. This workspace has been customized to show a bar chart that summarizes the current, average, and maximum response times for the monitored URLs.

25



HTTP Server SOCKET monitoring

REXX ProgramIssues “APPL=-D

STATS“

command

Parses returned data into fields

//APPL HTTPappl

//Name HTTPactivity E 3600 AddTimeStamp=YearMonth

//SOURCE SOCK 10.1.1.2[4500] E

//CONFIRM 'Data received'

//Attributes ';'

HTTPAS D 8

ThreadsRunning C 1000000

ThreadsIdle C 1000000

Requests C 1000000

BytesReceived C 10000000

BytesSent C 10000000

ActiveInboundConnections C 1000000

ActiveOutboundConnections C 1000000

ConnectionsSinceLastSMF C 1000000

NonSSLWaitingThreads C 1000000

SSLWaitingThreads C 1000000

AsyncIOWaitingThreads C 1000000

MsgQueueWaitingThreads C 1000000

WEBSRV;40;20;1

2493;7308505;4

3039143;20;0;2

32;0;20;0;0;

The Socket Data Provider is used by a REXX program that captures the output from the “APPL=-D STATS command, formats the desired statistics into a semi-colon delimited record, and sends the data to the Universal Agent. the Metafile maps the incoming data to specific attributes.

26



SOCKET Monitoring – Workspace Results

Each record in the workspace table shows the statistics for each interval. The workspace has been customized to show a plot chart of the HTTP outbound traffic volume trend over time.

27



HTTP Server FILE monitoring� Access Log

resides in USS file system

� z/OS SMB sharing used to share log file directory with other platforms

– EBCDIC/ASCII translation performed as part of the share

192.168.57.8 - - [03/Nov/2006:00:00:50 +0000]

"GET /cgi-bin/ivlookup

HTTP/1.1" 200 4522

//APPL IHSLOG

//Name LOGACTIVITY E

//SOURCE FILE 'o:\httpd-log.{MMMDDYYYY} TAIL'

//SUMMARY 300

//Attributes

-ClientAddress D 256

Clientlocation (ipAddressToName = ClientAddress)

ClientID D 32

UserName D 32

Date D 11 DLM='[:'

Time D 8

GMTOffest D 5 DLM=' ]'

Request D 256 SKEY=1 DLM='""'

Status C 999 SKEY=2

BytesReceived C 999999999 SKEY=SUM

Access Log entry

The File Data Provider is used to monitor and retrieve data from the IBM HTTP Server access log. The access log is normally located in a USS directory, so to use the File Data Provider the Universal Agent platform must have access. This can be done by enabling the USS file sharing functions (either NFS or SMB) to allow the agent platform to access the file via a directory share; to the agent the access log will appear as if it resides on a local file directory. The sharing also provides automatic EBCDIC to ASCII data translation of the file contents.

For this example the //SOURCE statement identifies the file to be monitored. The file name can be wild carded, for applications that produce timestamped logs whose name changes based on the current date and/or time. The //SUMMARY statement indicates that data summarization will be performed; the fields to use as summary keys are identified in the //ATTRIBUTES section by the “SKEY” parameter. In this Metafile the access log information will be summarized every five minutes, by request and status. TheBytesReceived field will be a sum of that field for the interval.

The //ATTRIBUTES statement for this Metafile shows a few additional capabilities:

-The “ipAddresstoName = ClientAddress” is a provided function for translating an IP address to a hostname.

-Custom delimiters can be used when individual data fields have different delimiters, by using the “DLIM” parameter with the attribute definition. In this Metafile the DATE field in the log ends when a bracket and colon is encountered. The GMTOffset field ends when a blank and bracket is encountered.

28



FILE Monitoring – Workspace Results

The results are shown in the bottom table. Each record represents a request and status, and the BytesReceived shows the total bytes received for the request over the interval. The graph was created to show the most requested URLs found in the log, and will change as additional records are logged.

29



DB2 z/OS Table Information

� If Universal Agent is on a Windows platform

– Install DB2 Connect or DB2 Enterprise Server on same system

– Catalog z/OS Location and define it as an ODBC resource

� If Universal Agent is on a Windows or Linux/Unix platform

– Use Java program as interface between database (JDBC) and the Universal Agent Data Provider (SOCKET, FILE)

To retrieve information from a DB2 Table, DB2 Connect or DB2 Enterprise Server can be used if the agent is on a Windows platform. These products allow z/OS DB2 Distributed Data Facility (DDF) locations to be cataloged in the distributed DB2 environment, and defined as ODBC data sources. The OBDC Data Provider can be used to access information in the database.

Another option, which much be used if the Universal Agent is on a Linux our Unix platform, is to use a program that access the data and either sends it to the agent via the Socket Data Provider, or record it into a file monitored by the File Data provider. A Java program can provide this functionality, using JDBC to access the database and then its own TCP/IP or file functions to send or record the data.

30



DB2 z/OS ODBC Example

catalog tcpip node db2zosn1 remote 10.1.1.2 server 448

catalog db db2zosd1 at node db2zosn1 authentication dcs

catalog dcs db db2zosd1 as tstdb202

For this example, a table defined in DB2 on z/OS contains product sales information that is to be incorporated into the TEP to provide high level business metrics. The process to defining DB/2 z/OS to the DB2 Connect or DB2 Enterprise Edition catalog is documented in the DB2 product manuals; at a high level, once the location name and IP port of the DB2 z/OS DDF Server address space is known (from the DB2 –DISPLAY DDF command), this information is used on DB2 Connect or DB2 Enterprise Server to catalog the DB2 z/OS location.

31



DB2 z/OS ODBC Example…

//APPL MYCOMPANY//NAME PRODSALES K 3700 INTERVAL 3600//SOURCE ODBC TSTDB202 User=BONETT Pswd=BONETT//SQL SELECT LOCATION, SUM(PRODA) AS

PRODA_TOT,SUM(PRODB) AS

PRODB_TOT,SUM(PRODC) AS PRODC_TOT,

SUM(PRODD) AS PRODD_TOT FROM

BONETT.PRODSALES GROUP BY

LOCATION

//ATTRIBUTESLOCATION D 10 KEYPRODA_TOT C 999999999PRODB_TOT C 999999999PRODC_TOT C 999999999PRODD_TOT C 999999999

Once the DB2 z/OS location is cataloged, it can then be defined as an ODBC data source from Windows.

In this Metafile, //the SOURCE statement identifies the ODBC resource that will be queried, along with a valid username and password for access. The //SQL statement contains the SQL statement that will be issued to retrieve the data. The INTERVAL parameter on the //NAME statement determines how frequently the agent will issue the SQL query. In the //ATTRIBUTES section, the KEY parameter indicates that LOCATION will be the key field for the data; along with the ‘K’parameter in the //NAME statement, this means that a single set of data (one row of data for each value found value in this field) will be displayed in the workspace table, instead of a set of data for each interval.

32



DB2 z/OS ODBC Example…

The results are show in the bottom table. The current number of product sales by product, for each location, is shown. The same information is shown in the pie chart.

33



Situation Triggering Example

� When a event occurs for a resource being monitored by IBM Tivoli Enterprise Console, the event is forwarded to the TEP to trigger a WTO message

� ITM 6.x provides the TEC Console GUI, but this does not permit access to event data for situations and policies

� Older versions of the CandleNet Portal contained a TEC Alert Adapter, but this is no longer available in ITM 6.x

� Solution: Send TEC event to the Universal Agent, and run situation against the UA workspace

Data received from the Universal Agent can be used to trigger automated actions from situations or policies. For example, an environment had to send events from the IBM Tivoli Enterprise Console (TEC) to a z/OS Hub TEMS, and based on the contents of the event issue a z/OS command. While ITM 6.1 and later provides a TEC console GUI, the data in the TEC Console GUI cannot be used in situations and policies. Earlier versions of the CandleNet Portal contained a TEC Alert Adapter for receiving TEC events, but this adapter is no longer available. The solution selected is to send the TEC event to the Universal Agent, and to define a situation against that data.

34



TEC Event in UA Workspace

There are many options for receiving TEC Event Data into the Universal Agent. For this example a Java TEC event receiver, using the TEC Event Integration Facility, was created to receive the event and then format and send it to the agent via the Socket Data Provider. The Metafile maps the TEC Event slot names to attribute fields, so that each field and value corresponds to a name-value pair within the event.

35



Situation Definition

A situation is created that will be true if a Event with CRITICAL or WARNING severity is received from a particular system, and the event is not closed.

36



Situation Definition

When the situation is true, the z/OS SEND command will be used to create a message containing information from the received event.

37



Situation Results

This results after the situation is started are displayed. Events matching the criteria have been received, and the situation turns true and can be seen in the TEP. On the z/OS console, the message is issued containing event information – the source system, severity, etc.

38



Packaged z/OS Related Solutions on OPAL - Example

This IBM zOS HTTP Server Agent collects HTTP Server statistics to enhance your ability to monitor WebSphere and IBM HTTP Server environments on zOS.

IBM zOS HTTP Server Monitoring Agent

This solution provides the capability of integrating zSeries monitoring data with ITM 6.x using a Rexx-based program and the Universal Agent socket data provider.

Rexx on zSeries Socket Monitoring Solution using the ITM 6.x Universal Agent

The examples covered in this presentation cover a few of the ways to get started with using the Universal Agent to monitor z/OS components. There are packages available on the OPAL Website specific to z/OS monitoring which may be useful. For example, one package provides a REXX based program that can be used to integrate data on z/OS with the Universal Agent via the Socket data provider. Another package provides in more detailed solution for using the Universal Agent to monitor the IBM HTTP Server. Other z/OS based packages may become available over time, so it is worth checking the OPAL website on a regular basis for new information.

39



For Further Information� Product Documentation

– IBM Tivoli Monitoring Information Center (http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/index.jsp?toc=/com.ibm.itm.doc/toc.xml)

• Users Guide (SC32-9459)

• API and Command Programming Reference Guide (SC32-9461)

• Metafile Examples

� Open Process Automation Library (OPAL)

– IBM Tivoli Monitoring 6.1: Uses of the Universal Agent (White Paper)

� IBM TechDocs (http://www.ibm.com/support/techdocs)

– Monitoring the IBM HTTP Server on z/OS from the Tivoli Enterprise Portal

– Sending Tivoli Enterprise Console Events to the Tivoli Enterprise Portal via the IBM Tivoli Universal Agent

This presentation introduced the Universal Agent, its context when working with z/OS and examples of how it can be used to monitor z/OS resources. It is not meant to be exhaustive, but meant to provide a starting point. The ITM 6.x product documentation provides much more detail on the Universal Agent installation, customization, and usage. OPAL provides packages for use with the Universal Agent. Articles on IBM TechDocs provide more detailed examples of using the Universal Agent with z/OS resources.

Even though the Universal Agent does not execute on z/OS, it provides very useful functions and capabilities, and is an option to be considered when evaluating ways to further integrate z/OS monitoring into a Tivoli Enterprise Portal Environment.

Please feel free to send any questions about this presentation to the author, Mike Bonett, IBM Advanced Technical Support, [email protected].

Documents

Using the IBM Tivoli Universal Agent to Enhance z/OS ... · PDF fileUsing the IBM Tivoli Universal Agent to Enhance ... Using the IBM Tivoli Universal Agent to Enhance z/OS Monitoring