© 2013 IBM Corporation TBSM v6.1 Troubleshooting Events September, 2013 Joy Hawkins

© 2013 IBM Corporation

TBSM v6.1 Troubleshooting Events

September, 2013Joy Hawkins

© 2013 IBM Corporation2

Agenda

2

Technote

UserID’s (Roles & Groups)

TBSM Event Environment

Event Results Walk-Through from GUI

Enable Event Logging

Results from event logging in trace.log

How to use event logging filters


Technote: (Events)

MustGather - for IBM support to help analyze and understand a TBSM event issue?

http://www.ibm.com/support/docview.wss?uid=swg21598760


Section Separator Section Separator

Technote


Technote



UserID’s - Roles & User Groups

• (admin1) - (or use ‘tbsmadm’ from install)

• Access Service Availability and Service Administration Pages

• (tbsmAdminUserRole) assigned by default to tbsmAdmins Group

• (user1) – (or use ‘tbsmuser” from install)

• Access Service Availability Page• (tbsmReadOnlyUserRole) assigned

by default to tbsmReadOnly Group



(admin1) UserID’s access


(user1) access



Netcool_Omnibus_Admin Group

• [ncw_user, ncw_gauges_editor,ncw_admin, ncw_dashboard_editor, netcool_rw]



to run tool

Netcool_Omnibus_User Group

• [ncw_user, netcool_ro]



• ISQWrite – omnibus privileges to run tool











TBSM Environment: (Templates)


TBSM Environment: (Incoming Status Rule)


TBSM Environment: (Preview data)


TBSM Environment: (Identification fields tab)


TBSM Environment: (ObjectServer is not Data Source)


TBSM Environment: (Data Fetcher)


TBSM Environment: (Data Fetcher’s View Data)

SelectDATAOPNED, LOWLEVELSERVICEID, REGION, SEVERITY, STATUS, SUMMARYTEXT, TICKETID, TICKETTYPE from TBSMDEMO.TICKETSwhere STATUS = ‘Open’ and SEVERITY >4 and REGION =‘US’

SelectDATAOPNED, LOWLEVELSERVICEID, REGION, SEVERITY, STATUS, SUMMARYTEXT, TICKETID, TICKETTYPE from TBSMDEMO.TICKETSwhere STATUS = ‘Open’ and SEVERITY >4 and REGION =‘US’


TBSM Environment: (Rule’ Preview Data)








TBSM Environment: (Service Instances & send test Event)


Event Results: (Service Administration Portlets)


Event Results: (AEL)


Event Details

Event Results: (Event Details)








Enabling Event Logging

• Available from TBSM 4.2.1 FP3, or TBSM 6.1 FP1, or TBSM 6.1.1 GA.

• Use setTraceLevel command to add the trace level for the EventLogging package.

UNIX:$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data –addtrace "EventLogging=finer"

Windows:%TBSM_HOME%\bin\setTraceLevel.bat <tipadmin user> <tipadmin password> data -addtrace “EventLogging=finer“


• For best effect, set any other TBSM trace levels to “info”

$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -addtrace "com.micromuse.sla.*=info:com.ibm.tbsm.*=info:EventLogging=finer”



• Other setTraceLevel.sh options

$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -display

$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -setstartup

• Event Logging should not be enabled permanently on customer sites – turn it off when not required.

$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -resetall

OR

$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -rmtrace "EventLogging=finer"









Collect new trace logs

• Run <installdir>/tbsm/bin/collect_logs

After running the utility on the TBSM Data server, the result will be a single zip file <installdir>/tbsm_logs_<datetimestamp>.zip

• $TIP_HOME/TBSMProfile/logs/server1/ (trace.log, SystemOut.log, SystemErr.log)


Trace log Entries

• With tracing set to EventLogging.

• The first entry is an event receipt – shows all fields in the Event Container. TBSM will use the filters to match the event to a service.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 RADEventProcessor execute "Prepared by TBSMOMNIbusEventReader"=(EventContainer=(RAD_TimeWindowStart=0, AlertKey=mail, Location=, RAD_TimeWindowEnd=0, RemotePriObj=, LocalRootObj=, RAD_FunctionName=, NmosEntityId=0, RemoteNodeAlias=, X733EventType=0, ReceivedWhileImpactDown=0, RAD_UserFunctionName=NCOMS564, RAD_TotalRawMetric=, TaskList=0, ProbeSubSecondId=0, RAD_ServiceID=0, RAD_TimeWindowLength=0, RemoteRootObj=, Tally=1, KeyField=564, NodeAlias=, RAD_NextThresholdRawMetric=, RAD_ServiceTypeName=, NmosDomainName=, RAD_LastCount=1, RAD_CurrentFunctionState=, RAD_FilterIDList=2,3, Serial=564, X733CorrNotif=, ……<also seen in same message, I can see…>

Identifier=steJoyTESTEvent110,Node=Mail3PortC, Class=0, Summary=JoySendEvent3,Severity=3,…),

<the message ends with the following…>Policies={Main}, PolicyName=Main, ReportsPolicyName=Main, ActionNodeName={Main}, ActionType=1)


Trace log Entries

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter getRawAttributeWrappersByEvaluatingFilters No Filters found in Event Container so must get filters.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter getRawAttributeWrappersByEvaluatingFilters Using discriminator: 0.

• Three parts of an Incoming Status Rule:

(Discriminator, Filter Thresholds and Instance fields).

1. Discriminator (or Class) is used to set the RAD_FilterIDList on the event.


[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch Testing AlertKey with comparator = 'mail'.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true.

[7/22/13 13:42:10:947 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch Testing Severity with comparator >= '3'.

[7/22/13 13:42:10:947 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter addFilterIDToEvent New RAD_FilterIDList for Event is to set to: 2.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter println testThresholdMatch Threshold matched and status is greater than 0.

Trace log Entries

2. Threshold Filter. Each field in the Event Container is tested against the Filter fields, to check if they match the threshold.


[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 AbstractMatchingInstanceGetter getInstanceIDFieldValueMappingForRawAttribute Returning field expression and value for attribute: {Node=Mail3PortC}.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 MemoryBasedMatchingInstanceGetter getMappingStringForRawAttribute Returning event string for raw attribute: Mail3PortC.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 GlobalInstanceFieldStore getCommaSeparatedInstanceIDs TagId is 85. attributeID is 1. eventMatchingString is Mail3PortC. List of Service Instance Ids which match raw attribute is: 89.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 MemoryBasedMatchingInstanceGetter getMatchingInstanceIDsFromDataStore Returning matching instances: [89].

Trace log Entries

3. Instance field. TBSM reads the value for the Event fields from the event, and then returns a list of Service Instances which match this.

11

4433

22


Trace log Entries

• Update State in TBSM

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 RADEventProcessor sendInstancesToStateModel1 Send to statemodel for ServiceInstanceName Mail3PortC Service instanceID: 89.[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelUpdater passToStateModel Status is 3.0.


Trace log Entries

• Update Object Server service_deps table

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelNodeAttributeState updateServiceDeps About to update service_deps in memory and at object server for SIB: Mail3PortC.[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 ObjectServerUpdater updateServiceDeps updateEventIDs for Mail3PortC and eventInstanceID NCOMS:_:564.


Trace log Entries

• Update Service Name in Event

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelUpdater setVarsInContexts RAD_ServiceName is: Mail3PortC being set in Event Container.


Trace log Entries

• Propagate up parents

[7/22/13 11:41:22:151 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState stateChanged About to update parents for current SIB.[7/22/13 11:41:22:151 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: Mail3PortC.

[7/22/13 11:41:22:153 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: MailServer3.

[7/22/13 11:41:22:156 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: App-Root.








Event Logging with Filters

• Filters reduce the number of events which are eligible for Event Logging

• Filters are dynamic

• Filters are persistent

• Steps:

• First eventloggingfilter Table must created

• Then use radshell commands to set up filters for Event Logging



• For TBSM 4.2.1 - create the postgres table eventloggingfilter

• Created by running the SQL file: createEventLoggingFilterTable.sql (This is shipped with TBSM 4.2.1 FP3)

UNIX:

cat ./createEventLoggingFilterTable.sql | $TBSM_HOME/bin/rad_db log

Windows:

type createEventLoggingFilterTable.sql | %TBSM_HOME%\bin\rad_db log

Windows:

type createEventLoggingFilterTable.sql | %TBSM_HOME%\bin\rad_db log



• For TBSM 6.1.X - create the DB2 table eventloggingfilter

• From a command line start db2

db2

connect to TBSM (to connect to DB)



• For TBSM 6.1.X - create the DB2 table eventloggingfilter

• Create Table

create table TBSMBASE.eventloggingfilter(

filterfieldname varchar(255),filterfieldvalue varchar(1024)

)


•Use radshell commands to set up filters for Event Logging

•Displays the current filter setting

•Clear the filter

•Adds fields and values to the current filter

•Removes a field from the filter


$TBSM_HOME/bin/rad_radshell

displayEventLoggingFilter();

clearEventLoggingFilter();

addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);

removeEventLoggingFilterValue(fieldName);


Filter examples: add a field to current filter:


•Only events with Node = “Mail3PortC” are logged, when the trace level for EventLogging package is finer

•addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”);




Filter examples: add a field to current filter:


•To change the filter, so that only events with Node = “Mail3PortC”,AND AlertKey = “mail”, are logged, when the trace level for EventLogging package is finer

•addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”);

•addUpdateEventLoggingFilterValue(“AlertKey”,”mail”);



Filter examples:



Filter examples: remove a field from current filter:

removeEventLoggingFilterValue(fieldName);

•To remove the Node fields from the filter (now filter will just match on AlertKey)

•removeEventLoggingFilterValue(“Node”);



Filter examples: remove a field from current filter:

clearEventLoggingFilter();

•Remove ALL filter fields – Logging will be done for all events if the trace level for EventLogging package is finer.

• clearEventLoggingFilter();


The filter set up by the addUpdateEventLoggingFilterValue radshell command, is only used when the trace level for the EventLogging package is set to finer.

If EventLogging package is not set, the radshell filter has NO effect.

Event Logging with Filters- Turn off afterwards


Documents

© 2013 IBM Corporation TBSM v6.1 Troubleshooting Events September, 2013 Joy Hawkins