Upload
myles-wiggins
View
225
Download
7
Embed Size (px)
Citation preview
© 2013 IBM Corporation
TBSM v6.1 Troubleshooting Events
September, 2013Joy Hawkins
© 2013 IBM Corporation2
Agenda
2
Technote
UserID’s (Roles & Groups)
TBSM Event Environment
Event Results Walk-Through from GUI
Enable Event Logging
Results from event logging in trace.log
How to use event logging filters
© 2013 IBM Corporation3
Technote: (Events)
MustGather - for IBM support to help analyze and understand a TBSM event issue?
http://www.ibm.com/support/docview.wss?uid=swg21598760
© 2013 IBM Corporation4
Section Separator Section Separator
Technote
UserID’s (Roles & Groups)
Technote
UserID’s (Roles & Groups)
© 2013 IBM Corporation5
UserID’s - Roles & User Groups
• (admin1) - (or use ‘tbsmadm’ from install)
• Access Service Availability and Service Administration Pages
• (tbsmAdminUserRole) assigned by default to tbsmAdmins Group
• (user1) – (or use ‘tbsmuser” from install)
• Access Service Availability Page• (tbsmReadOnlyUserRole) assigned
by default to tbsmReadOnly Group
© 2013 IBM Corporation6
UserID’s - Roles & User Groups
(admin1) UserID’s access
© 2013 IBM Corporation7
(user1) access
UserID’s - Roles & User Groups
© 2013 IBM Corporation8
Netcool_Omnibus_Admin Group
• [ncw_user, ncw_gauges_editor,ncw_admin, ncw_dashboard_editor, netcool_rw]
UserID’s - Roles & User Groups
© 2013 IBM Corporation9
to run tool
Netcool_Omnibus_User Group
• [ncw_user, netcool_ro]
UserID’s - Roles & User Groups
© 2013 IBM Corporation10
• ISQWrite – omnibus privileges to run tool
UserID’s - Roles & User Groups
© 2013 IBM Corporation11
UserID’s - Roles & User Groups
© 2013 IBM Corporation12
Section Separator Section Separator
UserID’s (Roles & Groups)
TBSM Event Environment
UserID’s (Roles & Groups)
TBSM Event Environment
© 2013 IBM Corporation13
TBSM Environment: (Templates)
© 2013 IBM Corporation14
TBSM Environment: (Incoming Status Rule)
© 2013 IBM Corporation15
TBSM Environment: (Preview data)
© 2013 IBM Corporation16
TBSM Environment: (Identification fields tab)
© 2013 IBM Corporation17
TBSM Environment: (ObjectServer is not Data Source)
© 2013 IBM Corporation18
TBSM Environment: (Data Fetcher)
© 2013 IBM Corporation19
TBSM Environment: (Data Fetcher’s View Data)
SelectDATAOPNED, LOWLEVELSERVICEID, REGION, SEVERITY, STATUS, SUMMARYTEXT, TICKETID, TICKETTYPE from TBSMDEMO.TICKETSwhere STATUS = ‘Open’ and SEVERITY >4 and REGION =‘US’
SelectDATAOPNED, LOWLEVELSERVICEID, REGION, SEVERITY, STATUS, SUMMARYTEXT, TICKETID, TICKETTYPE from TBSMDEMO.TICKETSwhere STATUS = ‘Open’ and SEVERITY >4 and REGION =‘US’
© 2013 IBM Corporation20
TBSM Environment: (Rule’ Preview Data)
© 2013 IBM Corporation21
Section Separator Section Separator
TBSM Event Environment
Event Results Walk-Through from GUI
TBSM Event Environment
Event Results Walk-Through from GUI
© 2013 IBM Corporation22
TBSM Environment: (Service Instances & send test Event)
© 2013 IBM Corporation23
Event Results: (Service Administration Portlets)
© 2013 IBM Corporation24
Event Results: (AEL)
© 2013 IBM Corporation25
Event Details
Event Results: (Event Details)
© 2013 IBM Corporation26
Section Separator Section Separator
Event Results Walk-Through from GUI
Enable Event Logging
Event Results Walk-Through from GUI
Enable Event Logging
© 2013 IBM Corporation27
Enabling Event Logging
• Available from TBSM 4.2.1 FP3, or TBSM 6.1 FP1, or TBSM 6.1.1 GA.
• Use setTraceLevel command to add the trace level for the EventLogging package.
UNIX:$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data –addtrace "EventLogging=finer"
Windows:%TBSM_HOME%\bin\setTraceLevel.bat <tipadmin user> <tipadmin password> data -addtrace “EventLogging=finer“
© 2013 IBM Corporation28
• For best effect, set any other TBSM trace levels to “info”
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -addtrace "com.micromuse.sla.*=info:com.ibm.tbsm.*=info:EventLogging=finer”
Enabling Event Logging
© 2013 IBM Corporation29
• Other setTraceLevel.sh options
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -display
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -setstartup
• Event Logging should not be enabled permanently on customer sites – turn it off when not required.
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -resetall
OR
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -rmtrace "EventLogging=finer"
Enabling Event Logging
© 2013 IBM Corporation30
Section Separator Section Separator
Enable Event Logging
Results from event logging in trace.log
Enable Event Logging
Results from event logging in trace.log
© 2013 IBM Corporation31
Collect new trace logs
• Run <installdir>/tbsm/bin/collect_logs
After running the utility on the TBSM Data server, the result will be a single zip file <installdir>/tbsm_logs_<datetimestamp>.zip
• $TIP_HOME/TBSMProfile/logs/server1/ (trace.log, SystemOut.log, SystemErr.log)
© 2013 IBM Corporation32
Trace log Entries
• With tracing set to EventLogging.
• The first entry is an event receipt – shows all fields in the Event Container. TBSM will use the filters to match the event to a service.
[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 RADEventProcessor execute "Prepared by TBSMOMNIbusEventReader"=(EventContainer=(RAD_TimeWindowStart=0, AlertKey=mail, Location=, RAD_TimeWindowEnd=0, RemotePriObj=, LocalRootObj=, RAD_FunctionName=, NmosEntityId=0, RemoteNodeAlias=, X733EventType=0, ReceivedWhileImpactDown=0, RAD_UserFunctionName=NCOMS564, RAD_TotalRawMetric=, TaskList=0, ProbeSubSecondId=0, RAD_ServiceID=0, RAD_TimeWindowLength=0, RemoteRootObj=, Tally=1, KeyField=564, NodeAlias=, RAD_NextThresholdRawMetric=, RAD_ServiceTypeName=, NmosDomainName=, RAD_LastCount=1, RAD_CurrentFunctionState=, RAD_FilterIDList=2,3, Serial=564, X733CorrNotif=, ……<also seen in same message, I can see…>
Identifier=steJoyTESTEvent110,Node=Mail3PortC, Class=0, Summary=JoySendEvent3,Severity=3,…),
<the message ends with the following…>Policies={Main}, PolicyName=Main, ReportsPolicyName=Main, ActionNodeName={Main}, ActionType=1)
© 2013 IBM Corporation33
Trace log Entries
[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter getRawAttributeWrappersByEvaluatingFilters No Filters found in Event Container so must get filters.
[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter getRawAttributeWrappersByEvaluatingFilters Using discriminator: 0.
• Three parts of an Incoming Status Rule:
(Discriminator, Filter Thresholds and Instance fields).
1. Discriminator (or Class) is used to set the RAD_FilterIDList on the event.
© 2013 IBM Corporation34
[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch Testing AlertKey with comparator = 'mail'.
[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true.
[7/22/13 13:42:10:947 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch Testing Severity with comparator >= '3'.
[7/22/13 13:42:10:947 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true.
[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter addFilterIDToEvent New RAD_FilterIDList for Event is to set to: 2.
[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter println testThresholdMatch Threshold matched and status is greater than 0.
Trace log Entries
2. Threshold Filter. Each field in the Event Container is tested against the Filter fields, to check if they match the threshold.
© 2013 IBM Corporation35
[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 AbstractMatchingInstanceGetter getInstanceIDFieldValueMappingForRawAttribute Returning field expression and value for attribute: {Node=Mail3PortC}.
[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 MemoryBasedMatchingInstanceGetter getMappingStringForRawAttribute Returning event string for raw attribute: Mail3PortC.
[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 GlobalInstanceFieldStore getCommaSeparatedInstanceIDs TagId is 85. attributeID is 1. eventMatchingString is Mail3PortC. List of Service Instance Ids which match raw attribute is: 89.
[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 MemoryBasedMatchingInstanceGetter getMatchingInstanceIDsFromDataStore Returning matching instances: [89].
Trace log Entries
3. Instance field. TBSM reads the value for the Event fields from the event, and then returns a list of Service Instances which match this.
11
4433
22
© 2013 IBM Corporation36
Trace log Entries
• Update State in TBSM
[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 RADEventProcessor sendInstancesToStateModel1 Send to statemodel for ServiceInstanceName Mail3PortC Service instanceID: 89.[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelUpdater passToStateModel Status is 3.0.
© 2013 IBM Corporation37
Trace log Entries
• Update Object Server service_deps table
[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelNodeAttributeState updateServiceDeps About to update service_deps in memory and at object server for SIB: Mail3PortC.[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 ObjectServerUpdater updateServiceDeps updateEventIDs for Mail3PortC and eventInstanceID NCOMS:_:564.
© 2013 IBM Corporation38
Trace log Entries
• Update Service Name in Event
[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelUpdater setVarsInContexts RAD_ServiceName is: Mail3PortC being set in Event Container.
© 2013 IBM Corporation39
Trace log Entries
• Propagate up parents
[7/22/13 11:41:22:151 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState stateChanged About to update parents for current SIB.[7/22/13 11:41:22:151 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: Mail3PortC.
[7/22/13 11:41:22:153 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: MailServer3.
[7/22/13 11:41:22:156 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: App-Root.
© 2013 IBM Corporation40
Section Separator Section Separator
Results from event logging in trace.log
How to use event logging filters
Results from event logging in trace.log
How to use event logging filters
© 2013 IBM Corporation41
Event Logging with Filters
• Filters reduce the number of events which are eligible for Event Logging
• Filters are dynamic
• Filters are persistent
• Steps:
• First eventloggingfilter Table must created
• Then use radshell commands to set up filters for Event Logging
© 2013 IBM Corporation42
Event Logging with Filters
• For TBSM 4.2.1 - create the postgres table eventloggingfilter
• Created by running the SQL file: createEventLoggingFilterTable.sql (This is shipped with TBSM 4.2.1 FP3)
UNIX:
cat ./createEventLoggingFilterTable.sql | $TBSM_HOME/bin/rad_db log
Windows:
type createEventLoggingFilterTable.sql | %TBSM_HOME%\bin\rad_db log
Windows:
type createEventLoggingFilterTable.sql | %TBSM_HOME%\bin\rad_db log
© 2013 IBM Corporation43
Event Logging with Filters
• For TBSM 6.1.X - create the DB2 table eventloggingfilter
• From a command line start db2
db2
connect to TBSM (to connect to DB)
© 2013 IBM Corporation44
Event Logging with Filters
• For TBSM 6.1.X - create the DB2 table eventloggingfilter
• Create Table
create table TBSMBASE.eventloggingfilter(
filterfieldname varchar(255),filterfieldvalue varchar(1024)
)
© 2013 IBM Corporation45
•Use radshell commands to set up filters for Event Logging
•Displays the current filter setting
•Clear the filter
•Adds fields and values to the current filter
•Removes a field from the filter
Event Logging with Filters
$TBSM_HOME/bin/rad_radshell
displayEventLoggingFilter();
clearEventLoggingFilter();
addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);
removeEventLoggingFilterValue(fieldName);
© 2013 IBM Corporation46
Filter examples: add a field to current filter:
addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);
•Only events with Node = “Mail3PortC” are logged, when the trace level for EventLogging package is finer
•addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”);
Event Logging with Filters
© 2013 IBM Corporation47
Event Logging with Filters
Filter examples: add a field to current filter:
addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);
•To change the filter, so that only events with Node = “Mail3PortC”,AND AlertKey = “mail”, are logged, when the trace level for EventLogging package is finer
•addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”);
•addUpdateEventLoggingFilterValue(“AlertKey”,”mail”);
© 2013 IBM Corporation48
Event Logging with Filters
Filter examples:
© 2013 IBM Corporation49
Event Logging with Filters
Filter examples: remove a field from current filter:
removeEventLoggingFilterValue(fieldName);
•To remove the Node fields from the filter (now filter will just match on AlertKey)
•removeEventLoggingFilterValue(“Node”);
© 2013 IBM Corporation50
Event Logging with Filters
Filter examples: remove a field from current filter:
clearEventLoggingFilter();
•Remove ALL filter fields – Logging will be done for all events if the trace level for EventLogging package is finer.
• clearEventLoggingFilter();
© 2013 IBM Corporation51
The filter set up by the addUpdateEventLoggingFilterValue radshell command, is only used when the trace level for the EventLogging package is set to finer.
If EventLogging package is not set, the radshell filter has NO effect.
Event Logging with Filters- Turn off afterwards
© 2013 IBM Corporation52