SAP BusinessObjects BI4 Log Analysis

Analyzing Logs for SAP BusinessObjects BI4!

EV Technologies!EV Technologies is an SAP BusinessObjects solutions firm !•  SAP Software Solutions Partner!•  SAP Certified Solutions provider!•  Sybase Certified Solutions provider!•  SAP BusinessObjects Enterprise Certified!•  SAP Mentors!•  SAP-PRESS Authors!•  ASUG Members/Volunteers!•  Migration experts – classic BusinessObjects products to

SAP BusinessObjects XI R2 – XI 3.1- BI4!!

Coy Yonce!

•  Manage product backlog and upcoming new features for Sherlock and related solutions!

•  Implement and advise customers on usage of Sherlock!

•  Previously with SAP for 11 years!–  6 years in support!–  3 years as a TAM!–  2 years as a Product Manager for CR!

!

Diversified Semantic Layer

•  Unprofessional journalism at its finest

•  Recorded by a bunch of guys in the SAP global community

•  Don’t miss podcasts both on product news and application, as well as interviews with other BOBJ people in the community

•  Follow on twitter at @dslayered

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

http://dslayer.net

We Approve this Message!

Agenda!

Troubleshooting & Monitoring!

Pesky User Errors

Performance and Stability

Con?nual Monitoring

Some pre-‐defined filters are no longer available in the Universe (IES

00003)

Connec?on or SQL sentence error (DA0005)

Internal error: Please contact your System Administrator. (CRS

200001)

Schedules are not comple?ng within a

defined window or “as quickly as they used to”

User logins are taking greater than

10 seconds to complete

Naviga?on “seems slow”

IntermiVent user error or service availability issue over an extended

period

Introduc?on of 500 new users and need to

monitor for impact on the deployment


Agenda!


Enable Server Logging!

For each service in CMC

For each applica?on in CMC


Enable Client Logging!

BO_Trace – Informa?on Design Tool WebIRichClient_trace – WebI Rich Client

designer_trace – Universe Designer

Log location!

BOBJ Service Log Loca/on Windows: <BOBJ Install Path>/SAP BusinessObjects Enterprise XI 4.0/logging/ UNIX: <BOBJ Install Path>/sap_bobj/enterprise_xi40/logging/lica?on server

Web Applica/on Log Loca/on $userHome/SBOPWebapp_$applica?on_$IPaddress_$port/. Where $userHome is the user’s home directory used to run your web applica?on server


End to End Trace!


Agenda!


Architecture!

Client Tier


Architecture!

Web Tier

Management Tier

Storage Tier


Architecture!

Processing Tier


Architecture!

Data Tier


Example Process Flow!


Troubleshooting Toolkit!Linux or Unix: Built-‐in binaries (e.g., grep, find, awk, uniq, lsof) Windows: Install UnxU?ls and Process Explorer

Develop a toolkit -‐  Log analysis (UnxU?ls, GLF Viewer) -‐  Database tes?ng and querying tool (RazorSQL, DatabaseBrowser, pklite) -‐  Archiving (WinRAR, 7-‐Zip) -‐  Network tracing (Wireshark, Fiddler) -‐  Process monitoring (Process Explorer) -‐  Web service tester (Wizdl)


Parsing Commands - GREP!


Parsing Commands - FIND!


Parsing Commands - FIND!


Parsing Commands - AWK!


Parsing Commands - AWK!

Parsing Commands - UNIQ!

See all unique errors for the same process ID with the same caller and callee

Parsing Commands - UNIQ!

See all unique errors from all logs regardless of process ID, caller, or callee


Parsing Commands - LSOF!


Format of logs!Loca/on|Guid|Time|Tzone|Trace|Log|Importance|Severity|Excep/on|DeviceName|ProcessID|ThreadID|ThreadName|ScopeTag|MajorTick|MinorTick|MajorDepth|MinorDepth|RootName|RootID|CallerName|CallerID|CalleeName|CalleeID|Ac?onID|DSRRootContextID|DSRTransac?on|DSRConnec?on|DSRCounter|User|ArchitectComponent|DeveloperComponent|Administrator|Unit|CSNComponent|Text

<< xs

<= s

== m

>= l

>> xl

Importance

S Success

W Warning

E Error

A Assert

F Fatal

Severity

Loca?on

Time

Trace

Importance

Severity

Excep?on

ProcessID

CallerName

CalleeName

Text

Important Fields


Format of logs!|6fed8a0b-‐f579-‐3784-‐78c0-‐188c8f4cd4b0|2012 12 27 15:55:55:303|-‐0600|Error|Error|>>|E| |WebIRichClient|11700|7124|| |1|12|1|0|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1|WebIRichClient.processDPCommandsEx|localhost:11700:7124.102:1|CrFvLIwIA0_yjUZ0gB1ZLVI46|||||||||||**ERROR:RequestProc:user: Administrator, doc: "SCI019 -‐ Desktop Intelligence Migra/on Planning", error stream: <ERRORS>

kc3cdbtbdacs.cpp:322:long __cdecl DoReportCSError(const class Connec/onServer::Error &,const int,const char *const ): TraceLog message 3 |e9d8fe68-‐9485-‐db74-‐6b85-‐6ae45762ea16|2012 12 27 15:55:51:755|-‐0600|Error|Error|>>|E| |WebIRichClient|11700|7124|| |1|0|1|0|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1|WebIRichClient.processDPCommandsEx|localhost:11700:7124.102:1|CrFvLIwIA0_yjUZ0gB1ZLVI46|||||||||||**ERROR:C3_dacs:CS::Error : (DBD(1),ID:769) "ORA-‐00936: missing expression " . [kc3cdbtbdacs.cpp;322]


Format of logs!kc3CoreEngineImpl.cpp:1750:enum IECore::IECoreStatusCode __stdcall IECore::kc3CoreEngineImpl::processDPCommandsEx(const class IECore::UTF8Buffer &,const class std::map<long,class std::vector<class IECore::CommandAeachment,class std::allocator<class IECore::CommandAeachment> >,struct std::less<long>,class std::allocator<struct std::pair<long const ,class std::vector<class IECore::CommandAeachment,class std::allocator<class IECore::CommandAeachment> > > > > &,class std::basic_ostream<char,struct std::char_traits<char> > &,class std::map<long,class std::vector<class IECore::OutputAeachment,class std::allocator<class IECore::OutputAeachment> >,struct std::less<long>,class std::allocator<struct std::pair<long const ,class std::vector<class IECore::OutputAeachment,class std::allocator<class IECore::OutputAeachment> > > > > &): TraceLog message 20 |7915e975-‐te7-‐f7b4-‐98b8-‐641128acecd7|2012 12 27 15:55:55:298|-‐0600|Error|Error|>>|E| |WebIRichClient|11700|7124|| |1|2|1|0|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1|WebIRichClient.processDPCommandsEx|localhost:11700:7124.102:1|CrFvLIwIA0_yjUZ0gB1ZLVI46|||||||||||**ERROR:IECore_kc3CoreEngineImpl:Caught a BOExcep/on: : ExecuteQuery/RefreshBatch failed [kc3CoreEngineImpl.cpp;1750]


Format of logs!../../shared/infostore_facade.cpp:81:unsigned long __cdecl InfoStoreFacade::GetConnec?onServerNames(const class ATL::CComPtr<struct ISEnterpriseSession> &,class std::vector<class ATL::CAdapt<class ATL::CComBSTR>,class std::allocator<class ATL::CAdapt<class ATL::CComBSTR> > > &): TraceLog message 3 |d3f3595e-‐a84d-‐a654-‐18a6-‐b4bc34c0160a|2012 11 25 07:21:32:035|-‐0800|Error| |>>| | |connec?onserver_EVTBOBJ12.Connec?onServer32| 6844|3592||}|2688|0|2|2|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:6644:1121.16164:1|webiserver_EVTBOBJ12.WebIntelligenceProcessingServer.openDocumentMDP|localhost:5156:2164.324:1|connec/onserver_EVTBOBJ12.Connec/onServer32.DBAliases|localhost:6844:3592.56:1|Cg3aECviNUxykU8buYB4IPM3c1f|||||||||CS||InfoStoreFacade::GetConnec/onServerNames: 0.172 Args=(session="3&1=System Account,0P&W={},2z&2r=evtbobj12:6400,8P&35=System Account,0P&3k=@evtbobj12:6400,8P&4E=4963JaihlCSvtpxhjjWxdMCINCs,0P&4F=10,0P&5U=4964JdIQkyNN2vPrgmTvWK7OZnS4963JaihlCSvtpxhjjWxdMCINCs,0P&63=server-‐token,0P&?u=1353856731473,9L&S5=1,08&Tm=36500,03&lu=1033,83&pa,0P&ua=AWmaEx4Z.NVPpAEthuTGAjc,0P&ub=AeLq4SaDG0dEqgPyY4C1Tp8,0P&ux=AdSP.rt24tREj8Jmr7nF_f8,0P&uy=-‐1352502874742,0L")

General Approach!

1.  Iden?fy applicable errors (notate errors and ?mes) 2.  Search for same ?me period from related logs 3.  Store all of this into a single text file 4.  Create a similar file of a “working” session 5.  Compare them to find differences 6.  Lookup error messages on SMP, SCN, & BOB 7.  Filter down to poten?al causes based on

understanding of architecture, process flow, components involved, error message, and knowledge of user workflow


User Errors!


Example: SQL Error!../odbcapi.cpp:399:short __cdecl WODBC3_MSSQLServer::ODBCAPI::SQLExecute(void *): TraceLog message 24256577 |69fedf49-‐25a9-‐6894-‐78c2-‐6400f5fd8b6a|2013 02 05 21:44:04:138|-‐0600|Informa?on| |>=| | |aps_EVTBOBJ12.Adap?veProcessingServer| 3284|8108||}|0|9|0|3|-‐|-‐|-‐|-‐|-‐|-‐||||||||||||CS::ODBCAPI::SQLExecute: 0 |87be3f6f-‐6131-‐dd44-‐c8fd-‐bea0e6955979|2013 02 05 21:44:04:140|-‐0600|Error|Error|>>|E| |aps_EVTBOBJ12.Adap/veProcessingServer| 3284|8108|| |0|10|0|2|-‐|-‐|-‐|-‐|-‐|-‐||||||||||||^[|ODBC^[|MS SQL Server 2008^[|JobId:2199138928^[|EXIT SQLGetDiagRec with return code 0 (SQL_SUCCESS) SQLSMALLINT 3 SQLHANDLE 0x00000000003FF160 SQLSMALLINT 1 SQLTCHAR * 0x000000006FBBDDD0 [42S02] SQLINTEGER * 0x000000006FBBDCB4 (208) SQLTCHAR * 0x000000006FBBDDE0 [[Microsol][SQL Server Na/ve Client 10.0][SQL Server]Invalid object name 'BOE140_Audit.dbo.Managed_En/ty_Status_Metrics'.] SQLSMALLINT 513 SQLSMALLINT * 0x000000006FBBDCB0 (123


Example: BI LaunchPad!|DBB5F33CD65E4177A2C808B20B476A351b|2013 02 25 14:07:38.184|-‐0600|Informa?on|Error|>=|E| |BIlaunchpad| 1092| 39|hVp-‐8080-‐5 | |12|0|0|0|BIlaunchpad.WebApp|evtbobj12:1092:39.466:1|-‐|-‐|BIlaunchpad.WebApp|evtbobj12:1092:39.466:1|Ci2QIVp7Lk7vmD5ZAzaOWwc1d0|||||||||com.businessobjects.adv_ivcdzview.DHTMLLogger||viewReport[IllegalArgumentExcep?on] <reference> (Error: RWI 00011) Invalid reference: refID=1.4, followBid=; Anchor=; Slots=/ |DBB5F33CD65E4177A2C808B20B476A351c|2013 02 25 14:07:42.369|-‐0600|Error| |>>| | |BIlaunchpad| 1092| 39|hVp-‐8080-‐5 | |424|0|0|0|BIlaunchpad.WebApp|evtbobj12:1092:39.471:1|-‐|-‐|BIlaunchpad.WebApp|evtbobj12:1092:39.471:1|Ci2QIVp7Lk7vmD5ZAzaOWwc1d5|||||||||com.sun.faces.renderkit.html_basic.LabelRenderer||component testSaveAs_SaveAsTypeLabel must be enclosed inside a form


Performance and Stability!


Example: Timings!../odbcapi.cpp:174:short __cdecl WODBC3_MSSQLServer::ODBCAPI::SQLBindParameter(void *,unsigned short,short,short,short,unsigned __int64,short,void *,__int64,__int64 *): TraceLog message 24253030 |c3b35919-‐21ff-‐2134-‐ca5e-‐138bdc261b70|2013 02 05 21:42:56:381|-‐0600|Informa?on| |>=| | |aps_EVTBOBJ12.Adap?veProcessingServer| 3284|5728||}|0|14|0|2|-‐|-‐|-‐|-‐|-‐|-‐||||||||||||CS::ODBCAPI::SQLBindParameter: 0.001 ||||||||||CS::ODBCAPI::SQLDescribeParam^^ ../odbcapi.cpp:346:short __cdecl WODBC3_MSSQLServer::ODBCAPI::SQLDescribeParam(void *,unsigned short,short *,unsigned __int64 *,short *,short *): TraceLog message 24253032 |c750414d-‐9f5c-‐f424-‐9a33-‐d7518d07cabb|2013 02 05 21:42:56:382|-‐0600|Informa?on| |>=| | |aps_EVTBOBJ12.Adap?veProcessingServer| 3284|5728||}|0|16|0|2|-‐|-‐|-‐|-‐|-‐|-‐||||||||||||CS::ODBCAPI::SQLDescribeParam: 0 ../odbcapi.cpp:399:short __cdecl WODBC3_MSSQLServer::ODBCAPI::SQLExecute(void *): TraceLog message 24253041 |af6ba143-‐f480-‐2a34-‐4a69-‐5964f19b87d0|2013 02 05 21:42:56:382|-‐0600|Informa?on| |>=| | |aps_EVTBOBJ12.Adap?veProcessingServer| 3284|5728||}|0|25|0|3|-‐|-‐|-‐|-‐|-‐|-‐||||||||||||CS::ODBCAPI::SQLExecute: 0


Example: Server Starting!.\ProcServerChild.cpp:636:-‐: TraceLog message 5 |cff5f301-‐de60-‐c874-‐7990-‐1a82cc1a99fc|2012 11 25 07:00:12:532|-‐0800|Error| |>>| | |cr2011proc_EVTBOBJ12.CrystalReports2011ProcessingServer_child_CRPE10| 7032|6836|| ||||||||||||||||||||||ras21-‐core: caught IPSSExcep?on: <RASServerExcep?on ( errCode=<ErrorFailure> message=<Internal error. Please contact your system administrator.> rootCause=<RASPROCREPORT0205> debugString=<CR config object not found OR the number of objects is not 1> )> .\ProcServerChild.cpp:1038:-‐: TraceLog message 6 |t9e4663-‐8ea5-‐7c24-‐6973-‐32e93eac64e5|2012 11 25 07:00:12:540|-‐0800|Error| |>>| | |cr2011proc_EVTBOBJ12.CrystalReports2011ProcessingServer_child_CRPE10| 7032|6836|| ||||||||||||||||||||||ras21-‐core: did not ini/alize sucessfully, so stopping immediately


Example: CMS Overloaded!(err.m_IDSError == IDS_ERR_CMSSYSTEMDB_NODATABASECONNECTIONSAVAILABLE : Error collec?ng resources: CMS system database is overloaded, please try again. If the problem persists, please contact the system administrator(FWB 00027) Wri/ng to the database Server Intelligence Agent: server SIAKWEB9VM.CentralManagementServer stopped with exit code = 1 (0x1).‑ |B3AF369E866B48DF969863AD19492A3713|2012 03 09 19:27:16.189|-‐0600|>>|E| |SIA_SIAKWEB9VM| 3516| 31|SIAKWEB9VM.CentralManagementServer (Process Agent)| ||||||||||||||||||||com.businessobjects.cad.syslog||[Node Name: SIAKWEB9VM] [User Name: SYSTEM]


Example: WPS Down!|01d8c282-‐b69c-‐cf94-‐9ae7-‐c322b6ce000e|2012 03 09 20:04:34:361|-‐0600|==| | | webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|5044|| | 159|236|3|2| CMC.WebApp|kweb9vm:6656:39.9950:1|cms_SIAKWEB9VM. CentralManagementServer.commitEx4|localhost:8476:8100.9967:1| webiserver_SIAKWEB9VM. WebIntelligenceProcessingServer. setParamEx|localhost:4396:5044.39:1| CqKqRpGoXEQQjHPyW2Diwec26db|||||||||||OSCASimpleFactory:: disabledServer() true


Example: WPS CPU!|25340224-‐4f65-‐5384-‐2955-‐a4729409dea4|2012 03 09 20:21:34:908|0600|>=| | |webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|4640|| |0|724|0|4||||||-‐||||||||||||GenericContainer_MetricManagerImpl:Total CPU /me (seconds):16.3594‑ |99b5b174-‐24e7-‐36f4-‐4b06-‐3cf8912t257|2012 03 09 20:28:05:455|0600|>=| | |webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|4640|| |0|968|0|4||||||-‐||||||||||||GenericContainer_MetricManagerImpl:Total CPU ?me (seconds):208.156‑


Example: WPS RAM!|332b35cd-‐a260-‐d924-‐9852-‐f0641f4c8e35|2012 03 09 20:24:42:455|-‐0600|>=| | |webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|8356|| |15369|82|1|6|BIlaunchpad.WebApp|kweb9vm:6656:39.9987:1|BIlaunchpad. WebApp|kweb9vm:6656:39.9987:1|webiserver_SIAKWEB9VM.WebIntelligence ProcessingServer.getMap|localhost:4396:8356.1828:1|CqKqRpGoXEQQjHPyW2Diwec2700||||||||||| GenericContainer_MetricManagerImpl:Memory level (0=normal, 1=low, 2=high, 3=max):0‑ |cdb8f014-‐2fce-‐ab64-‐39e1-‐0d74b9574308|2012 03 09 20:27:05:424|0600|>=| | |webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|2076|| |0|225|0|7||||||-‐||||||||||||GenericContainer_MetricManagerImpl:Virtual memory size (Mb):1043‑


Example: IFRS & OFRS!|da26f544-‐5c28-‐03c4-‐5829-‐4618bdd19a95|2012 03 13 07:32:46:002|-‐0500|>>| | |webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 9084|5636||}|2724|0|2|1|BIlaunchpad.WebApp|kweb9vm:6656:41.13148:1|webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer.openDocumentMDP|localhost:9084:5636.4457:1|CComFile.CopyTo|localhost:9084:5636.4457:314|CqKqRpGoXEQQjHPyW2Diwec3359|||||||||||Step: CopyTo: 0.141 InfoStoreOpera?onHelpers.cpp:1422:void __cdecl Helpers::InfoStore::ObjectFileHandle::extractContent(const class bo_u�8string &) const: TraceLog message 202744 kdgstoreCE.cpp:1054:long __cdecl StoreFacade::ImportState_orig(class bo_u�8string &,struct idgStore::DOCSPEC &): TraceLog message 202753 |fdb865ce-‐6d06-‐1aa4-‐4b2a-‐8756e85db811|2012 03 13 07:32:46:080|-‐0500|>>|E| |webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 9084|5636|| |2733|0|1|8|BIlaunchpad.WebApp|kweb9vm:6656:41.13148:1|BIlaunchpad.WebApp|kweb9vm:6656:41.13148:1|webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer.openDocumentMDP|localhost:9084:5636.4457:1|CqKqRpGoXEQQjHPyW2Diwec3359|||||||||||**ERROR:dg_storeCE:Unable to retrieve the document from the FRS [kdgstoreCE.cpp;1054]


Continual Monitoring!


Continual Monitoring!

SAP BOBJ

SAP BOBJ

SAP BOBJ

SAP BOBJ

SAP BOBJ

Log Processing

Log Storage


Agenda!


Automation!

How do they work?!

2012 12 27 15:55:51:755

**ERROR:C3_dacs:CS::Error : (DBD(1),ID:769) "ORA-‐00936: missing expression " . [kc3cdbtbdacs.cpp;322]

Caller: Webi SDK.CorbaServerImpl.doProcess()

Callee: WebIRichClient.processDPCommandsEx

Data store

How do they work?!

How do they work?!


Agenda!


More Information!•  Timothy Coffman and Toby Johnston with SAP @ Remote Supportability Blog •  SAP BusinessObjects BI4 Process Flows •  Business Intelligence Pla�orm Administrator Guide •  SAP BusinessObjects BI System Administra?on – Eric Vallo & Greg Myers @ SAP Press •  Universal Command Guide: For Opera?ng Systems – Guy Lotgering Command References •  Find •  Awk •  Lsof •  Grep •  Uniq

More Information!

Contact: Coy Yonce!Email: [email protected]!On the Web: evtechnologies.com!Twitter: @coyyonce!