Developer and Fusion Middleware 1 _ Kevin Powe _ Log files - a wealth of forensic evidence.pdf

The most comprehensive Oracle applications & technology content under one roof The most comprehensive Oracle applications & technology content under one roof

Log files: A wealth of forensic evidence

Kevin Powe Integral Technology Solu6ons

The most comprehensive Oracle applications & technology content under one roof

More info at h:p://bit.ly/kapowelogs


Forensic process Log files Case files Tools


The Forensic Process


Step One: Secure The Scene


Operating System Evidence

netstat for network issues top or Windows Task Manager for CPU issues iostat or vmstat for I/O issues


Rolling Log Files


Cause

4-‐6PM 2-‐4PM

Symptoms


Step Two: Investigate The Scene


Don’t. Search. The. Log. Files.


‘Error’ versus ‘Warning’

‘Failing’ versus ‘Failed’


Step Three: Gather And Correlate Evidence


Step Four: Build A Hypothesis


1) Secure the scene 2) Investigate the scene 3) Gather and correlate evidence 4) Build a hypothesis




AdminServer

managedServer2

managedServer1

WebLogic Server Domain

Java processes


HTTP Access Logs


192.168.5.6 -‐ -‐ [19/Nov/2010:13:34:49 +0800] "POST /AccountServices/ProxyServices/AccountServices HTTP/1.1" 200 29487

192.168.5.6 -‐ -‐ [19/Nov/2010:13:34:49 +0800] "POST /WarehousingServices/ProxyServices/RequestOrderDetails HTTP/1.1" 200 1167

192.168.5.6 -‐ -‐ [19/Nov/2010:13:34:49 +0800]

"POST /WarehousingServices/ProxyServices/RequestOrderDetails HTTP/1.1“

200 1167

Remote host

rfc931

authuser

date

request

status bytes


ELF = Extended Logging Format


Extended Logging Format Fields Common format fields date 6me bytes sc-‐status

Network fields c-‐ip s-‐ip c-‐dns s-‐dns

Request fields cs-‐method cs-‐uri cs-‐uri-‐stem cs-‐uri-‐query

The Good Stuff cs-‐comment 6me-‐taken custom


Server log files



####<2/08/2011 12:49:35 AM EST> <No6ce> <Server> <brother-‐eye> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-‐tuning)'> <<WLS Kernel>> <> <> <1312210175933> <BEA-‐002613> <Channel "Default" is now listening on 10.0.2.15:7001 for protocols iiop, t3, ldap, snmp, h:p.>

####<2/08/2011 12:49:35 AM EST> <No6ce> <WebLogicServer> <brother-‐eye> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-‐tuning)'> <<WLS Kernel>> <> <> <1312210175933> <BEA-‐000331> <Started WebLogic Admin Server "AdminServer" for domain "example1030Domain" running in Development Mode>

<2/08/2011 12:49:35 AM EST> <Nodce> <WebLogicServer> <brother-‐eye>

Timestamp Severity Subsystem Machine

<AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-‐tuning)'>

Server Thread ID

<<WLS Kernel>> <> <> <1312210175933> <BEA-‐002613> <Channel "Default" is

User Txn ID Diagn. Time (msecs) Message ID Text


Debug flags


HTTP: weblogic.servlet.DebugH:p SSL: default.DebugSSL JDBC: weblogic.jdbc.sql.DebugJDBCSQL


<4/08/2011 07:47:35 PM EST> <Warning> <netuix> <BEA-‐423420> <Redirect is executed in begin or refresh ac6on. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=HomePage1.> Loaded index.jsp page Loaded index.jsp page Loaded index.jsp page <4/08/2011 23:20:34 PM EST> <Info> <Health> <brother-‐eye> <AdminServer> <weblogic.GCMonitor> <<anonymous>> <> <> <1311830434630> <BEA-‐310002> <86% of the total memory in the server is free>

TO <4/08/2011 07:53:38 PM EST> <No6ce> <WebLogicServer> <BEA-‐000365> <Server state changed to RUNNING> <4/08/2011 07:53:38 PM EST> <No6ce> <WebLogicServer> <BEA-‐000360> <Server started in RUNNING mode> <4/08/2011 07:53:49 PM EST> <Nodce> <Stdout> <BEA-‐000000> <Loaded index.jsp page> <4/08/2011 07:53:50 PM EST> <Nodce> <Stdout> <BEA-‐000000> <Loaded index.jsp page> <4/08/2011 07:53:51 PM EST> <Nodce> <Stdout> <BEA-‐000000> <Loaded index.jsp page> <4/08/2011 08:20:34 PM EST> <Info> <Health> <brother-‐eye> <AdminServer> <weblogic.GCMonitor> <<anonymous>> <> <> <1311830434630> <BEA-‐310002> <86% of the total memory in the server is free>


Oracle Service Bus tracing


JMS Message Logs


SOA Suite Diagnostic Logs




Case File #1 An Unbalanced Load


Load balancer

Sun Reverse Proxy

Sun Reverse Proxy

WebLogic Server

WebLogic Server


cat access.log* | awk ‘{ print $x }’ | sort | uniq

(where x = posi-on of the cookie in the log file)


Case File #2 Fear Of Commitment


Oracle Service Bus

Tuxedo




Tools Editors The Gun vi

Querying data find grep sed awk tail

Analysis Excel R Splunk


@kapowe

kevinpowe

kapowe