View
1.282
Download
0
Category
Preview:
Citation preview
Imagination at work.
Ajith NarayananTechnical Lead-Oracle ERP Configuration ManagementGE Healthcare, Bangalore, India8th Dec-2014, Liverpool, U.K.
Best Practices For Large Oracle Apps R12 Implementations
The views/contents in this slides are those of the author and do not
necessarily reflect that of General Electric Company or Oracle
Corporation and/or their respective affiliates/subsidiaries.
The material in this document is for informational purposes only, and is
provided on an AS-IS basis.
11/14/2014
3
Who Am I?Ajith Narayanan
Technical Lead – Oracle ERP Configuration Management
GE Healthcare, Bangalore, India
• 10 years of Oracle [APPS] DBA/Technical Architect experience.
• Oracle ACE Associate
• Blogger :- http://oracledbascriptsfromajith.blogspot.com
• Member:-OAUG, AIOUG
• Website Chair (2011-2013) – Oracle RACSIG http://www.oracleracsig.org
• AIOUG Real Application Clusters SIG Leader
11/14/2014
4
AgendaWhy do we need to implement best practices?
• Application Tier
• R12 Tech Stack
• Forms
• Response Time
• Web Applications
• Concurrent Manager
• Workflow Engine
• Taking Diagnostics Help
• CBO Statistics
• Application Patching
• Network & Latency Check
11/14/2014
5
Agenda• Database Tier
• CPU Utilization Observations and analysis • High DB_IO observation and correlation with CPU usage. • Identify Top SQLs • Effective usage of AWR, ADDM Reports • ORACHK Check (From Oracle RAC assurance team)
• ADR Command Interpreter (ADRCI)
• Cluster Callout scripts example.
• Looking At Issues Proactively & From Greater Heights
• References
• Q & A
11/14/2014
6
Oracle Applications R12
• Huge infrastructure
– Oracle Applications R12
– 8 Node middle tiers
– 16 Node RAC database tier.
– Database Size 40+ TB
– Annual DB growth of 2 TB
– Daily Orders: 45 K –3 M order lines
– 1500 online users
– 1M+ XML messages/day
Best Practices
• Benefits
– Enables us being proactive
– simple in terms of manageability
– Helps in support cost reduction.
– Time & effort saving
Why do we need to implement best practices?
11/14/2014
8
Upgrade Tech Stack
• R12 Tech Stack
• OC4J: Oracle AS 10g 10.1.3./ Weblogic Managed Servers (FMW)
• Forms: Oracle AS 10.1.2
• JAVA, JRE Plugin
Useful MOS Articles
� Oracle E-Business Suite Technology Stack Release Notes for Release 12.1.3 (MOS Note - 1098650.1)
� Oracle E-Business Suite Technology Stack Release Notes for Release 12.2*
� Oracle E-Business Suite - Technology Stack : Navigation (landing) Page (MOS Note - 1480550.1)
� Troubleshooting Assistant: EBS Technology Stack (Doc ID 1607365.2)
Application Tier
11/14/2014
9
Forms
• Socket Mode for Internal users (MOS 384241.1)
• Servlet mode is default in R12 and recommended for external access.– cat $FORMS_WEB_CONFIG_FILE|grep serverURL=
If the serverURL parameter has no value then Forms is implemented in socket mode else it is servlet
• Forms dead client detection (DCD) and Abnormal termination handler.– Value specified in minutes : FORMS_TIMEOUT=10
env | grep -i FORMS_TIMEOUT
– Terminates fwebmx processes for dead clients.
– Enable Forms Abnormal Termination Handle
– Do not set FORMS_CATCHTERM
– SQLNET.EXPIRE_TIME – Probes client connection at regular interval.
Useful MOS Articles
Application Tier
11/14/2014
11
� Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 (Doc ID 745711.1)
Response Time/CPU Issues
• When users complain about response time, configure Apache to log the time it takes to service a request.
• Access logs monitoring should be good way to keep a tab on the response times of user requests.
– Configure Apache to log the time it takes to service a request
– Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*
– Edit: $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.confLogFormat "%h %T
(Or)– Enable Execution Content ID's (ECID) for the access_log in Release 12. ECID's allow correlation of
requests across log files so that one can map the flow of a given request across the various components of iAS.
LogFormat "%h %l %u %t [ecid: %{Oracle-ECID}i] \"%r\" %>s %b [%T (secs)]" combined
Refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for getting http return code definitions
Note - R12.2 replaces Oracle 10gAS OC4J containers with Weblogic managed servers.
Application Tier
11/14/2014
12
Web Applications
• Train users to logout when transactions are complete.– Logging out gracefully releases the memory and corresponding resources (i.e. connections, etc.)
Releases process resources sooner
• JVM/OC4J Sizing + Heap Size– OACORE, OAFORMS, OAFM etc..
– Start with no more than 100 active users per JVM/OC4J instance.
– Start with 1 active JVM/OC4J instance per CPU core
Note:- Each 'core' in a multi-core system is considered to be a separate CPU. For example, a dual quad-core chip would be the same as stating 2 x 4 x CPU or 8 CPUs.
Useful MOS Articles
� JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 (Doc ID 362851.1)
� MOS Note 357597.1 on how to enable SQL trace for OA Framework applications
Application Tier
11/14/2014
13
EBS Concurrent Processing Analyzer
• The EBS Concurrent Processing Analyzer is a self-service health-check script
• Reviews the overall Concurrent Processing
• Analyzes the current configurations & settings for the environment.
• Providing feedback and recommendations on best practices.
• APPS Workload (Concurrent, FORM, ICX) observations.
• Mandatory Concurrent Programs fro purging.
Sample Output
Useful MOS Articles
Application Tier
2DEMO_cp_analyzer.html
11/14/2014
14
� Best Practices for Performance for Concurrent Managers in E-Business Suite [Doc ID- 1057802.1]
� Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)
EBS Workflow Analyzer
• The EBS Workflow Analyzer is a script that reviews the current Workflow Footprint.
• Analyzes the workflow configurations & Provides feedback and makes recommendations on best practices and areas of concern.
• Provides immediate analysis and output of the EBS Workflow environment.
• Workflow Process Stuck & Work Flow Checks
Sample Output
Useful MOS Articles
Application Tier
2DEMO_wf_analyzer.html
11/14/2014
15
� How to run EBS Workflow Analyzer Tool as a Concurrent Request (Doc ID 1425053.1)
� 11i-12 Workflow Analyzer script for E-Business Suite Workflow Monitoring and Maintenance [Video] (Doc ID 1369938.1)
� How To Use Concurrent Program "Purge Order Management Workflow" (Doc ID 878032.1)
OAM Diagnostics Help
• Oracle Application Object Library Best Practices: Database Security Tests Health Check Test
• Oracle Application Object Library Best Practices: E-Business Suite Security Tests Health Check Test
• Oracle Application Object Library Best Practices: E-Business Suite Diagnostic Tests Health Check Test
• And many more……
Application Tier
11/14/2014
16
CBO Statistics
• FND_STATS uses the standard the DBMS_STATS package, but enhances the functionality as it provides several additional PL/SQL procedures for gathering statistics at the table, schema, or database level.
• The FND_STATS package also supports histograms, table exclusions, extended stats, incremental stats gathering for partitioned tables, and concurrent statistics gathering.
Useful MOS Articles
Application Tier
11/14/2014
17
� Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID - 1586374.1)
� bde_last_analyzed.sql - Verifies CBO Statistics (Doc ID - 163208.1)
Application Patching
• Thorough patch analysis with timing reports on sand box & non-prods environments before proceeding with the Prod patching.
• In Release 12, you register your customized files in the Register Flagged Files tool in Oracle Applications Manager (OAM), which replaces the $APPL_TOP/admin/applcust.txt file.
Useful MOS Articles
Application Tier
11/14/2014
18
� See Note 9766881 "My server does not have internet connection Can I Still use Patch Wizard?“
� Patching Best Practices And Reducing Downtime (Doc ID 225165.1)
• There are two diagnostic tools available in the Oracle E-Business Suite:
The Network test Form – This form can be added to user’s menu , So that they can run the tests when they feel a performance degradation
Note:- Network latency should be always low (<3ms) and bandwidth should be high.
Application TierNetwork & Latency
11/14/2014
19
Useful MOS Articles
The Client Analyzer
Application TierNetwork & Latency
11/14/2014
20
� Oracle E-Business Suite Network Utilities: Best Practices (Doc ID 556738.1)
CPU Utilization Observations and analysis
• CPU spikes for more than 30 minutes continuously is a candidate for RCA.
• High DB_IO observation and correlation with CPU usage.
• I/O Thresholds: Logical RDS > 300K/sec, Physical RDS> 3000/Sec are candidate for RCA.
– How many of you think RAM access is 10,000 times faster than Physical disk access?
– In real world, LIO is only 25-100 times cheaper than PIO
– Internal locks & latch serialization mechanisms involved.
– Targeting only PIO counts(or high cache hit ratios) important pitfall to avoid. Even with no PIOs, a query can still be outrageously inefficient
– LIO are a critical component of query cost
Database Tier
11/14/2014
21
AWR, ASH, ADDM Reports
• Identify Top SQL, Latch Spin Counts etc., Memory Advisors
• Identify top wait events (System, User IO, Cluster, Concurrency, Configuration, Applications, Network)
• Fragmented Interface tables
• Object to be pinned – (Frequently used database objects)
• RAC Check (From Oracle RAC assurance team) RACcheck - RAC Configuration Audit Tool (Doc ID 1268927.1)
• ORAchk - Health Checks for the Oracle Stack (Doc ID 1268927.2)
• Sample ORACHK report
• ORAchk-Collection Manager
– ORAchk daemon auto-start mode after node reboot (init integration)
– Merge multiple ORAchk collection reports
– Exclude checks based on profile
– Upload of installed patches to database
– Collection Manager for ORAchk, RACcheck and Exachk (Doc ID 1602329.1)
– ORAchk signature file in /tmp on all nodes to verify last ORAchk run
– New checks and bug fixes, including o 30 Oracle Ebusiness AP module data integrity checks
– 12 new Database checks
– 8 new Solaris system checks
Database Tier
ORAchk-Report-Example.html
11/14/2014
22
• ADRCI has multiple features such as
– Analyze diagnostic data within the Automatic Diagnostic Repository (ADR).
– Create a package with incident / problem information for Oracle Support.
– Diagnostic data includes incident and problem descriptions, trace files, dumps, health monitor reports, alert log entries, and more.
– Run the Health Checks
– Helps in purging old diagnostics trace & dump files
– Simple to use a rich command set
Set correct ORACLE_HOME and just enter adrci
The utility starts and displays the following prompt: adrci>
Database Tier
11/14/2014
23
ADR Command Interpreter (ADRCI)
• At the ADRCI prompt, enter the following command: adrci> HELP
To get help for a specific ADRCI command: adrci> HELP command
• For example, to get help on the SHOW TRACEFILE command, enter the following: adrci> HELP SHOW TRACEFILE
adrci> set homepath diag/rdbms/orclbi/orclbi2 adrci> show homes ADR Homes: diag/rdbms/orclbi/orclbi2
• To view the alert log: 1. Start ADRCI in interactive mode, and ensure that the homepath points to the correct directory within the ADR base directory hierarchy.
Database Tier
11/14/2014
24
Useful MOS Articles
2. At the ADRCI prompt, enter the following command: SHOW ALERT
The following are variations on the SHOW ALERT command: SHOW ALERT -TAIL
SHOW ALERT -TAIL 50
SPOOL /tmp/MYALERT.LOG SHOW ALERT SPOOL OFF This outputs the alert log, without XML tags, to the file /tmp/MYALERT.LOG.
SHOW ALERT -P "MESSAGE_TEXT LIKE '%ORA-600%'"
Database Tier
11/14/2014
25
� 11G ADR DOCUMENTATION (Doc ID 445022.1)
� How to Invoke ADR Command Interpreter (adrci) in FMW home? (Doc ID 1669923.1)
Database Tier#!/bin/ksh# # Author: Ajith Narayanan## http://oracledbascriptsfromajith.blogspot.com## Version 1.0## This callout script is extended to report/mail the affected weblogic services when any Oracle cluster event occurs.##umask 022FAN_LOGFILE=$ORACLE_HOME/racg/usrco/`hostname`_uptime.logEVENTLINE=$ORACLE_HOME/racg/usrco/`hostname`_eventline.logEVENTLINE_MID=$ORACLE_HOME/racg/usrco/`hostname`_eventline_mid.logMAIL_CONT=$ORACLE_HOME/racg/usrco/`hostname`_mail.logWEBLOGIC_DS=$ORACLE_HOME/racg/usrco/weblogic_dsecho $* "reported="`date` >> $FAN_LOGFILE &tail -1 $FAN_LOGFILE > $EVENTLINEawk '{for (f = 1; f <= NF; f++) { a[NR, f] = $f } } NF > nf { nf = NF } END {for (f = 1; f <= nf; f++) { for (r = 1; r <= NR; r++) { printf a[r, f] (r==NR ? RS : FS) } }}' $EVENTLINE > $EVENTLINE_MIDSER=`grep "service=" $EVENTLINE_MID|awk -F= '{print $2}'`DB=`grep "database=" $EVENTLINE_MID|awk -F= '{print $2}'`INST=`grep "instance=" $EVENTLINE_MID|awk -F= '{print $2}'`HOST=`grep "host=" $EVENTLINE_MID|awk -F= '{print $2}'`STAT=`grep "status=" $EVENTLINE_MID|awk -F= '{print $2}'`if [ "$SER" != " " | "$DB" != " " | "$INST" != " " | "$HOST" != " " | "$STAT" != " " ]; thenif [ $STAT = nodedown ]; thencat $EVENTLINE_MID > $MAIL_CONT
11/14/2014
26
Cluster callout scripts. – Quickly get alerted on any cluster events.
echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS >> $MAIL_CONTelif [ $STAT = up ]; thencat $EVENTLINE_MID > $MAIL_CONTecho "**============================SERVICES RESTORED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONTelsecat $EVENTLINE_MID > $MAIL_CONTecho "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT#ficat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.comfi#cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.comfirm $EVENTLINE $EVENTLINE_MID $MAIL_CONT
Database Tier
11/14/2014
27
References
• My Oracle Support
• Steven Chan’s blog (https://blogs.oracle.com/stevenChan)
11/14/2014
29
Recommended