ebs

Gather stats fails after upgrade to Oracle 10GGather Schema Statistics or Gather Table Statistics or running FND_STATS fails with "ORA-20005: object statistics are locked".

Example:

ERROR: While GATHER_TABLE_STATS: object_name=AR.AR_REV_REC_QT***ORA-20005: object statistics are locked (stattype = ALL)***

This can happen with Advance Queue tables.

In 10g, if a queue is created or recreated during the upgrade, automatic statistics gather gets locked (disabled) on these queue.The following statement can be used to check the tables which have statistics locked:

select owner, table_name, stattype_locked from dba_tab_statistics where stattype_locked is not null;

SOLUTION:

Unlock gather stats running on those queues :

exec dbms_stats.unlock_schema_stats('schema_owner'); exec dbms_stats.unlock_table_stats('table_owner','table_name');

Example:SQL> exec dbms_stats.unlock_schema_stats ('AR'); SQL> exec dbms_stats.unlock_table_stats('AR', 'AR_REV_REC_QT');

How to resolve cloning issue: BEGIN failed--compilation aborted at adcfgclone.pl line 26.

While running adcfgclone.pl as a part of database Tier clone, the below mentioned error occurs:

-bash-3.00$ perl adcfgclone.pl dbTechStack

Can't locate strict.pm in @INC (@INC contains: /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/5.8.3/i686-linux-thread-multi /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/5.8.3 /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl/5.8.3/i686-linux-thread-multi /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl/5.8.3 /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl .) at adcfgclone.pl line 26.BEGIN failed--compilation aborted at adcfgclone.pl line 26.

As indicated in the above error message, this is due to the PERL setting. We dint have the PERL5LIB set :

-bash-3.00$ echo $PERL5LIB

-bash-3.00$ which perl/d02/oradev/devdb/10.2.0/perl/bin/perl

Resolution: $PERL5LIB was set to resolve the issue:

Contents under $ORACLE_HOME/perl :

-bash-3.00$ ls -ltr /d02/oradev/devdb/10.2.0/perl

total 16drwxr-x--- 2 oradev dba 4096 Feb 9 04:33 includedrwxr-x--- 2 oradev dba 4096 Feb 9 04:33 bindrwxr-x--- 4 oradev dba 4096 Feb 9 04:33 libdrwxr-x--- 4 oradev dba 4096 Feb 9 04:33 man

Contents under $ORACLE_HOME/perl/lib:

-bash-3.00$ ls -ltr /d02/oradev/devdb/10.2.0/perl/libtotal 8drwxr-x--- 3 oradev dba 4096 Feb 9 04:33 site_perldrwxr-x--- 44 oradev dba 4096 Feb 9 04:33 5.8.3

-bash-3.00$ PERL5LIB=/d02/oradev/devdb/10.2.0/perl/lib/5.8.3-bash-3.00$ export PERL5LIB

-bash-3.00$ id

uid=500(oradev) gid=502(dba) groups=502(dba)

-bash-3.00$ echo $PERL5LIB/d02/oradev/devdb/10.2.0/perl/lib/5.8.3

Issue running adcfgclone.pl on Database Oracle home.

While running adcfgclone.pl as oracle user, the session just hangs without giving any error in the log file. The issue was that it wasn’t able to connect to the database as: sqlplus "/as sysdba"

Deep dive:

Most generally this can occur due to improper shutdown of the database. While in the course of a database shutdown, if any of the oracle process is not completed killed and it holds a large shared memory segment, such issues can occur which would prevent from normal database operation.

So next time when an instance tries to start it is not able to grab a large chunk of memory to get started.

This can be checked as follows:

After the database has been shut down , there shouldn’t be any processes running as oracle on the server level:

ipcs -m | grep <oracle user>

eg:

ipcs -m | grep oracle

key shmid owner perms bytes nattch status0x185ee230 884763 oracle 640 789785472 0

Here in the above example, a single oracle process holds 789785472 bytes of shared memory, even after shutdown. If the database instance is started, then we would expect abnormal behaviors on the database side, like the login issue that we are discussing now.This can cleaned as mentioned below:

ipcrm -m < shmid>

eg:

ipcrm -m 884763

This would free up the memory held.

This resolved the issue.

Issue starting 10g Forms and reports server

Error:

Ping URL for proc: 887947305 failed while pinging url:/reports/rwservlet/pingserver?start=auto

12/11/01 08:18:46 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_BI_Forms~default_island~1 (887947305:8511)

12/11/01 08:19:10 [2] [libopmnoc4j] AJP13: Receiving response code: 500 while configured success code is: 200

12/11/01 08:19:10 [2] [libopmnoc4j] Ping URL for proc: 887947305 failed while pingingurl:/reports/rwservlet/pingserver?start=auto

Search String:

REP-56105

tail –f OC4J~OC4J_BI_Forms~default_island~1

ConfigFileName: /u01/oracle/product/ais10g/forms90/server/forms/server/formsweb.cfg

REP-56105: Engine rwEng-0 died with error

http://knoworacleappsdba.blogspot.in/2012/10/issue-starting-10g-forms-and-reports.html

SOLUTION:

Bug no: 3557723

Work Around:

Disable pingserver command in $ORACLE_HOME/opmn/conf/opmn.xml- Take the backup of opmn.xml file

@ cd $ORACLE_HOME/opmn/conf

@ cp opmn.xml opmn.xml.PINGAUTO

@ - Open file opmn.xml

@ - Take out the following lines from opmn.xml

@ < category id="urlping-parameters">

@ < data id="/reports/rwservlet/pingserver?start=auto" value="200"/>

@ < /category>

@ - Bounce the opmn

@ cd $ORACLE_HOME/opmn/bin

@ ./opmnctl stopall ; ./opmnctl startall

Error: CACHE mismatching

SOLUTION:

Also Check for the domain name in /u01/oracle/product/ais10g/webcache

Webcache.xml ----change the domain name.

Issue: Concurrent request running from a very long time

.Application version: 11.5.10.2 multi-node (4 nodes)Database: 11g RAC

Scenario:

Business critical concurrent request is running from more than 2 hours which was supposed to complete within 1 hour.The standard manager and the manager responsible to execute the critical concurrent request (custom manager) were overloaded with many requests. The server on which these managers run had very high server load and all CPU’s were utilized.

After diagnosis, we had to kill few requests which were running from more than 2 hours and were creating load on the server. These requests were in fact not performing anything and had gone zombie.

The server load came down and CPU’s available were set free to execute new requests/processes.

However the business critical request was still running and wasn’t moving ahead at all. The database session associated was also not executing anything.

Challenge:

Since this being a critical business request, we couldn’t terminate the request coz we had already lost few hours running it. At the same time the database session also wasn’t performing any execution and the whole request was not going anywhere even though it showed running normal status.

Trick:

Not recommended, however can be a life saver at times.

---- Forcibly change the status of the request to pending normal.---- Kill the database session of the request.

Use the below scripts:

update applsys.fnd_concurrent_requests set phase_code = 'P', status_code = 'I',

actual_start_date = null, crm_release_date = null, controlling_manager=null, logfile_name = null, logfile_node_name = null, outfile_name = null, outfile_node_name = null, crm_tstmp = nullwhere request_id in ('request-id');

alter system kill session 'sid, serial#, @instance-id';

Hence a new database session will get created as soon as the old one is killed and the request starts running. Since the server on which the request is supposed to get processed also had no issues now, the request went through very fast without any issues ;)

Login issues in Oracle Apps R12

Application version: 12.0.4 running on 11g RAC database.

None of the users are able to login to the Oracle Apps R12 URL. The URL just hangs without giving any information.

All the application services were up. Couldn’t find any significant load on the CPU and the number of concurrent requests running at the time was normal.

The http error_log had the below error:

oc4j_socket_recvfull timed out

The Application log has the below error:

12/04/26 03:15:13.511 html: Servlet errorjava.lang.OutOfMemoryError: Java heap space12/04/26 03:50:35.503 html: Servlet error

The opmn log had the below error:

12/04/26 03:15:00 java.lang.OutOfMemoryError: Java heap space12/04/26 03:15:02 java.lang.OutOfMemoryError: Java heap space12/04/26 03:15:06 java.lang.OutOfMemoryError: Java heap space12/04/26 03:15:13 Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap spaceoracle.jbo.DMLException: JBO-26066: Error during rollback

Solution:

We would need to do changes to -Xms ( for start heap memory ) and -Xmx ( for maximum heap memory ) of the opmn.xml. Changing the Xmx to 512 MB and Xms to 128 MB will help.

>>>> Increase Java Heap Size for the OC4J instance as per the available memory (RAM) & restart the instance. You can do it with AS Control -> OC4J Instance -> Administration Tab -> Server Properties -> Java options. Update -Xms -Xmx java options. Once updated, stop and start the OC4J instance to pick up the changes.

Login issues in Oracle EBS 11i after Clone ( from hotbackup)

Error in login page:

ORA-01110: data file 98: '/disk09_250/oradata/mine/a_txn_data14.dbf' ORA-26040: Data block was loaded using the NOLOGGING option

SOLUTION:

1. Compile jsp:cd $FND_TOP/patch/115/binperl ojspCompile.pl --compile --flush

2. Access the form directly using the URL:http://sanop.mine.com:8020/forms/frmservlet?form_params=config=PRODMINE

3. Run the request:"Synchronize WF LOCAL tables" - Parameters: ALL"Synchronize WF LOCAL tables" - Parameters: FND_USER

OACORE fails to start after increasing the JVM processes in Oracle R12

Issue: OACORE processes are not starting.

Recent changes: Increased the JVM processes in the context file.

Error:

adoacorectl.sh: exiting with status 206

Solution:

1) Stop the middle tier processes:

adopmnctl.sh stop

2) Kill any runaway/zombie processes.

3) Remove the following jms lock files:

$ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence/forms_default_group_1/jms.state.lock,

$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence/oacore_default_group_1/jms.state.lock

$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence/oafm_default_group_1jms.state.lock

4) Restart middle tier:adopmnctl.sh start

Restore FNDLIBR executable

Restore FNDLIBR executable which has been deleted accidentally:

1. Invoke adadmin on concurrent manager node2. Select option 2. Maintain Applications Files menu3. Select 1. Relink Applications programs4. Enter FND when it prompts forEnter list of products to link ('all' for all products) [all]5. When prompted forGenerate specific executable for each selected product [No] ? YESselect YES& from list of executable select FNDLIBR.

This will create new FNDLIBR executable.

Move Concurrent Processing Server from one node to another in Oracle Apps 11iExample:

Oracle E-Business Suite (11.5.10.2) with 2 nodes:

Node 1: Database and the Concurrent ProcessingNode 2: Forms, Reports and Web Server.

Goal: To Move the concurrent processing from Node1 to node 2.

Requirement: Shared appl_top / Shared application tier file system. If not follow the Step 2 below.

How to perform:

1. This is recommended to be performed on a NON-PRODUCTION system first with necessary backups.

2. Create a shared APPL_TOP by following Metalink Note 233428.1 for sharing the application tier filessystem and allow the node 2 to share for concurrent manager node ( Node 1).

3. Edit context file on the Forms and Web node2 and change following context variables:

s_isConc from "NO" to " YES "s_concstatus from "disabled" to " enabled "

On Node1: make the below changes

s_isConc from "YES" to "NO"s_concstatus from "enabled" to "disabled"

4. Run autoconfig on all the APPS nodes.

Concurrent managers are down with status: “System Hold, Fix Manager”

Issue:

All the concurrent managers are down with a status message: system hold, fix manager.

The administer Concurrent manager screen has Actual: 0 and Target: 1 for all managers.

Solution: Ensure all application services are down.

- Please check the system profile: Concurrent: GSM Enabled. This should be set to ‘Y’

- Execute: afdcm037.sql Path : $FND_TOP/patch/115/sql

Description of the script:

PL/SQL script to create libraries for FNDSM and FNDSVC and create Managers for Preexisting Nodes. Future nodes will be handled by db trigger.

- Relink the below mentioned executables: adrelink.sh force=y “fnd FNDLIBR” adrelink.sh force=y “fnd FNDSM” adrelink.sh force=y “fnd FNDFS” adrelink.sh force=y “fnd FNDCRM”

- Execute cmclean.sql to clean out the concurrent manager tables.

- Start all the application services and restest the issue.

ORA-00257: archiver error. Connect internal only, until freed

While trying to connect to the database, we get the below mentioned error:

ERROR:ORA-00257: archiver error. Connect internal only, until freed.

Cause:

- The error is caused by the fact that the database archiver process is unable to archive the current online redolog due to lack of space in the destination for the archivelogs.

- The flash_recovery_area is full.

Applicable to 9i, 10g and 11g database versions.

Solution:

The basic idea is to free space so as to make room for new archiving to happen. This can be addressed in multiple ways.

- Allow more space in the DB_RECOVERY_FILE_DEST with the DB_RECOVERY_FILE_DEST_SIZE parameter :

alter system set db_recovery_file_dest_size=5G ;

- Imagine a situation where the space increased in step 1 gets filled, ie, 5 GB. Set an alternative archiving destination, so that when the primary destination is full, archiving is automatically performed to the alternate destination:

Following changes in parameter file can enable the same:

log_archive_dest_1='LOCATION=use_db_recovery_file_dest NOREOPEN ALTERNATE=LOG_ARCHIVE_DEST_2' log_archive_dest_2='LOCATION=/other_dest_for_archiving' log_archive_dest_state_1='enable' log_archive_dest_state_2='alternate' db_recovery_file_dest='/db01/mine/oracle/product/10.1.0/db_1/flash_recovery_area' db_recovery_file_dest_size=5G

- Check the archive_log_dest and verify if the path mentioned exists and is valid.

- Take a backup of the existing archive files (recommended) OR it can be deleted to make more free archiving space.

This can be done at the server level where files can be manually deleted. However incase of ASM storage with rman, below mentioned way is recommended (which would perform backup and then deleting):

Example :

rman target sys/sys RMAN> backup archive log all ; RMAN> delete archive until time 'trunc(sysdate)';

Unable to authenticate session in Oracle apps R12 (12.0.4): Login Issue

Unable to authenticate session in Oracle apps R12 (12.0.4): Login Issue

The Release 12.0.4 instance was cloned from another identical instance.

When we try to login, the login page does not display

Have tried clearing the cache (_pages). Have tried recompiling jsps and bouncing however the problem still persists.

Cleared all the Apache and related logs and found an interesting error.

It is in the following log file:

$INST_TOP/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

08/08/21 15:17:00.601 10.1.3.0.0 Started08/08/21 15:17:02.769 html: 10.1.3.0.0 Started08/08/21 15:17:33.132 html: oracle.apps.fnd.security.LeakDetectionFilter.initialized:121934985313208/08/21 15:17:33.132 html: LeakSetting:global:true,session:false,aggressive:false,stderr:false,appslog:false,corelog:falseignoring:null08/08/21 15:17:34.859 html: chain failedjavax.servlet.ServletException: java.lang.RuntimeException: Unable to create user session. Please contact

your System Administrator.

at com.evermind[Oracle Containers for J2EE 10g

(10.1.3.0.0) ].server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:780)

Solution:

Enable FND debug

The FND debug log had the below error:

Oracle error -1654: ORA-01654: unable to extend index APPLSYS.FND_LOGINS_N2 by 16 in tablespace

APPS_TS_ARCHIVE has been detected in FND_SIGNON.AUDIT_USER.

Check if the APPS_TS_ARCHIVE Tablespace is full.

Increase the size of tablespace APPS_TS_ARCHIVE / add a datafile to the tablespace. This will resolve the issue.

ICX: Session Timeout and autoconfig variable s_oc4j_sesstimeout in 12.0.4

Recently we came across a forms time out issue. All the user forms sessions were getting disconnected after an Hour. The FORMS_TIMEOUT variable was set to 120 mins though. This should practically disconnect the forms session after 2 hours.

Here the reason for the issue is because of the mismatch of the ICX and OC4J timeout session parameters.OC4J session timeout, if not set, defaults to 20 minutes. This will cause the oc4j session to timeoutbefore the Apps/ Forms session timeout which is defined by profile option "ICX: Session Timeout"

Profile option "ICX: Session Timeout" and autoconfig variable "s_oc4j_sesstimeout" in context file should match.

Example: In case you need the session to timeout after 2 hrs then the profile option "ICX: Session Timeout" and autoconfig variable "s_oc4j_sesstimeout" should be set to 120.

Unable to apply RDBMS patch (Opatch) on 10G ENV after postclone and upgrade from 9i.

UNABLE TO APPLY RDBMS PATCH (Opatch) ON 10G ENV after postclone and upgrade from 9i.

SOLUTION:Some options of opatch:Opatch apply –invPtrLocOpatch apply –no_inventoryWhich opatch~/Oraclemine/10.2.0/OPatch/opatch(Oracle06) 5005469 - -bash $ opatch apply -invPtrLoc $ORACLE_HOME/oraInst.locInvoking OPatch 10.2.0.2.0. Oracle interim Patch Installer version 10.2.0.2.0@ Copyright (c) 2005, Oracle Corporation. All rights reserved... Oracle Home: /mine/oracle/oraclemine/10.2.0Central Inventory: /mine/oracle/oraclemine/10.2.0/oraInventory from : /mine/oracle/oraclemine/10.2.0/oraInst.locOPatch version: 10.2.0.2.0

OUI version: 10.2.0.2.0OUI location: /mine/oracle/oraclemine/10.2.0/ouiLog file location: /mine/oracle/oraclemine/10.2.0/cfgtoollogs/opatch/opatch-2006_Apr_25_21-58-34-PDT_Tue.log. ApplySession applying interim patch '5005469' to OH/mine/oracle/oraclemine/10.2.0'List of Homes on this system:OPatchSession cannot load inventory for the given Oracle Home /mine/oracle/oraclemine/10.2.0. Possible causes are: No read or write permission to ORACLE_HOME/.patch_storage Central Inventory is locked by another OUI instance No read permission to Central Inventory The lock file exists in ORACLE_HOME/.patch_storage The Oracle Home does not exist in Central Inventory.Apply Session failed: ApplySession failed to prepare the system. No detail available.System intact, OPatch will not attempt to restore the system.OPatch failed with error code 73===========================================================================I have checked that patch_lock file is not available on $ORACLE_HOME/.patch_storage and no lock on Inventory. Also required permissions are exisitng for .patch_storage and oraInventory. .lsinventory gives the following info===========================================================================(oracle06) 5005469 - -bash $ opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.locInvoking OPatch 10.2.0.2.0. Oracle interim Patch Installer version 10.2.0.2.0@ Copyright (c) 2005, Oracle Corporation. All rights reserved...Oracle Home : /mine/oracle/oraclemine/10.2.0Central Inventory : / mine /oracle/ oraclemine /10.2.0/oraInventory from : / mine /oracle/ oraclemine /10.2.0/oraInst.locOPatch version : 10.2.0.2.0OUI version : 10.2.0.2.0OUI location : / mine /oracle/ oraclemine /10.2.0/ouiLog file location : / mine /oracle/ oraclemine /10.2.0/cfgtoollogs/opatch/opatch-2006_Apr_25_22-00-16-PDT_Tue.log .List of Homes on this system:Inventory load failed... OPatch cannot load inventory for the given Oracle Home.Possible causes are: Oracle Home dir. path does not exist in Central Inventory Oracle Home is a symbolic link Oracle Home inventory is corruptedLsInventory Session failed: OracleHomeInventory::load() gets null oracleHomeInfo .OPatch failed with error code 72=========================================================================== Looks like $ORACLE_HOME is not attached to the local inventory. Attaching the ORACLE_HOME to the inventory.Run runClone.sh as mentioned under "Run 10g Cloning" step . This will resolve the issue

How to recover a cloned database.

Source Instance:

oracle@mine01$ sqlplus "/as sysdba"

SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 03:26:33 2007Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.Connected to:Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit ProductionWith the Partitioning, OLAP and Oracle Data Mining optionsJServer Release 9.2.0.7.0 – Production

SQL> !pwd/d02/oracle/proddb/9.2.0/admin/PROD_mine01/udump

SQL> alter session set tracefile_identifier=mine; ( this would set the trace file name as mine)

Session altered.

SQL> alter database backup controlfile to trace;

Database altered. oracle@mine01$ ls -ltr *MINE*

-rw-rw-r-- 1 oracle dba 67001 Apr 21 03:28 prod_ora_23305_MINE.trc

uuencode prod_ora_23305_MINE.trc prod_ora_23305_MINE.trc|[email protected] ( This would send an email to your mailbox with the trace file attached) Get this controlfile to target instance.

Target Instance:

oradev@ mine02# ls -ltr *MINE*

-rwxrwxrwx 1 oradev dba 67001 Apr 21 03:29 prod_ora_23305_MINE.trc

oradev@mdsuaor09# cp prod_ora_23305_MINE.trc mine.sql

This would copy the trace file to a new file named mine.sql and all the required changes can be done to mine.sql as mentioned below:

vi mine.sql

esc:se nu

check where the create controlfile line starts ( double copy of create control file line)and delete till there.

esc:1,60d esc shift+g -------will take to last line

esc:?CREATE --------press n and check if there are 2 sets of create controlfile

if yes then bring the cursor to the last line of the wanted set

escd shift+g change like this :

CREATE CONTROLFILE SET DATABASE "DEV" RESETLOGS NOARCHIVELOG

Now change the path of data files and save the file .

esc:%s,oracle/proddata,oradev/devdata,g

The entire practice above is to create a new set of controlfile and below mentioned steps would check if all datafiles have been copied properly:

oradev@ mine02# grep / mine.sql > mine.sh

vi mine.sh

esc:%s,',,g

esc:%s/,//g esc:%s,^,ls -ltr ,g

save the file . chmod +x mine.sh

./mine.sh 2> mine.log

check the log file and see if there any data file missing.( mine.log file size shud be zero )

At this point we are ready with the control file and we are also sure that all data files have been copied to the target instance.

oradev@ mine02# sqlplus "/as sysdba"

SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 03:59:48 2007Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.Connected to an idle instance.

SQL> startup nomount;

Now open another session and keep monitoring the alert log file.

SQL> @mine.sql

Control file created.

SQL>recover database using backup controlfile;

Meanwhile go to the source instance and go to alert log file and do a tail -500 and check for the path of the log file using the sequence number asked for recovery.

Change the path accordingly and give it for the recovery. SQL> recover database using backup controlfile;

ORA-00279: change 7254772161098 generated at 04/20/2007 14:08:25 needed forthread 1ORA-00289: suggestion : /d02/oradev/devdb/9.2.0/dbs/arch1_88922.dbfORA-00280: change 7254772161098 for thread 1 is in sequence #88922

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}/d04/oradev/devdata/log4a.dbf

Log applied.Media recovery complete.

( In this case the sequence number is 88922, hence check in source instance for the log file path for this

sequence number . This will be there in alertlog file. Then change the path of log file according to the target

instance and give it. Sometimes this will ask for multiple files with different sequence number)

SQL> select * from v$recover_file;

no rows selected

SQL> alter database open resetlogs;

Database altered.

SQL> select * from v$tempfile;

no rows selected

Check the alert log file while performing the below task.

There should be no errors.

SQL> shutdown immediate;

Database closed.Database dismounted.ORACLE instance shut down.SQL> exitDisconnected from Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit ProductionWith the Partitioning, OLAP and Oracle Data Mining optionsJServer Release 9.2.0.7.0 - Productionoradev@ mine02# sqlplus "/as sysdba"

SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 04:26:54 2007

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Connected to an idle instance.

SQL> startup;

ORACLE instance started.

Total System Global Area 1487899976 bytesFixed Size 732488 bytesVariable Size 973078528 bytesDatabase Buffers 503316480 bytesRedo Buffers 10772480 bytesDatabase mounted.Database opened.SQL> exit

At this point there shouldn't be any errors in the alertlog.

Start the listener:

oradev@ mine02# lsnrctl start DEV

Then Drop temp tablespace and create new temp tablespace. Then add temp files into it.

Change sys and system passwords if required.

OPMN and process monitoring in 10G Application server

OPMN and process monitoring

Oracle Process Manager (PM) is the centralized process management mechanism in Oracle Application Server and is used to manage Oracle AS processes. It starts, stops, restarts and detects death of these processes. The Oracle AS processes that PM is configured to manage are specified in the opmn.xml file. The start and stop functions are executed by the OPMN as per the user commands.

OPMN (Oracle Process monitoring and Notification services ) are a set of processes that manage mid-tier Application Server components like Oracle HTTP Server (Apache) and OC4J containers. OPMN consists of the Process Manager and the Notification Server.

Oracle Notification Server (ONS) is the transport mechanism for failure, recovery, startup and other related notifications between components in Oracle Application Server. It operates according to a publish-subscribe model: an Oracle AS component receives a notification of a certain type as per its subscription to ONS. When such a notification is published, ONS sends it to the appropriate subscribers.

Four parameters determine the behavior of the Oracle Process Manager and Notification services process in managing the iAS middle tier comprising of OC4J instances and the Apache HTTP server. They are

a) restart-on-death b) ping timeout c) ping interval d) reverse-ping timeout

The settings for these parameters need to be governed by the heap sizing of the OC4J container JVMs, the latencies involved with garbage collection algorithms and the response times of the HTTP server.

The way in which the parameters affect the functioning of OPMN is as follows:

- OPMN pings the process managed by it,ie, OC4J instances and http server periodically and expects a response within a certain time out period. OPMN would try this 3 times before it declares the process as dead (due to no response) and then it would kill and restart the particular process. OPMN manages the failover and availability of the processes it manages.

Reasons for Oracle HTTP Server not responding to OPMN ping could be:- due to a high load of concurrent requests which requires many http server attention. - because of timeouts between the various modules of Apache and the servers servicing client requests - bulk data received from any of the modules.- all new connections are not getting processed due to the server being fully occupied. - it can spawn, with many connections remaining in the CLOSE_WAIT state - due to thrashing when NFS hiccups cause files that need to be served to be unavailable to it - because of synchronization issues with the various mutexes that it needs to support for the proper functioning of its modules.

Reasons for OC4J containers not responding to OPMN ping could be:While the container is processing servlet or EJB logic within the JVM within which it runs, new objects get created in its heap memory area all the while.

When the garbage collection thread starts to run, it looks for objects that it can release to the heap memory pool based upon several algorithms that depend upon the kind of references the objects have to themselves. Since the collection is "generational", i.e., objects having references are promoted to an older generation and presumed to have a longer lifetime, objects with weak references are candidates for "cleaning" up, and their occupied memory gets released to the global heap of the JVM.

In this way, memory is reclaimed back into the the heap memory pool and made available for use in the creation of newer objects. The forays made by the garbage collector to reclaim memory in the heap are governed by several algorithms and every such collection takes a finite amount of time during which no other work of application processing is possible. And when the collection is over the entire heap, the full GC consists of a mark/sweep/compact cycles that "mark" the memory to be reclaimed, "sweep" the memory into the corresponding

generations and "compact" the holes created when the memory is reclaimed, in order as to create contiguous memory for future object creation.

These strategies consume more time, as to be expected, and can result in a delay in the container responding to an OPMN ping cycle. During such full GC scans, OPMN can and will kill and restart the container, causing it to lose the state of the application or request it was processing at that time.

Since full GC scans (referred to to as stop-the-world scans) can happen at any time during the lifetime of a request or an application, there is always the danger of OPMN killing a perfectly functioning container on the assumption that it was "hung" since it was "unresponsive". Heap memory settings for OC4J containersChanges are made in the opmn.xml file in the <java-option> sections for each OC4J instance:

-Xms ( for start heap memory ) and -Xmx ( for maximum heap memory ).

The recommended settings for the -Xmx value are 512MB as typical applications need that much of memory to avoid java.lang.OutOfMemoryError exceptions as seen from experience.

Start with an -Xms value of 128MB to prevent side effects of "Too many files open" errors when this setting is higher, as garbage collection kicks in only later for higher -Xms values, resulting in open file handles not getting released by the GC.

<java-option>-server -Xms128M -Xmx512M

Thread pool sizingIn the server.xml file, set the thread-pool sizes as follows for optimum operation of the thread pool:

<global-thread-pool min="40" max="40" queue="80" keepAlive="-1"/>

This sets the min and max thread-pool sizes to the same value and the keepAlive parameter to "-1" - recommended for production environments, this will ensure that idle threads are never destroyed to allow for thread reuse without the overhead for new thread creation. The min, max and queue values can be left at the default as specified here.

Redundancy and load balancingMore than one OC4J instance can be started to accommodate the higher volume of concurrent requests that the container may need to handle. This is set through the "numProcs" parameter in the opmn.xml file and this parameter takes the value of 1 by default, to start a single OC4J instance. For multiple instances, the "numProcs" parameter can be adjusted to different values ( 2 for two instances, and so on ) and PM needs to be restarted with this value for the modules under its control. Very often, the applications that are being run may be process or memory intensive and may require one to adjust the value of the "numProcs" parameter to effect load-balancing via multiple instances.

DMZ setup on Oracle EBS R12

EXTERNAL SERVER SETUP

Environment Name:

CRP3

Machine:

Database/Conc/Admin Node: minez

Internal web/Form Node: minej

External Web Node: mine8

Pre-Clone Steps on Internal Serverq Run adpreclone.pl on MINEJand MINEZ with applcrp3 and oracrp3 users.

As oracrp3 user:

cd $ORACLE_HOME/appsutil/scripts/CRP3_MINEZ

perl adpreclone.pl dbTier

As applcrp3 user:

cd $ADMIN_SCRIPTS_HOME

perl adpreclone.pl appsTier

q Take a backup of /d21/applcrp/CRP3 folder on MINEJ

q Restore the Backup into MINE8 server under /d21 mount point.

q Setup Host File:

Put the following entries in the Hosts File (/etc/hosts)

10.218.17.36 minez.sonapglobal.com minez

10.221.15.167 minej.bn.sonap.net minej.bn

q Change the owner Ship of /d21/oracrp3 folder and the file under it to applcrp3

cd /d21

chown –R oracrp3:dba oracrp3

Creating External Web TierNote: Your steps should be more specific than the examples shown.

http://www.blogger.com/blogger.g?blogID=383029185363901069








q Create XML file for External Server:

su – applcrp3

cd /d21/applcrp3/CRP3/apps/apps_st/comn/clone/bin

perl adclonectx.pl contextfile=/d21/applcrp3/CRP3/inst/apps/CRP3_minej/appl/admin/CRP3_minej.xml

Enter the APPS password: sonapcrp3

Target System Hostname (virtual or normal) [mine8]:

Do you want the inputs to be validated (y/n) [n]? :

Target System Database SID: CRP3

Target System Database Server Node [mine8]: minez

Target System Base Directory: /d21/oracrp3/CRP3

Target System Forms ORACLE_HOME Directory [/d21/oracrp3/CRP3/apps/tech_st/10.1.2]:

Target System Web ORACLE_HOME Directory [/d21/oracrp3/CRP3/apps/tech_st/10.1.3]:

Target System APPL_TOP Mountpoint [/d21/oracrp3/CRP3/apps/apps_st/appl]:

Target System COMMON_TOP Directory [/d21/oracrp3/CRP3/apps/apps_st/comn]:

Target System Instance Home Directory [/d21/oracrp3/CRP3/inst]:

Username for the Applications File System Owner [applcrp3]:

Group for the Applications File System Owner [dba]:

Target System Root Service [enabled]:

Target System Web Entry Point Services [enabled]:

Target System Web Application Services [enabled]:

Target System Batch Processing Services [disabled]:

Target System Other Services [enabled]:

Do you want to preserve the Display [minej:0.0] (y/n)? : n

Target System Display [mine8:0.0]:

Do you want the the target system to have the same port values as the source system (y/n) [y]? : n

Target System Port Pool [0-99]: 1


Choose a value which will be set as APPLPTMP value on the target node [1]: 2

New context path and file name [/d21/oracrp3/CRP3/inst/apps/CRP3_mine8/appl/admin/CRP3_mine8.xml]:

Cross Check if the Context File generated is correct or not, check with following command if the respective components are enabled on MINE8:

grep –i status $CONTEXT_FILE

q Run adpreclone.pl to add the MINE8 server as a node to CRP3:

su - applcrp3

cd /d21/applcrp3/CRP3/apps/apps_st/comn/clone/bin

perl adcfgclone.pl appsTier /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/appl/admin/CRP3_mine8.xml

Enter the APPS password:

Check the logfile for any error.

q CONTEXT_FILE configuration:

Modify the following CONTEXT_FILE parameters:

Context File Variable

Existing Value New Value

s_applcsf

/d21/applcrp3/CRP3/inst/apps/CRP3_mine8/logs/appl/conc

/d21/applcrp3/CRP3/conc

s_appltmp

/d21/applcrp3/CRP3/inst/apps/CRP3_mine8/temp

/crp3_appltmp

s_applptmp

/d21/applcrp3/CRP3/inst/apps/CRP3_mine8/ptemp

/crp3_applptmp

s_formshost

mine8 crp3ext

s_chronosURL

http://mine8.sonapglobal.com:8001/oracle_smp_chronos/oracle_smp_chronos_sdk.gif

http://crp3ext.sonapglobal.com:8001/oracle_smp_chronos/oracle_smp_chronos_sdk.gif

s_external_url

http://mine8.sonapglobal.com:8001 http://crp3ext.sonapglobal.com:8001

s_webentryhost

mine8 crp3ext



s_login_page

http://mine8.sonapglobal.com:8001/OA_HTML/AppsLogin

http://crp3ext.sonapglobal.com:8001/OA_HTML/AppsLogin

q Run AutoConfig on al l the Nodes.

Configuring MINE8 for DMZq

Run the script txkChangeProfH.sql for the Profile option setup:

# sonapssh minez

$ su – applcrp3

$ sqlplus appUpdate Hierarchy Types/sonapcrp3 @$FND_TOP/patch/115/sql/txkChangeProfH.sql SERVRESP

Run AutoConfig on all nodes.

q Update Node Trust Level

To change the value of the Node Trust Level profile option value to External for a particular node, perform the following steps:

1. Login to Oracle E-Bsonapness Suite as sysadmin user sonapng the internal URL

2. Select the System Administrator Responsibility

3. Select Profile / System

4. From the 'Find system profile option Values' window, select the server and get the valie MINE8 into it.

5. Query for %NODE%TRUST%. You will see a profile option named 'Node Trust Level'. The value for this profile option at the site level will be Normal. Leave this setting unchanged.

6. Set the value of this profile option to External at the server level. The site level value should remain set to Normal

q Update List of Responsibility

To change the value of the Responsibility Trust Level profile option at the responsibility level for a particular responsibility, perform the following steps:

7. Login to Oracle E-Bsonapness Suite as sysadmin user sonapng the internal URL

8. Select System Administrator Responsibility

9. Select Profile / System







10. From the 'Find system profile option Values' window, select the responsibility that you want to make available to users logging in via the external web tier

11. Query for %RESP%TRUST%. You will see a profile option named 'Responsibility trust level'. The value for this profile option at site level will be Normal. Leave this setting unchanged.

12. Set the value of this profile option for the chosen responsibility to External at the responsibility level. The site-level value should remain Normal.

13. Repeat for all responsibilities that you want to make available from the external web tier.

List of Responsibilities which can be enabled on External Server is as followed:

Product Name Externally Accessible Responsibilites Additional Profile Options

iSupplier

POS Supplier Guest User

Plan to Pay Supplier View

Plan, Source, Pay Supplier View

Source to Pay Supplier View

Supplier Profile Manager

Procure to Pay Supplier View

POS: External URL

POS: Internal URL

Oracle Sourcing Sourcing Supplier PON: External Applications Framework Agent

PON: External login URL

Oracle iProcurement

Self Registered Employee Default Responsibility

Self Registered New User Default Responsibility

q Enable Oracle E-Business Suite Application Server Security

1. Set the value of Application Server Security Authentication (s_appserverid_authentication) to SECURE, in the CONTEXT_FILE on all the nodes.

2. Run AutoConfig on each Applications middle tier to complete the configuration.


3. After AutoConfig completes successfully, restart Oracle HTTP Server and OC4J processe

q Increase JVM Size

Change the following JVM parameter in the CONTEXT_FILE as mentioned in the below table:

Note: Take a backup of Context File before Changing.

Variable Exisiting Value New Value

s_oacore_jvm_start_options -server -verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=160M

-server -verbose:gc –Xmx1024M –Xms521M -XX:MaxPermSize=256M

q Run Autoconfig

Run AutoConfig from ADMIN_SCRIPTS_HOME sonapng adautocfg.sh.

Enable SSL LoginNote:Include a subset of test steps that will confirm that the customization has been installed properly.

REFERENCE

Metalink Document: “Oracle E-Bsonapness Suite R12 Configuration in a DMZ” Document ID: 380490.1

Oracle EBS R12 with servlet forms mode and more

Forms Servlet mode in Oracle EBS R12:

By default, Oracle EBS R12 has forms 10g Stack in servlet mode.Now the question is how is the client server connections handled?In Servlet mode we have a java servlet called the forms listener servlet which is responsible for the communication between the Forms java client and the oracle application serverforms services.

Unlike Socket mode, the Forms Servlet mode doesn’t require an additional forms port to handle communication between client and the application server. The forms servlet architecture operates through the http server port. All forms connections happen through the http port. Hence desktop clients can’t access forms directly in servlet mode.





The servlet architecture is designed to work with industry standards, and also to be fully supported for advanced network configurations with no additional ports or SSL configuration.

In Oracle Applications R12 customers who want load balancing must use the servlet implementation in which forms servers can be balanced across multiple application servers by using a hardware load balancer along with the 10g Application Server.

The way to find if the servlet mode/socket mode is implemented:

Check the forms configuration file : appsweb.cfg pointed by environment variable"FORMS_WEB_CONFIG_FILE".

The entries: serverURL= connectMode=Would be as mentioned below:

In servlet mode In Socket mode

serverURL=/forms/lservlet connectMode=servlet

serverURL=(should be blank) connectMode=Socket

This can also be verified in the context file in 11i and R12:

context variable value for Servlet value for Socket

s_forms_servlet_serverurl 11i: /forms/formservletr12: /forms/lservlet

(no value, blank)

s_forms_servlet_comment (no value, blank) #

s_frmConnectMode r12: servlet r12: socket

We can change the forms servlet mode to socket and how we do it is:

1. Source the environment on the application tier.2. Stop all the application tier services.3. Execute :$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode \-contextfile=$INST_TOP/appl/admin/mycontext.xml \-mode=socket \-port=9095 \-runautoconfig=No

Note: context file is the full path of the context file of the instance. Mode is the mode to which the form has to be changed. Port number used to run Forms in socket mode. The default port number is 9095.A port number is not needed if servlet mode is used. Recommended value for autoconfig is NO.

4. Run autoconfig on apps tier.5. Start all the application tier services.

6. Check whether Forms Server is running:$INST_TOP/admin/scripts/adformsrvctl.sh status7. Log in to Oracle Applications and launch a Forms-based application.8. Open the Sun Java Console (from Tools Menu in Internet Explorer).9. Check whether the "mode" directive displayed in Sun Java Console when launching forms-based applications is set to socket.10. The direct launch URL for Forms Socket Mode is:<web_protocol>://<web_host>.<web_domain>:<web_port>/OA_HTML/frmservlet

Note: If we need to change the forms mode from socket to servlet mode then all steps remain same except for the script which we run and the verification step:

1. $FND_TOP/bin/txkrun.pl -script=ChangeFormsMode \-contextfile=$INST_TOP/appl/admin/mycontext.xml \-mode=servlet \-runautoconfig=No No Port number is required as it would use the http port.

2. Verification of servlet mode should reflect:

The "mode" directive displayed in Sun Java Console when launching forms- based applications, is set to http,nativeDirect Forms Servlet Launch is:web_protocol>://<web_host>.<web_domain>:<web_port>/forms/frmservlet

How to Find version

1. How to find Apps Version (11i/R12/12i) >> Connect to database as user apps

SQL> select release_name from apps.fnd_product_groups;Output would be like 12.0.4 or 11.5.10.2

2. Web Server/Apache or Application Server in Apps 11i/R12

>> Log in as Application user, set environment variable and run below:

$IAS_ORACLE_HOME/Apache/Apache/bin/httpd –version

Output for 11i should be like :

Server version: Oracle HTTP Server Powered by Apache/1.3.19 (Unix)Server built: Jan 26 2005 11:06:44 (iAS 1.0.2.2.2 rollup 5)

Output for R12 should be like :

Server version: Oracle-Application-Server-10g/10.1.3.0.0Oracle-HTTP-ServerServer built: Dec 4 2006 14:44:38

3. Forms & Report version (aka developer 6i) in 11i

>> Log in as Application user, set environment variable and run below query:

$ORACLE_HOME/bin/f60run | grep Version | grep Forms

Output would be like :Forms 6.0 (Forms Runtime) Version 6.0.8.25.2 (Production)Check fourth character in version 25 which means Forms 6i patchset 16 (25-9)

4. Forms & Report version in R12/12i


$ORACLE_HOME/bin/rwrun | grep Release

Output should be like:

Report Builder: Release 10.1.2.2.0You can safely ignore warnings

5. Oracle Jinitiator in 11i/R12/12i


grep jinit_ver_comma $CONTEXT_FILE

(Default is Java Plug-In for R12/12i )

6. Oracle Java Plug-in in 11i/R12/12i

>>Log in as Application user, set environment variable and run below query:

grep plugin $CONTEXT_FILE

7. File Version on file system

>>adident Header <filename>

Or

strings <file_name> | grep Header

Here adident is AD Utility (Oracle Apps) and strings is Unix utility

8. Version of pld file

>> *.pld are source code of *.pll which are inturn source of *.plx. *.pll is in $AU_TOP/resource and to find its version check

adident Header $AU_TOP/resource/<filename>.pll

IGSAU012.pll:$Header IGSAU012.pld 115.1.115100.1 2004/04/01 05:40:18 appldev ship $

Or

strings $AU_TOP/resource/<filename>.pll | grep -i header

FDRCSID(’$Header: IGSAU012.pld 115.1.115100.1 2004/04/01 05:40:18 appldev ship $’);

9. OA Framework Version

>> http:// hostname.domainName:port/OA_HTML/OAInfo.jsp (Only for 11i); Log in as Application user, set environment variable and run below query:

adident Header $FND_TOP/html/OA.jspadident Header $OA_HTML/OA.jsp

output for both should look like$Header OA.jsp 115.60 2006/03/31 00:47:28 atgops1 noship $

120.21 means OA Framework Version (coming soon..)115.60 means OA Framework Version (coming soon..)115.56 means OA Framework Version (coming soon..)115.36 means OA Framework Version 5.7115.27 means OA Framework Version 5.6E115.26 means OA Framework Version 5.5.2E

10. Discoverer Version for 11i (3i or 4i)


$ORACLE_HOME/bin/disc4ws | grep -i Version

Discoverer Version for 11i or R12 (10g AS) >> Check under Application Server Section as 10g AS Discoverer is on standalone.

11. Workflow Version with Apps

>> Connect to Database as apps user

SQL> select TEXT Version from WF_RESOURCES where NAME = ‘WF_VERSION’;

Output like 2.6.0 means workflow version 2.6.0

12. Oracle Single Sign On

>> Connect to database which holds SSO repository:

SQL>select version from orasso.wwc_version$;

13. Oracle Internet Directory

>> There are two components in OID (Software/binaries & Schema/database)

>>> To find software/binary version

$ORACLE_HOME/bin/oidldapd –version

output should look like

oidldapd: Release 10.1.4.0.1 - Production on thu sep 11 11:08:12 2008Copyright (c) 1982, 2006 Oracle. All rights reserved.

>>> To find Schema Version/ database use

ldapsearch -h <hostname> -p <port> -D “cn=orcladmin” -w “<password>” -b “” \-s base “objectclass=*” orcldirectoryversion

and output should be like

version: 1dn:orcldirectoryversion: OID 10.1.4.0.1

or

Run following query in database

SQL> select attrval from ods.ds_attrstore where entryid = 1 and attrname = ‘orcldirectoryversion’;

Output should be like OID 10.1.4.0.1

14. Application Server

>> Oracle Application Server (Prior to Oracle WebLogic Server)

If application server is registered in database (Portal, Discoverer) check from database:

SQL> select * from ias_versions;

Or

SQL>select * from INTERNET_APPSERVER_REGISTRY.SCHEMA_VERSIONS;

AOC4J (Oracle Container for J2EE)Set ORACLE_HOMEcd $ORACLE_HOME/j2ee/homejava -jar oc4j.jar -version

15. Oracle Portal

>> SQL> select version from portal.wwc_version$;

16. Database Component

I) Oracle Database

To find database version:

SQL> select * from v$version;

Or

All component version in database:

$ORACLE_HOME/OPatch/opatch lsinventory -detail

17. Oracle Enterprise Manager

>> Metalink Note 605398.1: How to find the version of the main EM components

18. Unix Operating System

Solaris -> cat /etc/releaseRed Hat Linux -> cat /etc/redhat-release

Documents

ebs