1.Example of Using Oracle Clusterware Commands to Create Application ResourcesExample of Using Oracle Clusterware Commands to Create Application Resources The example in this section creates an application named postman. The Oracle Clusterware uses the script /opt/email/bin/crs_postman to start, stop, and monitor whether the application is running (action_script). The Oracle Clusterware checks postman every five seconds as specified by the setting for the check_interval parameter. The Oracle Clusterware restarts postman no more than once if it fails. When deciding on which node to place the postman application, Oracle Clusterware considers the value for the optional_resources parameter. If possible, Oracle Clusterware places postman on the same node. Finally, in order for postman to run, the resource network1 must be running on the same node as specified by the setting for the required_resources parameter. If network1 fails or if it is relocated to another node, then Oracle Clusterware stops or moves the postman application. Do not use the Oracle Clusterware commands crs_ Note: register, crs_profile, crs_start or crs_stop on resources with names beginning with the prefix ora unless either Oracle Support asks you to, or unless Oracle has certified you as described in http://metalink.oracle.com. Server Control (SRVCTL) is the correct utility to use on Oracle resources. You can create resources that depend on resources that Oracle has defined. You can also use the Oracle Clusterware commands to inspect the configuration and status. Using crs_profile to Create An Application Resource Profile The following example uses the crs_profile command to create an application profile for monitoring email. # crs_profile -create postman -t application -B /opt/email/bin/crs_postman-d quot;Email Applicationquot; -r network1 -l application2-a postman.scr -o ci=5,ft=2,fi=12,ra=2The contents of the application profile file that the example creates are as follows: NAME=postman TYPE=application ACTION_SCRIPT=/oracle/crs/script/postman.scr ACTIVE_PLACEMENT=0 AUTO_START=0 CHECK_INTERVAL=5 DESCRIPTION=email app FAILOVER_DELAY=0 FAILURE_INTERVAL=12 FAILURE_THRESHOLD=2 HOSTING_MEMBERS= OPTIONAL_RESOURCES=application2 PLACEMENT=balanced REQUIRED_RESOURCES=network1 RESTART_ATTEMPTS=2 SCRIPT_TIMEOUT=60 See Also: Appendix B, quot;High Availability Oracle Clusterware Command-Line Reference and C APIquot; for an explanation of the command optionsMaking Applications Highly Available Using Oracle Clusterware 14-7

2. Example of Using Oracle Clusterware Commands to Create Application ResourcesThe Oracle Clusterware Required Resources ListThe Oracle Clusterware uses the required resources list, with the placement policy andhosting nodes list, to determine the cluster nodes that are eligible to host anapplication. Required resources must be ONLINE on the nodes on which theapplication is running or started.The failure of a required resource on a hosting node causes Oracle Clusterware toattempt to restart the application on the current node. If RESTART_ATTEMPTS is notset to 0, and if the application cannot start on the current node, then OracleClusterware attempts to fail the application over to another node that provides therequired resource. Alternatively, Oracle Clusterware stops the application if there isno suitable node. In this case, Oracle Clusterware posts a not restarting eventnotification.You can also use required resource lists to start, stop, and relocate groups ofinterdependent applications when you use the crs_start, crs_stop, or crs_relocate commands with the force (-f) option. In other words, you can configure aset of resources to have other required resources. For example you can configureresources A, B, and C where A and B depend on C. Then if you stop resource C with-force option, this action will stop all three resources. The same is true for the crs_relocate command.In addition, using the -force option can relocate an online dependency, if necessary,to enable the starting of a resource. For instance, assume that resource VIP_A that hasa primary node assignment of node A is running instead on node B. If you perform acrs_start on the instance resource for node A, then this operation will fail. This isbecause the VIP_A resource is required for the instance on node A and the VIP_Aresource is online. However, the resource is on a node where the resource cannot run.Performing a crs_start -f on instance B, however, forces the VIP_A resource torelocate and then start the instance. Creating VIPs for ApplicationsIf your application is accessed by way of a network, then Oracle recommends that youcreate a virtual internet protocol address for the application as a dependent resource.In the previous example, network1 is an application VIP address. Create applicationVIP addresses as follows:crs_profile create network1 -t application -a $CRS_HOME/bin/usrvip -o oi=eth0,ov=138.3.83.78,on=255.255.240.0 In this example, $CRS_HOME is the home directory for the Oracle Clusterwareinstallation. In addition, eth0 is the name of the public network adapter,138.3.83.78 which resolves by way of DNS to a new IP address that will be used tolocate your application regardless of the node it is running on. Finally,255.255.240.0 is the netmask for the public IP address. As the oracle user,register the VIP address with Oracle Clusterware as follows:crs_register network1 On UNIX-based operating systems, the application VIP address script must run as theroot user. As the root user, change the owner of the resource as follows:crs_setperm network1 o root As the root user, enable the oracle user to run this script:crs_setperm network1 u user:oracle:r-x14-8 Oracle Clusterware and RAC Administration and Deployment Guide 3. Example of Using Oracle Clusterware Commands to Create Application ResourcesAs the oracle user, start the VIP address as follows: crs_start network1Application Placement Policies The placement policy specifies how Oracle Clusterware selects a node on which to start an application and where to relocate the application after a node failure. Only cluster nodes on which all of the required resources are available, as listed in an application's profile, are eligible to be considered as hosting nodes for an application. The Oracle Clusterware supports the following placement policies: balancedThe Oracle Clusterware favors starting or restarting the application on the node that is currently running the fewest resources. A placement that is based on optional resources is considered first. Next, the host with the fewest resources running is chosen. If no node is favored by these criteria, then any available node is chosen. favoredThe Oracle Clusterware refers to the list of nodes in the HOSTING_ MEMBERS attribute of the application profile. Only cluster nodes that are in this list and that satisfy the resource requirements are eligible for placement consideration. Placement due to optional resources is considered first. If no node is eligible based on optional resources, then the order of the hosting nodes determines which node runs the application. If none of the nodes in the hosting node list are available, then Oracle Clusterware places the application on any available node. This node may or may not be included in the HOSTING_MEMBERS list. restrictedSimilar to favored except that if none of the nodes on the hosting list are available, then Oracle Clusterware does not start or restart the application. A restricted placement policy ensures that the application never runs on a node that is not on the list, even if you manually relocate it to that node. You must specify hosting nodes in the HOSTING_MEMBERS attribute to use a favored or restricted placement policy. Do not specify hosting nodes in the HOSTING_ MEMBERS attribute with a balanced placement policy. Otherwise, the application will not validate and you cannot register it. If ACTIVE_PLACEMENT is set to 1, then the placement of the application is re-evaluated whenever you add a node to the cluster or if the cluster node restarts. This enables Oracle Clusterware to relocate applications to a preferred node after the node recovers from a failure. To have Oracle Clusterware relocate an application to a preferred node at a time other than when the node rejoins the cluster, use the REBALANCE attribute to specify a time at which placement can be re-evaluated. Optional Resources in Placement Decisions The Oracle Clusterware uses optional resources to choose a hosting node based on the number of optional resources that are in an ONLINE state on the hosting node. If each node has an equal number of optional resources in an ONLINE state, then Oracle Clusterware considers the order of the optional resources as follows: The Oracle Clusterware compares the state of the optional resources on each node starting at the first resource that you list in the application profile and then proceeds through the list. For each consecutive resource in your list, if the resource is ONLINE on one node, then any node that does not have the resource ONLINE is not considered.Making Applications Highly Available Using Oracle Clusterware 14-9 4. Oracle Clusterware Action Program Guidelines The Oracle Clusterware evaluates each resource in the list in this manner until only one node is available to host the resource. The maximum number of optional resources is 58. If this algorithm results in multiple preferred nodes, then the resource is placed on one of these nodes chosen according to its placement policy. Oracle Clusterware Action Program Guidelines This section provides the following guidelines for writing Oracle Clusterware action programs that interpret Oracle Clusterware start, stop, and check commands: The Oracle Clusterware relies on a status code upon exiting from an action program to set the resource state to ONLINE or OFFLINE. On Windows-based systems, the program should be non-blocking, which may imply a Windows service or a Windows resource that does not block during console interactions. Action programs must return a status code to indicate success or failure. For example, on UNIX-based systems if the script is written in Bourne Shell, then the program should issue exit(1) to indicate failure, and exit(0) to indicate success. Similarly, on Windows-based systems, the action program should return a status of (0) to indicate success and (1) to indicate failure. After application failure, Oracle Clusterware calls the action program with a check parameter. The action program replies to Oracle Clusterware with a status of (1) to indicate failure. After receiving this failure status, Oracle Clusterware calls the action program with a stop parameter. It is important that the action program returns a status of (0) to indicate a successful stop of the application, even though the application was not running when the stop request was called. The Oracle Clusterware sets the resource state to UNKNOWN if an action program's stop entry point fails to exit within the number of seconds in the SCRIPT_ TIMEOUT value, or if the action program returns with a code that indicates failure. This may occur during a start, relocation, or stop operation. Ensure that the action programs stop entry point exits with a value that indicates success if the resource is successfully stopped or if the resource is not running. When a daemon or service starts, it usually needs to start as a background process, depending on the platform. However, a resource started in this way always returns success during a start attempt. This means that the default scripts cannot detect failures caused by minor errors, such as misspelled command paths.On UNIX-based systems, if a resource does not move to the background immediately upon startup, then you can start the application in thebackground by adding an ampersand (&) to the end of the line that starts theapplication.On Windows-based systems, you can use net start to start a service that needs to start in the background.When using commands to start daemons or services in the background, interactively make test runs of the commands used in the script to eliminateerrors before using the script with Oracle Clusterware. How Oracle Clusterware Runs Action Programs This section describes how Oracle Clusterware runs action programs. The first argument to an action program is the command start, stop, or check depending on which action Oracle Clusterware is running. The second argument is the Oracle 14-10 Oracle Clusterware and RAC Administration and Deployment Guide 5. Oracle Clusterware Action Program Guidelines Clusterware resource name of the application. This enables a script to determine which instance of the resource that Oracle Clusterware is starting, stopping, or checking. An action program can retrieve any of its Oracle Clusterware resource attributes from the environment by using $_CAA_attribute_name. For example, $_CAA_NAME contains the application name, the second argument to the script, and $_CAA_ HOSTING_MEMBERS contains its HOSTING_MEMBERS attribute. User Defined Attributes The Oracle Clusterware supports user-defined attributes in Oracle Clusterware applications, which are attributes having names that contain USR. User-defined attributes are stored as part of the Oracle Clusterware application profile for the application. You can reference them in an action program using $_USR_ attributename. To add a user-defined attribute, add it to the file CRS home/crs/template/application.tdf using the following syntax: # # an example user-defined attribute # #!=========================== attribute: USR_EXAMPLE type: string switch: -o example default: required: noThe attribute parameter contains the name of the new attribute. The type parameter defines the type of the user-defined attribute and can be one of the following: string boolean integera numeric attribute positive_integera numeric attribute that must be positive name string name_lista comma-delimited list of names The switch parameter describes how the attribute is specified for the crs_profile command. Set the required field to no for user-defined attributes. Note: User-defined attribute names that begin with USR_ORA are reserved for use by Oracle. Windows crsuser Program This section describes the Windows-based crsuser program. If you create an Oracle Clusterware application, then run the following command: crsuser add [domain]usernameProvide the user's Windows password. This creates the service OracleCRSToken_ user that Oracle Clusterware needs for starting the Oracle Clusterware resources Making Applications Highly Available Using Oracle Clusterware14-11 6. Using Oracle Clusterware Commands under the given user ID. You can also use the crsuser command remove [domain]username to remove a token service and crsuser list to list a registered users. Using Oracle Clusterware Commands This section describes how to use the Oracle Clusterware commands under the following topics:Registering Application Resources Starting Application Resources Relocating Applications and Application Resources Stopping Applications and Application Resources Unregistering Applications and Application Resources Displaying Clusterware Application and Application Resource Status Information Registering Application Resources Each application that you manage with Oracle Clusterware must have an application profile and the profile must be registered in the OCR. Use the crs_register command to register applications in the OCR. For example, enter the following command to register the mail monitoring application from the previous example: # crs_register postman If you modify a profile, then update the OCR by re-running the crs_register -u command. Starting Application Resources To start an application resource that is registered with Oracle Clusterware, use the crs_start command. For example: # crs_start postmanThe following text is an example of the command output: Attempting to start `postman` on node `rac1` Start of `postman` on node `rac1` succeeded.The application now runs on the node named rac1. The name of the application resource may or may not be theNote:same as the name of the application.Appendix B, quot;High Availability Oracle ClusterwareSee Also :Command-Line Reference and C APIquot; for examples of OracleClusterware command outputThe command waits for the amount of time specified by the setting for the SCRIPT_ TIMEOUT parameter to receive a notification of success or failure from the action program each time the action program is called. Application resources can be started if 14-12 Oracle Clusterware and RAC Administration and Deployment Guide 7. Using Oracle Clusterware Commands they have stopped due to exceeding their failure threshold values. You must register a resource with crs_register before you can start it. To start and resources, use the crs_start and crs_stop commands. Manual starts or stops outside of Oracle Clusterware can invalidate the resource status. In addition, Oracle Clusterware may attempt to restart a resource on which you perform a manual stop operation. All required resources must be online on the node where you start the resource. If the resources that the REQUIRED_RESOURCES parameter identifies are offline, then the command crs_start resource_name will start the required resources before starting the resource. Running the crs_start command on a resource sets the resource target value to ONLINE. The Oracle Clusterware attempts to change the state to match the target by running the action program with the start parameter. When a resource is running, both the target state and current state are ONLINE.Starting an Application on an Unavailable Node When starting an application on a cluster node that is unavailable, crs_start can give indeterminate results. In this scenario, the start section of the action program is run, but the cluster node fails before notification of the start is displayed on the command line. The crs_start command returns a failure with the error Remote start for resource_name failed on node node_name. The application is actually ONLINE but fails over to another node making the application appear as though it were started on the incorrect node. If a cluster node fails while you are starting a resource on that node, then check the state of the resource on the cluster by using the crs_stat command to determine the state of that resource. Relocating Applications and Application Resources Use the crs_relocate command to relocate applications and application resources. For example, to relocate the mail monitoring application to the node known as rac2, enter the following command: # crs_relocate postman -c rac2Each time that the action program is called, the crs_relocate command waits for the duration identified by the value for the SCRIPT_TIMEOUT parameter to receive notification of success or failure from the action program. A relocation attempt fails if: The application has required resources that are ONLINE on the initial node Applications that require the specified resource are ONLINE on the initial node To relocate an application and its required resources, use the -f option with the crs_ relocate command. The Oracle Clusterware relocates or starts all resources that are required by the application regardless of their state. Stopping Applications and Application Resources To stop applications and application resources, use the crs_stop command. Immediately after the crs_stop command completes, the application status converts to OFFLINE. Because Oracle Clusterware always attempts to match a resource's state to its target, the Oracle Clusterware subsystem stops the application. The following example stops the mail application from the example: # crs_stop postman Making Applications Highly Available Using Oracle Clusterware 14-13 8. Using Oracle Clusterware CommandsThe following text is an example of the command output: Attempting to stop `postman` on node `rac1` Stop of `postman` on node `rac1` succeeded.You cannot stop an application if the application is a required resource for another online application unless you use the force (-f) option. If you use the crs_stop -f resource_name command on an application that is required by other resources and if those resources are online, then Oracle Clusterware stops the application. In addition, all of the resources that require that application that are online are also stopped.The Oracle Clusterware can only stop applications andNote:application resources. The Oracle Clusterware cannot stop network,tape, or media changer resources. Managing Automatic Oracle Clusterware Resource Operations for Action Scripts The following section explains additional information for controlling how Oracle Clusterware manages restarts. You can prevent Oracle Clusterware from automatically restarting a resource by setting several action program attributes. You can also control how Oracle Clusterware manages the restart counters for your action programs. In addition, you can customize the timeout values for the start, stop, and check actions that Oracle Clusterware performs on action scripts. These topics are described under the following headings:Preventing Automatic Database Instance Restarts Automatically Manage Restart Attempts Counter for Resources Implications of Restart and Timeout Features for Previous Releases Preventing Automatic Database Instance Restarts When a node stops and restarts, Oracle Clusterware starts the instance as soon as the node starts. This may not be desirable because instance startup might fail if system components on which the instance depends, such as a volume manager or a file system, are not running. This is especially true if Oracle Clusterware does not manage the system component resources on which the instance depends. To manage automatic restarts, you can use the AUTO_START attribute to specify whether Oracle Clusterware should automatically start a resource when a node restarts. Valid AUTO_START values are:alwaysCauses the resource to restart when the node restarts regardless of the resources state when the node stopped.restoreDoes not start the resource at restart time if it was in an offline state, such as STATE=OFFLINE, TARGET=OFFLINE, when the node stopped. Theresource is restored to its state when the node went down. The resource is startedonly if it was online before and not otherwise.neverThe Oracle Clusterware never restarts the resource regardless of the resources state when the node stopped.Oracle only supports lower-case values for always, restore,Note:and never.14-14 Oracle Clusterware and RAC Administration and Deployment Guide 9. Using Oracle Clusterware Commands Automatically Manage Restart Attempts Counter for Resources When a resource fails, Oracle Clusterware restarts the resource for only the number of times specified in the profile attribute RESTART_ATTEMPTS regardless of how often the resource fails. The CRSD process maintains an internal counter to track how often a resource has been restarted. There is a mechanism by which Oracle Clusterware can automatically manage the restart attempts counter based on the stability of a resource. Use the UPTIME_THESHOLD attribute to indicate resource stability. You can specify the time for the UPTIME_THRESHOLD attribute in different units of measure, such as seconds (s), minutes (m), hours (h), days (d) or weeks (w). Examples of valid values for this attribute are: 7d for seven days, 5h for five hours, 180m for 180 minutes, and so on. Specify the time period as a numeric value and the unit of measure as the last character, s, m, h, d, or w. After the time period that you have indicated by the setting for UPTIME_THRESHOLD has elapsed, Oracle Clusterware resets the value for RESTART_COUNTS to 0. The Oracle Clusterware can alert you when the value for RESTART_COUNT reaches the value that you have set for RESTART_ATTEMPTS. Note: Oracle Clusterware writes an alert to the CRSD log file when the value for RESTART_COUNT reaches the value that you have set for RESTART_ATTEMPTS. The Oracle Clusterware does not write this message to the alert file. RESTART_ATTEMPTS and RESTART_COUNT Failure Scenarios Some of the failure scenarios for the RESTART_ATTEMPTS and RESTART_COUNT attributes are: When a resource keeps restartingThe resource does not meet the uptime threshold criteria and it will be stopped after restarting for the number of attempts set by the value for RESTART_ATTEMPTS. When a node fails or restartsThe Oracle Clusterware resets the value for RESTART_COUNTER to 0 either when the resource relocates or when it restarts on the same node. If the crsd process failsBecause both RESTART_COUNT and RESTART_ ATTEMPTS are stored in OCR, the behavior is not affected.Implications of Restart and Timeout Features for Previous Releases If you have a pre-Oracle Database 10g release 2 (10.2) installation, add the release 2 (10.2) attributes to your profiles by doing one of the following: Modify your resources with the release 2 (10.2) attributes and re-register the resources. This causes the separate timeouts to be effective. Do not modify your resources. This retains the pre-release 2 (10.2) behavior of using the value in SCRIPT_TIMEOUT as the timeout for all start, stop, and check actions. Unregistering Applications and Application Resources To unregister an application or an application resource, use the crs_unregister command. You cannot unregister an application or resource that is ONLINE or required by another resource. The following example unregisters the mail application: # crs_unregister postman Making Applications Highly Available Using Oracle Clusterware 14-15 10. Using Oracle Clusterware Commands The unregistration process frees space on the OCR. Additionally, run the crs_ unregister command as a clean-up step when a resource is no longer managed by Oracle Clusterware. Generally, you should unregister all permanently stopped applications. Displaying Clusterware Application and Application Resource Status Information To display status information about applications and resources that are on cluster nodes, use the crs_stat command. The following example displays the status information for the postman application: # crs_stat postman NAME=postman TYPE=application TARGET=ONLINE STATE=ONLINE on rac2Enter the following command to view information about all applications and resources in tabular format: # crs_stat -tThe following text is an example of the command output: Name TypeTargetState Host ---------------------------------------------------------------- cluster_lockd application ONLINEONLINE rac2 dhcpapplication OFFLINE OFFLINEEnter the following command to determine:How many times an application resource has been restarted How many times an application resource has failed within the failure interval The maximum number of times that an application or resource can be restarted or failThe target state of the application or resource and the normal status information # crs_stat -vTo view verbose content in tabular format, enter the following command: # crs_stat -v -tThe following text is an example of the command output: NameType R/RA F/FTTargetStateHost ---------------------------------------------------------------------- cluster_lockd application 0/30 0/0ONLINEONLINE rac2 dhcpapplication 0/1 0/0 OFFLINE OFFLINE named application 0/1 0/0 OFFLINE OFFLINE network1application 0/1 0/0 ONLINEONLINE rac1Enter the following command to view the application profile information that is stored in the OCR: # crs_stat -pThe following text is an example of the command output: NAME=cluster_lockd TYPE=application 14-16 Oracle Clusterware and RAC Administration and Deployment Guide 11. Using Oracle Clusterware Commands ACTION_SCRIPT=cluster_lockd.scr ACTIVE_PLACEMENT=0 AUTO_START=1 CHECK_INTERVAL=5 DESCRIPTION=Cluster lockd/statd FAILOVER_DELAY=30 FAILURE_INTERVAL=60 FAILURE_THRESHOLD=1 REBALANCE= HOSTING_MEMBERS= OPTIONAL_RESOURCES= PLACEMENT=balanced REQUIRED_RESOURCES= RESTART_ATTEMPTS=2 SCRIPT_TIMEOUT=60 ...Refer to the crs_stat command for more information. See Also: Appendix B, quot;High Availability Oracle ClusterwareCommand-Line Reference and C APIquot; for detailed informationabout the Oracle Clusterware commandsMaking Applications Highly Available Using Oracle Clusterware 14-17 12. Using Oracle Clusterware Commands 14-18 Oracle Clusterware and RAC Administration and Deployment Guide 13. 15Application-Specific Deployment TopicsThis chapter discusses topics for deploying online (OLTP), data warehouse, and general purpose (hybrid) applications in Oracle Real Application Clusters (RAC) environments. The topics in this chapter are: General Deployment Strategies for Real Application Clusters-Based Applications Deploying OLTP Applications in Real Application Clusters Deploying Data Warehouse Applications with Real Application Clusters Data Security Considerations in Real Application Clusters General Deployment Strategies for Real Application Clusters-Based Applications All single-instance application development and deployment techniques apply to RAC. If your applications run well on a single-instance Oracle database, then they will run well on a RAC database. Deploying OLTP Applications in Real Application Clusters Cache Fusion makes RAC databases the optimal deployment servers for online transaction processing (OLTP) applications. This is because these types of applications require: High availability in the event of failures Scalability to accommodate increased system demands Load balancing according to demand fluctuations The high availability features of Oracle and RAC can re-distribute and load balance workloads to surviving instances without interrupting processing. RAC also provides excellent scalability so that if you add or replace a node, then Oracle re-masters resources and re-distributes processing loads. Flexible Implementation with Cache Fusion To accommodate the frequently changing workloads of online transaction processing systems, RAC remains flexible and dynamic despite changes in system load and system availability. RAC addresses a wide range of service levels that, for example, fluctuate due to: Varying user demands Peak scalability issues like trading storms (bursts of high volumes of transactions) Application-Specific Deployment Topics 15-1 14. Deploying Data Warehouse Applications with Real Application ClustersVarying availability of system resources Deploying Data Warehouse Applications with Real Application Clusters This section discusses how to deploy data warehouse systems in RAC environments by briefly describing the data warehouse features available in shared disk architectures. The topics in this section are:Speed-Up for Data Warehouse Applications on Real Application Clusters Parallel Execution in Data Warehouse Systems and RAC Using Parallel Instance Groups Speed-Up for Data Warehouse Applications on Real Application Clusters RAC is ideal for data warehouse applications because it augments the single instance benefits of Oracle. RAC does this by maximizing the processing available on all of the nodes that belong to a RAC database to provide speed-up for data warehouse systems. The query optimizer considers parallel execution when determining the optimal execution plans. The default cost model for the query optimizer is CPU+I/O and the cost unit is time. In RAC, the query optimizer dynamically computes intelligent defaults for parallelism based on the number of processors in the nodes of the cluster. An evaluation of the costs of alternative access paths, table scans versus indexed access, for example, takes into account the degree of parallelism (DOP) available for the operation. This results in Oracle selecting the execution plans that are optimized for your RAC configuration. Parallel Execution in Data Warehouse Systems and RAC Oracle's parallel execution feature uses multiple processes to run SQL statements on one or more CPUs. Parallel execution is available on both single-instance Oracle databases and RAC databases. RAC takes full advantage of parallel execution by distributing parallel processing across all available instances. The number of processes that can participate in parallel operations depends on the DOP assigned to each table or index.See Also: Oracle Database Performance Tuning Guide for more information about the query optimizerUsing Parallel Instance Groups You can control the instances that allocate parallel execution server processes with instance groups. To do this, assign each active instance to at least one or more instance groups. Then dynamically control which instances spawn parallel processes by activating a particular group of instances. An instance can belong to one or more groups. You can Note: enter several instance group names with the INSTANCE_GROUPS parameter using a comma as a separator. Data Security Considerations in Real Application Clusters This section describes the following two RAC security considerations:15-2 Oracle Clusterware and RAC Administration and Deployment Guide 15. Data Security Considerations in Real Application Clusters Transparent Data Encryption and Wallets Windows Firewall Considerations Transparent Data Encryption and Wallets Wallets used by RAC instances for Transparent Database Encryption may be a local copy of a common wallet shared by multiple nodes or a shared copy residing on a network file system that all of the nodes can access. A deployment with a single wallet on a shared disk requires no additional configuration to use Transparent Data Encryption. Deployments where no shared storage exists require that each RAC node maintain its own local wallet. Details about creating and provisioning a wallet can be found in the Database Security Guide. After you create and provision a wallet a single node, you must copy the wallet and make it available to all of the other nodes. For systems using Transparent Data Encryption with encrypted wallets, you can use any standard file transport protocol. For systems using Transparent Data Encryption with obfuscated wallets, file transport through a secured channel is recommended. The wallet must reside in the directory specified by the setting for the WALLET_LOCATION or ENCRYPTION_WALLET_ LOCATION parameter in sqlnet.ora. The local copies of the wallet need not be synchronized for the duration of Transparent Data Encryption usage until the server key is re-keyed though the ALTER SYSTEM SET KEY SQL statement. Each time you run the ALTER SYSTEM SET KEY statement at a database instance, you must again copy the wallet residing on that node and make it available to all of the other nodes. To avoid unnecessary administrative overhead, reserve re-keying for exceptional cases where you are certain that the server master key is compromised and that not re-keying it would cause a serious security problem. Windows Firewall Considerations By default, all installations of Windows Server 2003 Service Pack 1 and higher enable the Windows Firewall to block virtually all TCP network ports to incoming connections. As a result, any Oracle products that listen for incoming connections on a TCP port will not receive any of those connection requests, and the clients making those connections will report errors. Depending upon which Oracle products you install and how they are used, you may need to perform additional Windows post-installation configuration tasks so that the Firewall products are functional on Windows Server 2003. This section contains these topics: Oracle Executables Requiring Firewall Exceptions Configuring the Windows Firewall Troubleshooting Windows Firewall Exceptions Oracle Executables Requiring Firewall Exceptions Table 151 lists the Oracle Database 10g Release 1 (10.1) or later executables that listen on TCP ports on Windows. If they are in use and accepting connections from a remote client computer, then Oracle recommends that you add them to the Windows Firewall exceptions list to ensure correct operation. Except as noted, they can be found in ORACLE_HOMEbin. The RMI registry application and daemon executable listed in Table 151 are used by Oracle Ultra Search to launch a remote crawler. They must be added to the Windows Application-Specific Deployment Topics 15-3 16. Data Security Considerations in Real Application ClustersFirewall exception list if you are using the Ultra Search remote crawler feature, and ifthe remote crawler is running on a computer with the Windows Firewall enabled.If multiple Oracle homes are in use, then several firewall Note: exceptions may be needed for the same executable: one for each home from which that executable loads.Table 151Oracle Executables Requiring Windows Firewall Exceptions File Name Executable Name OracleCRService CRS_HOMEbincrsd.exe OracleEVMService CRS_HOMEbinevmd.exe Event manager logging daemon CRS_HOMEbinevmlogger.exe Oracle Object Link Manager (GUI version) CRS_HOMEbinGuiOracleObjManager.exe OracleCSService CRS_HOMEbinocssd.exe Oracle Object Link Manager (CLI version) CRS_HOMEbinOracleObjManager.exe Virtual Internet Protocol Configuration Assistant CRS_HOMEbinracgvip.exe Oracle Cluster Volume Service CRS_HOMEcfsOcfsfindvol.exe Data Guard Manager dgmgrl.exe Oracle Database Control emagent.exe External Procedures extproc.exe Generic Connectivity hdodbc.exe Oracle Internet Directory LDAP Server oidldapd.exe Oracle Services for Microsoft Transaction Server omtsreco.exe Oracle Database oracle.exe Apache Web Server ORACLE_HOMEapacheapacheapache.exe Enterprise Manager Agent ORACLE_HOMEbinemagent.exe Oracle services for Microsoft Transaction Server ORACLE_HOMEbinomtsreco.exe Oracle ORACLE_HOMEbinORACLE.EXE RACG ORACLE_HOMEbinracgimon.exe Oracle Listener ORACLE_HOMEbinTNSLSNR.exe Java Virtual Machine ORACLE_HOMEjdkbinjava.exe RMI daemon executable ORACLE_HOMEjdkbinrmid.exe RMI registry application ORACLE_HOMEjdkbinrmiregistry.exe ORACLE_HOMEjdkjrebinrmiregistry.exe RMI registry application Oracle Notification Service ORACLE_HOMEopmnbinons.exe Oracle Process Manager ORACLE_HOMEopmnbinopmn.exe Oracle Procedural Gateway for APPC pg4arv.exe Oracle Procedural Gateway for Websphere MQ pg4mqc.exe Oracle Procedural Gateway for Websphere MQ pg4mqs.exe Oracle Procedural Gateway for APPC pg4t4ic.exe15-4 Oracle Clusterware and RAC Administration and Deployment Guide 17. Data Security Considerations in Real Application ClustersTable 151 (Cont.) Oracle Executables Requiring Windows Firewall Exceptions File Name Executable Name Oracle Transparent Gateway for DRDA tg4drsrv.exe Oracle Transparent Gateway for MS-SQL Server tg4msql.exe Oracle Transparent Gateway for SYBASE tg4sybs.exe Oracle Transparent Gateway for Teradata tg4tera.exe Oracle TNS Listener tnslsnr.exe System file for Oracle Cluster File System WINDOWS_HOMEsystem32driversOcfs.sysConfiguring the Windows FirewallPost-installation configuration for the Windows Firewall must be undertaken if all ofthe following conditions are met: Oracle server-side components are installed. These components include the Oracle Database, network listeners, and any Web servers or services. The computer services connections from other computers over a network. If no other computers connect to the computer with the Oracle software, then no post-installation configuration steps are required and the Oracle software will function as expected. The Windows Firewall is enabled. If the Windows Firewall is not enabled, then no post-installation configuration steps are required.You can configure Windows Firewall by opening specific static TCP ports in thefirewall or by creating exceptions for specific executables so that they are able toreceive connection requests on any ports they choose. To configure the firewall, chooseControl Panel > Windows Firewall > Exceptions or enter netsh firewalladd... at the command line.Alternatively, Windows will inform you if a foreground application is attempting tolisten on a port, and it will ask you if you wish to create an exception for thatexecutable. If you choose to do so, then the effect is the same as creating an exceptionfor the executable either in the Control Panel or from the command line. Troubleshooting Windows Firewall ExceptionsIf you cannot establish certain connections even after granting exceptions to theexecutables listed in Table 151, then follow these steps to troubleshoot theinstallation: Examine Oracle configuration files (such as *.conf files), the Oracle key in the1. Windows registry, and network configuration files in ORACLE_ HOMEnetworkadmin. Pay particular attention to any executable listed in ORACLE_2. HOMEnetworkadminlistener.ora in a PROGRAM= clause. Each of these must be granted an exception in the Windows Firewall, because a connection can be made through the TNS Listener to that executable. Examine Oracle trace files, log files, and other sources of diagnostic information3. for details on failed connection attempts. Log and trace files on the database client Application-Specific Deployment Topics 15-5 18. Data Security Considerations in Real Application Clusters computer may contain useful error codes or troubleshooting information for failed connection attempts. The Windows Firewall log file on the server may contain useful information as well. If the preceding troubleshooting steps do not resolve a specific configuration issue4. on Windows XP Service Pack 2, then provide the output from command netsh firewall show state verbose=enable to Oracle Support for diagnosis and problem resolution.See Also:http://www.microsoft.com/downloads/details.aspx?Fam ilyID=a7628646-131d-4617-bf68-f0532d8db131&displaylang=en for information on Windows Firewall troubleshootinghttp://support.microsoft.com/default.aspx?scid=kb;e n-us;875357 for more information on Windows Firewallconfiguration 15-6 Oracle Clusterware and RAC Administration and Deployment Guide 19. A TroubleshootingThis appendix explains how diagnose problems for Oracle Clusterware and Oracle Real Application Clusters (RAC) components. The appendix explains how to do this by using dynamic debugging, enabling tracing, as well as by using trace and log file information. This appendix also includes information about the Cluster Verification Utility (CVU). The topics in this appendix are: Overview of Troubleshooting Real Application Clusters Diagnosing Oracle Clusterware High Availability Components Diagnosing Oracle Real Application Clusters Components Using the Cluster Verification Utility Overview of Troubleshooting Real Application Clusters Both Oracle Clusterware and Oracle RDBMS components have sub-components that you can troubleshoot as described in this chapter. You can enable dynamic debugging to troubleshoot Oracle Clusterware processing. You can also enable debugging and tracing for specific components and specific Oracle Clusterware resources to focus your troubleshooting efforts. A special clusterware health check command enables you to determine the status of several Oracle Clusterware components at one time. The trace and log files that Oracle generates for RAC are available for both the Oracle Clusterware components and for the Oracle RDBMS with RAC components. For Oracle Clusterware, Oracle stores these under a unified directory log structure. In addition to debugging and tracing tools, you can use the Cluster Verification Utility (CVU) and instance-specific alert files for troubleshooting. Diagnosing Oracle Clusterware High Availability Components This section describes the diagnosability features for Oracle Clusterware which include log files for the Oracle Clusterware daemon, as well as log files for the Event Manager daemon (EVM) and the Oracle Notification Service (RACGONS) under the following topics: Dynamic Debugging Component Level Debugging Oracle Clusterware Shutdown and Startup Diagnostics Collection Script The Oracle Clusterware Alerts Troubleshooting A-1 20. Diagnosing Oracle Clusterware High Availability ComponentsResource Debugging Checking the Health of the Clusterware Clusterware Log Files and the Unified Log Directory Structure Troubleshooting the Oracle Cluster Registry Enabling Additional Tracing for Real Application Clusters High Availability Dynamic Debugging You can use crsctl commands as the root user to enable dynamic debugging for Oracle Clusterware, the Event Manager (EVM), and the clusterware subcomponents. You can also dynamically change debugging levels using crsctl commands. Debugging information remains in the Oracle Cluster Registry (OCR) for use during the next startup. You can also enable debugging for resources. The crsctl syntax to enable debugging for Oracle Clusterware is: crsctl debug log crs quot;CRSRTI:1,CRSCOMM:2quot;The crsctl syntax to enable debugging for EVM is: crsctl debug log evm quot;EVMCOMM:1quot;The crsctl syntax to enable debugging for resources is: crsctl debug log res quot;resname:1quot;Component Level Debugging You can enable debugging for the CRS daemons, EVM, and their modules by setting environment variables or by running crsctl commands as follows where module_ name is the name of the module, crs, evm, or css and debugging_level is a level from 1 to 5: crsctl debug log module_name component:debugging_levelRun the following command to obtain component names where module_name is the name of the module, crs, evm, or css: crsctl lsmodules module_nameYou do not have to be the root user to run the crsctlNote:command with the lsmodules option. Oracle Clusterware Shutdown and Startup You can use a crsctl command as follows to stop Oracle Clusterware and its related resources on a specific node: crsctl stop crsYou can use a crsctl command as follows to start Oracle Clusterware and its related resources on a specific node: crsctl start crs A-2 Oracle Clusterware and RAC Administration and Deployment Guide