ASM Using ISCSI on Sun Storage

Deploying Oracle® 10g or 11gR1 ASM Using iSCSI on Sun™ Storage 7000 Unified Storage Systems (2009.Q3)Sridhar Ranganathan, Jeffrey WrightDecember 2009

This article was published by BigAdmin at: http://www.sun.com/bigadmin/features/articles/7000_oracle_iscsi_asm_2009q3.jspTo keep track of the latest content published by BigAdmin, subscribe to the BigAdmin newsletter: http://www.sun.com/bigadmin/newsletter/.

http://www.sun.com/bigadmin/features/articles/7000_oracle_iscsi_asm_2009q3.jsp

http://www.sun.com/bigadmin/newsletter/

BigAdmin Portal for Sun System Administrators Sun Microsystems, Inc.

Copyright © 2009 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved.

U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc. standard license agreement and applicable provisions of the FAR and its supplements. Use is subject to license terms. This distribution may include materials developed by third parties.

Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd. X/Open is a registered trademark of X/Open Company, Ltd.

ORACLE is a registered trademark of Oracle Corporation.

Sun, Sun Microsystems, the Sun logo, Solaris, Sun BluePrints, and SunSolve are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries.

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.

This product is covered and controlled by U.S. Export Control laws and may be subject to the export or import laws in other countries. Nuclear, missile, chemical biological weapons or nuclear maritime end uses or end users, whether direct or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to entities identified on U.S. export exclusion lists, including, but not limited to, the denied persons and specially designated nationals lists is strictly prohibited.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

BigAdmin Portal for Sun System Administrators Sun Microsystems, Inc.

Table of Contents

1.0 Introduction................................................................................................................................................1

2.0 Tested Configurations...................................................................................................................................2

3.0 Configuring Sun Storage 7000 Unified Storage System......................................................................................3

3.1 Storage Configuration ...........................................................................................................................................3

3.2 Adding iSCSI Initiator Access .................................................................................................................................3

3.3 Creating a Project...................................................................................................................................................4

3.4 Creating Volumes...................................................................................................................................................4

4.0 Configuring Oracle ASM on the Solaris OS.......................................................................................................7

4.1 Discovering iSCSI LUNs...........................................................................................................................................7

4.2 Configuring Oracle ASM Instance and ASM Diskgroup............................................................................................9

4.3 Configuring the Oracle Environment....................................................................................................................10

5.0 Configuring Oracle ASM on Red Hat Linux 4 ..................................................................................................11

5.1 Discovering iSCSI LUNs.........................................................................................................................................11

5.2 Configuring ASMLib..............................................................................................................................................14

5.3 Configuring Oracle ASM Instance and ASM Diskgroup..........................................................................................16

5.4 Configuring the Oracle Environment....................................................................................................................17

6.0 Creating Oracle Database Using an ASM Diskgroup.........................................................................................18

7.0 Glossary....................................................................................................................................................20

8.0 Author-Recommended Resources..................................................................................................................21

9.0 For More Information..................................................................................................................................21

1.0 Introduction Sun Microsystems, Inc.

1.0 Introduction

The Internet SCSI (iSCSI) protocol allows a client system (initiator) to send SCSI commands to targets on a remote server over an existing IP network infrastructure, thus providing a low cost alternative to a Storage Area Network (SAN) infrastructure. The Sun™ Storage 7000 Unified Storage System supports the iSCSI, NFS, CIFS, FTP, and HTTP protocols and provides comprehensive data services that can provide unlimited snapshots, a cloning feature, rollback, and remote replication.

Automatic Storage Management (ASM) is a feature in Oracle® Database 10g and above that provides a simpler data management and a vertically integrated file system and volume manager that are specifically built for Oracle databases. The steps provided in this document are to be used with Oracle 10gR1, 10gR2, and 11gR1 versions only. Please refer to the Oracle documentation for details on different modes of deployment with ASM.

This quick start guide provides an overview of configuring the volumes in a Sun Storage 7000 Unified Storage System (2009.Q3) to be exported as iSCSI targets, then configuring those iSCSI logical unit numbers (LUNs) at the database server to be used for Oracle ASM.

The document is intended for Oracle database administrators, system administrators and anyone who would like to deploy Oracle ASM using the Sun Storage 7000 Unified Storage System. This is to be used as a general guideline only. For in-depth details, please refer to the Oracle documentation and to the Sun Storage 7000 Unified Storage System documentation.

The topics covered are:➢ Configuring volumes in the Sun Storage 7000 Unified Storage System➢ Configuring iSCSI LUNs in the Solaris™ 10 Operating System to deploy Oracle ASM ➢ Configuring iSCSI LUNs in the Red Hat Enterprise Linux 4U7 to deploy Oracle ASM

1 Deploying Oracle 10g or 11gR1 ASM Using iSCSI on Sun Storage 7000 Unified Storage Systems (2009.Q3)

2.0 Tested Configurations Sun Microsystems, Inc.

2.0 Tested Configurations

The table lists the tested configurations. Deployment of Oracle Database is supported on the various Solaris Operating System and Linux versions that Oracle supports. Please check the Oracle compatibility matrix with OS versions.

Oracle Version Operating System PlatformOracle 10.2.x Solaris 10 05/08 OS SPARC®, x86 (64-bit)Oracle 10.2.x RHEL 4.x, 5.x x86 (64-bit)Oracle 11.1.0.6, 11.1.0.7 Solaris 10 05/08 OS SPARCOracle 11.1.0.6, 11.1.0.7 RHEL 4.x, 5.x x86 (64-bit)


3.0 Configuring Sun Storage 7000 Unified Storage System Sun Microsystems, Inc.

3.0 Configuring Sun Storage 7000 Unified Storage System

3.1 Storage Conf igurat ion

The Sun Storage 7000 Unified Storage System offers varied storage configurations, such as mirrored, striped, single parity striped, dual parity striped, and RAID-Z with dual parity. A mirrored configuration provides the best performance and reliability for database applications. Refer to the documentation for Sun Storage 7000 Unified Storage System for more details.

3.2 Adding iSCSI Ini t iator Access

The volumes created in the Sun Storage 7000 Unified Storage System are the iSCSI targets and the client systems that access those iSCSI LUNs are the initiators. A many-to-many relationship between the targets and the initiators is possible. In the following example, the client system isv-6300b is registered as an authorized iSCSI initiator.

From the Solaris 10 iSCSI initiator system, note the IQN value.

# iscsiadm list initiator-nodeInitiator node name: iqn.1986-03.com.sun:01:00144faddf2c.48161da1Initiator node alias: -

Login Parameters (Default/Configured):Header Digest: NONE/-Data Digest: NONE/-

Authentication Type: NONERADIUS Server: NONERADIUS access: unknownConfigured Sessions: 1

1. From the Sun Storage 7000 Unified Storage System browser user interface (BUI), click Configuration → SAN.2. Click the Targets tab.3. Click (+) and create a target alias for the iSCSI LUNs to be accessed by the client isv-6300b. Choose the appropriate network.



4. From the Sun Storage 7000 Unified Storage System browser user interface (BUI), click Configuration → SAN.

5. Click the Initiators tab.

6. Click (+) and create an iSCSI initiator (without any special characters) and the IQN value, that can be copied and pasted.

The target and initiator created in these steps are automatically grouped by default. Optionally, the relevant targets as well as the initiators are grouped so that only certain initiators can be given access to certain targets via certain networks and so on. This is done by assigning a target group and initiator group to the individual LUN. For details, please refer to the Sun Storage 7000 documentation.

With that step, the initiator information is registered with the Sun Storage 7000 Unified Storage System. Later the initiator can be given access to the volumes that are created.

3.3 Creating a Project

A project is a collection of shares and each share is either a file system or a volume.

The following steps are done in the BUI to create a project:

1. From the BUI, create a new project: Click Share -> Projects.

2. In the Projects menu click Add, type a name for the project oracle_asm and click Apply.Note: The number of projects you create for a database depends on the application. For consistent snapshots or for preserving write-ordering for full database remote replication, all the shares used by an application that needs to be in a consistent state should be under a single project.

3.4 Creating Volumes

Volumes are the zvols that are created in the zpool. These volumes are exported as iSCSI targets that are accessed as the iSCSI LUNs by the initiator. The initiator hosts the Oracle ASM instance which uses those iSCSI LUNs as ASM disks. While the database files are used to store data on those ASM disks, for storing archived redo logs and flash recovery area (FRA), separate file systems are created



either in the same project or in a different project. Oracle 11gR2 allows all types of Oracle files to be stored in the ASM.

1. Click the oracle_asm project on the left panel.

2. Click Shares.

3. Click (+) LUN which opens up a pop-up. Input the project, LUN name (for example, asmvol1) and the size. Block size 64k seem to provide optimal performance. If Target group and Initiator groups are defined, choose them. Leave the LUN auto-assigned. Click Apply. Since the storage already provides mirroring for data protection, you can either have just one large iSCSI LUN or have two or more for redundancy.

4. Repeat the above step for multiple volumes. For this exercise, two volumes asmvol1 and asmvol2 were created. Each volume will have a different IQN value and will be exported as an iSCSI target.(This screen appears after adding the necessary LUNs.)

5. By default, the write-cache is disabled for the iSCSI LUNs. Make sure that the write cache check box is not selected, by clicking the edit button for the iSCSI LUNs.




4.0 Configuring Oracle ASM on the Solaris OS Sun Microsystems, Inc.

4.0 Configuring Oracle ASM on the Solaris OS

Configuring the Oracle ASM in the client system is addressed in three steps. The first step is to identify and configure the iSCSI LUNs that are exported from the Sun Storage 7000 Unified Storage System. The second step is to configure the Oracle ASM instance and create diskgroups which use those iSCSI LUNs. The third step is to create a database which is laid over the ASM diskgroup.

4.1 Discover ing iSCSI LUNs

The iSCSI LUNs can be discovered by using either a static or a dynamic mode. In the static mode, you configure the individual LUNs and in the dynamic mode, the initiator itself configures the visible LUNs.

The procedure defined here is based on a dynamic configuration.

1. Verify the discovery mode and enable dynamic discovery:

#iscsiadm list discovery Discovery:

Static: disabledSend Targets: enablediSNS: disabled

# # iscsiadm add discovery-address 172.20.67.159# iscsiadm list discovery-address -vDiscovery Address: 172.20.67.159:3260

Target name: iqn.1986-03.com.sun:02:42cb0da7-6b0a-42ce-b5a5-ddeb31595d8eTarget address: 172.20.67.159:3260, 2

Target name: iqn.1986-03.com.sun:02:59f367dc-e808-e961-c8f3-8dc59f76431dTarget address: 172.20.67.159:3260, 2

2. Run the format command and match the GUID listed in the Sun Storage 7000 GUI for choosing appropriate LUN.

# format Searching for disks...done..c5t600144F0D478D72D0B8C4AE234530008d0: configured with capacity of 199.93GBc5t600144F0D478D72D0B8C4AE234680009d0: configured with capacity of 199.93GB..^D

# format c5t600144F0D478D72D0B8C4AE234680009d0

c5t600144F0D478D72D0B8C4AE234680009d0: configured with capacity of 199.93GBselecting c5t600144F0D478D72D0B8C4AE234680009d0[disk formatted]

FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk



format - format and analyze the disk repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels save - save new disk/partition definitions inquiry - show vendor, product and revision volname - set 8-character volume name !<cmd> - execute <cmd>, then return quitformat> p

PARTITION MENU: 0 - change `0' partition 1 - change `1' partition 2 - change `2' partition 3 - change `3' partition 4 - change `4' partition 5 - change `5' partition 6 - change `6' partition 7 - change `7' partition select - select a predefined table modify - modify a predefined partition table name - name the current table print - display the current table label - write partition map and label to the disk !<cmd> - execute <cmd>, then return quitpartition> pCurrent partition table (default):Total disk cylinders available: 6499 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks 0 root wm 0 - 4 157.51MB (5/0/0) 322580 1 swap wu 5 - 9 157.51MB (5/0/0) 322580 2 backup wu 0 - 6498 199.93GB (6499/0/0) 419289484 3 unassigned wm 0 0 (0/0/0) 0 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 usr wm 10 - 6498 199.63GB (6489/0/0) 418644324 7 unassigned wm 0 0 (0/0/0) 0

partition> q

FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels



verify - read and display labels save - save new disk/partition definitions inquiry - show vendor, product and revision volname - set 8-character volume name !<cmd> - execute <cmd>, then return quitformat>q

5. Change ownership to oracle:dba for the devices:

# chown oracle:dba /dev/rdsk/c5t600144F0D478D72D0B8C4AE234530008d0s6# chown oracle:dba /dev/rdsk/c5t600144F0D478D72D0B8C4AE234680009d0s6

The disks are now ready to be used by the Oracle ASM instance for creating ASM diskgroup(s).

4.2 Conf igur ing Oracle ASM Instance and ASM Diskgroup

Oracle ASM instance is to be created before the diskgroups are configured. Use either Oracle's database configuration assistant GUI (dbca) or manual steps. When using dbca, click New to create ASM diskgroups and choose the iSCSI LUNs. Using these devices, one or more ASM logical diskgroups are created. Since the Sun Storage 7000 Unified Storage System already provides RAID protection, you can choose “external redundancy” while creating ASM diskgroups. This example follows the SAME (stripe and mirror everything) concept which can greatly minimize the hot spot of any data access.

1. While configuring ASM manually, set the ORACLE_SID to +ASM.

2. Then create the init+ASM.ora file:

$ cat init+ASM.oraasm_diskstring='/dev/rdsk/c5t600144F0D478D72D0B8C4AE2346*' instance_type='asm' large_pool_size=12M

asm_diskgroups='ASM_DG'

3. Start the ASM Oracle instance using init+ASM.ora with no mount:

$ sqlplus / as sysdbaSQL> startup nomount pfile=init+ASM.oraASM instance started

Total System Global Area 125829120 bytesFixed Size 1301456 bytesVariable Size 124527664 bytesDatabase Buffers 0 bytesRedo Buffers 0 bytes

4. Create the ASM diskgroups either as external redundancy or as mirrored with normal redundancy:

SQL> create diskgroup asm_dg external redundancy disk '/dev/rdsk/c1t010000144FA6E7AC00002A00486A42FFd0s6', '/dev/rdsk/c1t010000144FA6E7AC00002A00486A4634d0s6', '/dev/rdsk/c1t010000144FA6E7AC00002A00486A4666d0s6',



'/dev/rdsk/c1t010000144FA6E7AC00002A00486A4693d0s6'; Diskgroup created.

SQL> create diskgroup asm_dg external redundancy disk '/dev/rdsk/c5t600144F0D478D72D0B8C4AE234530008d0s6', '/dev/rdsk/c5t600144F0D478D72D0B8C4AE234680009d0s6'; Diskgroup created.

(or)

SQL> create diskgroup asm_dg failgroup data_fg1 disk '/dev/rdsk/c5t600144F0D478D72D0B8C4AE234530008d0s6',failgroup data_fg2 disk ''/dev/rdsk/c5t600144F0D478D72D0B8C4AE234530008d0s6';

SQL> alter diskgroup asm_dg mount ;

Now, the asm_dg diskgroup is created and ready to be used by the Oracle database. Using Oracle asmcmd, subdirectories can be created in the diskgroup and can be used in the database creation script. In this example, under ASM_DG diskgroup, DATA and LOG directories are created.

-bash-3.00$ asmcmd ASMCMD> ls ASM_DG/ ASMCMD> cd ASM_DG ASMCMD> ls ASMCMD> mkdir DATA ASMCMD> mkdir LOG ASMCMD> exit

4.3 Conf igur ing the Oracle Environment

There are some generic steps that are followed to deploy Oracle Database in the system. In the case of Oracle ASM, the database files are stored in the ASM diskgroups while the FRA and archived logs are stored in a separate file system which is created in the Sun Storage 7000 Unified Storage System and mounted using NFS share.

On the client (DB server):1. Update /etc/system with the following entry and reboot:

set noexec_user_stack=1

2. Add user oracle with group dba.

3. Create mount points to store the archived redo logs, change the ownership to oracle:dba and update /etc/vfstab. Here are example /etc/vfstab entries:

# cat /etc/vfstab | grep oraarchiveisv-USS:/export/oracle/oraarchive - /oradata/archive nfs - no rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,vers=3,noac,forcedirectio

4. As root user, mount the share:# mount /oradata/archive



With that step, the basic steps that are required to create an Oracle database are complete.

5.0 Configuring Oracle ASM on Red Hat Linux 4

To prepare a Red Hat Linux system for deploying the Oracle database using iSCSI LUNs, similar to the steps that were performed on the Solaris host, the iSCSI initiator configuration is done, then followed by the Oracle ASM configuration. For this setup, four iSCSI LUNs were created.

5.1 Discover ing iSCSI LUNs

All these steps are done on a Linux system.

1. Get the RPM package that has the iSCSI initiator utilities:http://rpm.pbone.net/index.php3/stat/4/idpl/5186811/com/iscsi-initiator-utils-4.0.3.0-2.x86_64.rpm.html

2. Content of RPM:

/etc/initiatorname.iscsi /etc/iscsi.conf /etc/rc.d/init.d/iscsi /etc/sysconfig/iscsi /sbin/iscsi-device /sbin/iscsi-iname /sbin/iscsi-ls /sbin/iscsi-rescan /sbin/iscsid /usr/share/doc/iscsi-initiator-utils-4.0.3.0 /usr/share/doc/iscsi-initiator-utils-4.0.3.0/README /usr/share/man/man1/iscsi-ls.1.gz /usr/share/man/man5/iscsi.conf.5.gz /usr/share/man/man8/iscsid.8.gz

3. Generate the iSCSI IQN name:$ iscsi-iname

4. Copy and paste the content to /etc/initiatorname.iscsi. You can edit the name after :01.

$ cat /etc/initiatorname.iscsi InitiatorName=iqn.1987-05.com.cisco:01.AR-Linux-iscsi

5. Update /etc/iscsi.conf and add the discovery address to the IP address of the target. Address=<ip address of the appliance>.


5.0 Configuring Oracle ASM on Red Hat Linux 4 Sun Microsystems, Inc.

6. To manually start the iSCSI driver enter:

# service iscsi start Checking iscsi config: [ OK ] Loading iscsi driver: [ OK ] Starting iscsid: [ OK ]

7. To see the iSCSI devices currently available on this system: #/sbin/iscsi-ls -l

Note: If there are problems loading the iSCSI kernel module, diagnostic information will be placed in /var/log/iscsi.log.

8. To manually stop the iSCSI driver enter: #service iscsi stop

9. List the iSCSI targets that are visible to the client:

# /sbin/iscsi-ls -l ******************************************************************************* SFNet iSCSI Driver Version ...4:0.1.11(12-Jan-2005) ******************************************************************************* TARGET NAME : iqn.1986-03.com.sun:02:3a97d4cc-25a2-67b3-8536-f81c285abedf TARGET ALIAS : pool_0/local/oracle_linux_iscsi/vol4 HOST ID : 5 BUS ID : 0 TARGET ID : 0 TARGET ADDRESS : 10.9.160.116:3260,1 SESSION STATUS : ESTABLISHED AT Wed Jul 2 16:06:15 MDT 2008 SESSION ID : ISID 00023d000001 TSIH a

DEVICE DETAILS: --------------- LUN ID : 0 Vendor: SUN Model: SOLARIS Rev: 1 Type: Direct-Access ANSI SCSI revision: 05 page83 type2: 010000144fa6e7ac00002a00486beeaa page80: 0a Device: /dev/sdf ******************************************************************************* TARGET NAME : iqn.1986-03.com.sun:02:c4f9d765-dc54-e4ca-a73e-f39fe4377374 TARGET ALIAS : pool_0/local/oracle_linux_iscsi/vol3 HOST ID : 6 BUS ID : 0 TARGET ID : 0 TARGET ADDRESS : 10.9.160.116:3260,1 SESSION STATUS : ESTABLISHED AT Wed Jul 2 16:06:15 MDT 2008 SESSION ID : ISID 00023d000001 TSIH 9

DEVICE DETAILS: --------------- LUN ID : 0 Vendor: SUN Model: SOLARIS Rev: 1 Type: Direct-Access ANSI SCSI revision: 05



page83 type2: 010000144fa6e7ac00002a00486beea6 page80: 0a Device: /dev/sdc ******************************************************************************* TARGET NAME : iqn.1986-03.com.sun:02:e0eb951f-53d6-cdcf-a79f-f86808341f5d TARGET ALIAS : pool_0/local/oracle_linux_iscsi/vol2 HOST ID : 7 BUS ID : 0 TARGET ID : 0 TARGET ADDRESS : 10.9.160.116:3260,1 SESSION STATUS : ESTABLISHED AT Wed Jul 2 16:06:15 MDT 2008 SESSION ID : ISID 00023d000001 TSIH 7

DEVICE DETAILS: --------------- LUN ID : 0 Vendor: SUN Model: SOLARIS Rev: 1 Type: Direct-Access ANSI SCSI revision: 05 page83 type2: 010000144fa6e7ac00002a00486beea8 page80: 0a Device: /dev/sde ******************************************************************************* TARGET NAME : iqn.1986-03.com.sun:02:ebf99081-f8cc-c038-d35e-c4e7be5cd90e TARGET ALIAS : pool_0/local/oracle_linux_iscsi/vol1 HOST ID : 8 BUS ID : 0 TARGET ID : 0 TARGET ADDRESS : 10.9.160.116:3260,1 SESSION STATUS : ESTABLISHED AT Wed Jul 2 16:06:15 MDT 2008 SESSION ID : ISID 00023d000001 TSIH 8

DEVICE DETAILS: --------------- LUN ID : 0 Vendor: SUN Model: SOLARIS Rev: 1 Type: Direct-Access ANSI SCSI revision: 05 page83 type2: 010000144fa6e7ac00002a00486beea7 page80: 0a Device: /dev/sdd

10. Use the fdisk utility to verify that the devices are being displayed:

# fdisk -l

Disk /dev/sda: 36.4 GB, 36420075008 bytes 255 heads, 63 sectors/track, 4427 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System /dev/sda1 * 1 13 104391 83 Linux /dev/sda2 14 4427 35455455 8e Linux LVM

Disk /dev/sdc: 107.3 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes



Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdd: 107.3 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdd doesn't contain a valid partition table

Disk /dev/sde: 107.3 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sde doesn't contain a valid partition table

Disk /dev/sdf: 107.3 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

11. Create partitions on the devices. The following is an example for the device /dev/sdd. Repeat that for all the devices.

# fdisk /dev/sdd

The number of cylinders for this disk is set to 13054. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-13054, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-13054, default 13054): Using default value 13054

Command (m for help): w The partition table has been altered!

With that step, the devices are ready to be configured with Oracle ASM.

5.2 Conf igur ing ASMLib

1. Download and install the ASMLib utility from Oracle's web site. Get the right RPM packages based on the Red Hat Linux version.

# ls -loracleasmlib-2.0.2-1.x86_64.rpm oracleasm-2.6.9-42.0.2.EL-2.0.3-1.ia64.rpm oracleasm-support-2.0.3-1.x86_64.rpm



2. Install the packages:

# rpm -ihv oracleasm-support-2.0.3-1.x86_64.rpm # rpm -ihv oracleasm-2.6.9-42.0.2.EL-2.0.3-1.x86_64.rpm # rpm -ihv oracleasmlib-2.0.2-1.x86_64.rpm

3. Verify that the packages are correctly installed:

# rpm -qa | grep oracleasm oracleasmlib-2.0.2-1 oracleasm-2.6.9-42.EL-2.0.3-1 oracleasm-support-2.0.3-1

4. Configure the devices to be used as ASM disks:

# ./oracleasm configure Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle Default group to own the driver interface []: oinstall Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: [ OK ] Loading module "oracleasm": [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ]

5. Use the /etc/init.d/oracleasm utility to mark the devices as ASM disks:

# ./oracleasm createdisk disk1 /dev/sdc1 Marking disk "/dev/sdc1" as an ASM disk: [ OK ] # ./oracleasm createdisk disk2 /dev/sdd1 Marking disk "/dev/sdd1" as an ASM disk: [ OK ] # ./oracleasm createdisk disk3 /dev/sde1 Marking disk "/dev/sde1" as an ASM disk: [ OK ] # ./oracleasm createdisk disk4 /dev/sdf1 Marking disk "/dev/sdf1" as an ASM disk: [ OK ]

6. Use the Oracle ASM utility to list the disks (they are typically under the /dev/oracleasm/disks directory):

# /etc/init.d/oracleasm listdisks disk1 disk2 disk3 disk4



7. Verify that the devices are marked as ASM disks. Repeat the step for all devices.

# /etc/init.d/oracleasm querydisk /dev/sdc1Disk “/dev/sdc1” is marked as ASM disk with the label “disk1”

8. Enable the Oracle ASM daemons:

# /etc/init.d/oracleasm enable Writing Oracle ASM library driver configuration: [ OK ] Scanning system for ASM disks: [ OK ]

With that step, the ASM disks are ready to be used by the Oracle ASM instance.

5.3 Conf igur ing Oracle ASM Instance and ASM Diskgroup

First, the Oracle ASM instance is configured and started. Then ASM diskgroup(s) are created and finally the Oracle database can be created on those ASM diskgroups. Creation of the ASM instance and ASM diskgroups also can be done using the dbca GUI.

The following steps create Oracle ASM manually:

1. Set ORACLE_SID to +ASM. Then create the ASM initialization parameter file.

-bash-3.00$ cat init+ASM.ora asm_diskstring='/dev/oracleasm/disks/*' instance_type='asm' large_pool_size=12M asm_diskgroups='ASM_DG'

2. Create the ASM diskgroup using the external redundancy method:

SQL> CREATE DISKGROUP ASM_DG External REDUNDANCY DISK '/dev/oracleasm/disks/DISK1' SIZE 102398M , '/dev/oracleasm/disks/DISK2' SIZE 102398M , '/dev/oracleasm/disks/DISK3' SIZE 102398M , '/dev/oracleasm/disks/DISK4' SIZE 102398M

3. Create subdirectories (optional) using the asmcmd utility:

-bash-3.00$ asmcmd ASMCMD> ls ASM_DG/ ASMCMD> cd ASM_DG ASMCMD> ls ASMCMD> mkdir DATA ASMCMD> mkdir LOG ASMCMD> exit

With that step, the ASM diskgroup is created and ready to be used with the database.



5.4 Conf igur ing the Oracle Environment

Following are the generic steps for configuring the Oracle environment.

1. Add user oracle with group dba.

2. Increase the default values for the oracle user:# cat /etc/security/limits.conf | grep oracleoracle soft nofile 131072oracle hard nofile 131072oracle soft nproc 131072oracle hard nproc 131072oracle soft core unlimitedoracle hard core unlimited

3. Add/modify entries in /etc/sysctl.conf:

# Controls IP packet forwardingnet.ipv4.ip_forward = 0# Controls source route verificationnet.ipv4.conf.default.rp_filter = 1# Do not accept source routingnet.ipv4.conf.default.accept_source_route = 0# Controls the System Request debugging functionality of the kernelkernel.sysrq = 0# Controls whether core dumps will append the PID to the core filename.# Useful for debugging multi-threaded applications.kernel.core_uses_pid = 1## Oraclekernel.shmall = 2097152kernel.shmmax = 2147483648 # Smallest of -> (Half the size of the physical memory) or (4GB - 1 byte)kernel.shmmni = 4096# semaphores: semmsl, semmns, semopm, semmnikernel.sem = 250 32000 100 128fs.file-max = 65536 # 512 * PROCESSESnet.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default=4194304net.core.rmem_max=4194304net.core.wmem_default=262144net.core.wmem_max=262144

4. As root user, run the following command for the changes to take effect: #sbin/sysctl -p

5. Create mount points /oradata/data, /oradata/logs and /oradata/archive.

6. Update /etc/fstab entries:

isv-USS:/export/oracle_linux/oraarchive /oradata/archive nfs defaults,hard,tcp,nfsvers=3,suid,nointr,actimeo=0,rsize=32768,wsize=32768 0 0

7. Mount the shares and change ownership to oracle:dba.



8. Mount the archive log directory:# mount /oradata/archive

9. Change the ownership for all the Oracle related mount points including the database mount points to oracle:dba.

10. As oracle user, set the appropriate $ORACLE_BASE, $ORACLE_HOME and $ORACLE_SID.

11. Verify whether you can create a file in the mount point /oradata/data or /oradata/logs by doing $touch /oradata/data/test. If that fails, check the permissions.

12. Install the Oracle binaries under $ORACLE_HOME location by running the runInstaller script. (Please refer to Oracle documentation for step-by-step procedures and screens for the installation).

13. You can then create the database by placing logs, control file(s) and database files in the appropriate locations.

6.0 Creating Oracle Database Using an ASM Diskgroup

The steps provided here are common to Solaris and Red Hat Linux environment. Change the ORACLE_SID to the Oracle instance, for example, asmdb. The steps shown below are to provide some guidelines on configuring ASM. Please refer to Oracle documentation for details.

1. In the initasmdb.ora file, apart from regular entries, the control files are stored in the ASM diskgroup.

control_files = ( "+ASM_DG/DATA/control1.ctl", "+ASM_DB/LOG/control2.ctl")

2. The Oracle flash recovery area and the archived logs are stored under the /oradata/archive location which was NFS mounted earlier.

3. In the database creation script, locate the physical files to the diskgroup and directories under the diskgroup:

CREATE DATABASE asmdb CONTROLFILE REUSE MAXINSTANCES 2 MAXLOGHISTORY 0 MAXLOGFILES 16 MAXLOGMEMBERS 3 MAXDATAFILES 64 DATAFILE '+ASM_DG/DATA/asmdb_system00.dbf' SIZE 512M REUSE AUTOEXTEND ON NEXT 1024K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE '+ASM_DG/DATA/asmdb_temp00.dbf' SIZE 512M REUSE AUTOEXTEND ON NEXT 640K


6.0 Creating Oracle Database Using an ASM Diskgroup Sun Microsystems, Inc.

MAXSIZE UNLIMITED UNDO TABLESPACE "UNDOTBS1" DATAFILE '+ASM_DG/DATA/asmdb_undo00.dbf' SIZE 512M REUSE AUTOEXTEND ON NEXT 1024K MAXSIZE UNLIMITED SYSAUX DATAFILE '+ASM_DG/DATA/asmdb_sysaux00.dbf' SIZE 512M REUSE AUTOEXTEND ON NEXT 1024K MAXSIZE UNLIMITED CHARACTER SET US7ASCII NATIONAL CHARACTER SET UTF8 logfile group 1 ( '+ASM_DG/LOG/asmdb_redo00-a.log' ) size 256M REUSE, group 2 ( '+ASM_DG/LOG/asmdb_redo01-a.log' ) size 256M REUSE;

4. The regular data files and index files are created under the +ASM_DG/DATA directory.

create tablespace data datafile '+ASM_DG/DATA/asmdb_data000.dbf' size 756M REUSE AUTOEXTEND ON NEXT 756M MAXSIZE UNLIMITED extent management local segment space management auto; ...create tablespace indexes datafile '+ASM_DG/DATA/asmdb_indexes000.dbf' size 512M REUSE AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED extent management local segment space management auto;

…

With that step, the database is configured and ready for use.


7.0 Glossary Sun Microsystems, Inc.

7.0 Glossary

The following table contains common abbreviations and acronyms that are used in this document.

Abbreviation Explanation

ASM Automatic Storage Management

BUI Browser User Interface

CHAP Challenge-Handshake Authentication Protocol

CIFS Common Internet File System

FRA Flash Recovery Area

IQN iSCSI Qualified Name

iSCSI Internet Small Computer Systems Interface

LUN Logical Unit Number

NFS Network File System

RAID Redundant Array of Independent Disks

RHEL Red Hat Enterprise Linux

RPM Red Hat Package Manager

SAME Stripe And Mirror Everything

SAN Storage Area Network


8.0 Author-Recommended Resources Sun Microsystems, Inc.

8.0 Author-Recommended Resources

● Oracle Technology Network: http://otn.oracle.com

● Sun product documentation at http://docs.sun.com, such as Sun Storage System documentation

9.0 For More Information

Here are additional Sun resources:

Sun Storage resources:

● Sun Storage 7000 Unified Storage System web site: http://www.sun.com/storage/disk_systems/unified_storage

● Sun Open Storage web page: http://www.sun.com/storage/disk.jsp

● Sun Storage Solutions web page: http://www.sun.com/storage/solutions.jsp

● Discussions, such as the Sun Storage forums (http://forum.java.sun.com/category.jspa?categoryID=66) and the Sun Hardware forums (http://forums.sun.com/forum.jspa?forumID=830)

● Wikis, such as Sun Storage Administration wiki (http://wikis.sun.com/display/StorageAdmin/Home) and the BigAdmin Storage Tech Tips wiki (http://wikis.sun.com/display/BigAdmin/Storage+Tech+Tips)

● Storage Stop Blog: http://blogs.sun.com/storage

● Resources on BigAdmin, such as the Storage resource collection (includes community submissions): http://www.sun.com/bigadmin/collections/storage.jsp

General Sun links:

● Sun download site: http://www.sun.com/download/

● Sun training courses web site: http://www.sun.com/training/

● Discussions, such as Sun forums (http://forums.sun.com/index.jspa) and the BigAdmin Discussions collection (http://www.sun.com/bigadmin/discussions/)

● Product documentation at http://docs.sun.com and the Documentation Center (http://www.sun.com/documentation/)

● Sun wikis, such as the Sun BluePrints™ wiki (http://wikis.sun.com/display/BluePrints/Main)and the BigAdmin wiki (http://wikis.sun.com/display/BigAdmin/Home)


http://wikis.sun.com/display/BigAdmin/Home

http://wikis.sun.com/display/BluePrints/Main

http://www.sun.com/documentation/

http://docs.sun.com/

http://www.sun.com/bigadmin/discussions/

http://forums.sun.com/index.jspa

http://www.sun.com/training/

http://www.sun.com/download/

http://www.sun.com/bigadmin/collections/storage.jsp

http://blogs.sun.com/storage

http://wikis.sun.com/display/BigAdmin/Storage+Tech+Tips

http://wikis.sun.com/display/StorageAdmin/Home

http://forums.sun.com/forum.jspa?forumID=830

http://forums.sun.com/forum.jspa?forumID=830

http://forum.java.sun.com/category.jspa?categoryID=66

http://forum.java.sun.com/category.jspa?categoryID=66

http://www.sun.com/storage/solutions.jsp

http://www.sun.com/storage/disk.jsp

http://www.sun.com/storage/disk_systems/unified_storage

http://docs.sun.com/

http://otn.oracle.com/

9.0 For More Information Sun Microsystems, Inc.

● Support:

● Sun resources:

● Register your Sun gear: https://inventory.sun.com/inventory/

● Services: http://www.sun.com/service/index.jsp

● SunSolveSM: http://sunsolve.sun.com

● Community system administration experts: http://www.sun.com/bigadmin/content/communityexperts/

10.0 Licensing Information

Unless otherwise specified, the use of this software is authorized pursuant to the terms of the license found at http://www.sun.com/bigadmin/common/berkeley_license.html.


http://www.sun.com/bigadmin/common/berkeley_license.html

http://www.sun.com/bigadmin/content/communityexperts/

http://sunsolve.sun.com/

http://www.sun.com/service/index.jsp

https://inventory.sun.com/inventory/

Documents

ASM Using ISCSI on Sun Storage