Disk Storage Setup with Linux on System z Susanne ... · Compared to other architectures, Linux on System z makes use of its own partitioning tool for DASD devices. The common Linux

IBM and Linux: Community Innovation for your Business

© 2009 IBM Corporation

Disk Storage Setupwith Linux on System z

Susanne Wintenberger ([email protected])IBM Lab Boeblingen, Germany


© 2009 IBM Corporation2

TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

The following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented asillustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about nonIBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the Performance, compatibility, or any other claims related to nonIBM products. Questions on the capabilities of nonIBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Javabased trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA, WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean

that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by ® are registered trademarks of IBM in the United States;

all others are trademarks or common law marks of IBM in the United States.



Agenda

✱ Storage– File Systems– Partitions

✱ DASD

✱ LVM– Physical Volumes (PVs)– Volume Groups (VGs)– Logical Volumes (LVs))

– Advanced LVM Topics

✱ LVM Architecture

✱ dmsetup

✱ Summary



Why is disk storage critical ?

Host system

Kernel

FileSystem

Appl.Data

SwapSpace

File System

Mem. Mgmt

SystemUtilities

ApplicationBinaries

Page Cache

Runtime environment

✱ Disk Storage usage– application data but used at

runtime– Swap space

user heap and stack data– program text

loaded on demand– shared memory segments

paged out on memory shortage.



Linux File Systems

✱ Traditional file systems– ext2– minix– MSDOS/VFAT

✱ Journaling file systems– ext3

– reiserFS– NTFS (New Technology File System)



Linux File Systems (cont'd)

Virtual File System (VFS)

ext2

Process 1 *

**

ext3 reiserfs VFAT ...

Process 2 Process 2



Linux Partition✱ Partitioning is a means to divide a single hard drive into many logical

drives.

✱ A partition is a contiguous set of blocks on a drive that are treated as an independant disk

✱ Why have multiple partitions?– Encapsulate your data. – Since file system corruption is local to a partition, you stand to loose only some of your

data if an accident occurs.– Increase disk space efficiency.– Limit data growth.

– Runaway processes or maniacal users can consume so much disk space that the operating system no longer has room on the hard drive for its bookkeeping operations.

hans@larsson:~> df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/123[...]-part1 247G 22G 212G 10% //dev/sda1 1004M 31M 923M 4% /boot/dev/mapper/home 247G 87G 157G 36% /home0[...]



Linux on System z Partitioning

disk type driver format with partition with

ECKD

FBA —

SCSI —

EDEV —

dasd dasdfmt fdasd

dasd fdisk

zfcp+scsi fdisk

dasd fdisk



Querying information about the current DASD Setup

hans@larsson:~> lsdasd Bus-ID Status Name Device Type BlkSz Size Blocks======================================================================= 0.0.ec24 active dasda 94:0 ECKD 4096 7043MB 1803060

Printing a list of active DASD devices:

The same information can also be obtained from the file /proc/dasd/devices

How to check which DASDs a currently configured for your guest

If the name of your guest if different from your hostname use vmcp q dasd only

root@larsson:~> vmcp q dasd |grep -i `hostname`DASD EC24 ATTACHED TO LARSSON EC24 R/W 0XEC24DASD EC25 ATTACHED TO LARSSON EC25 R/W 0XEC25DASD EC26 ATTACHED TO LARSSON EC26 R/W 0XEC26DASD EC27 ATTACHED TO LARSSON EC27 R/W FREEDASD EC28 ATTACHED TO LARSSON EC28 R/W 0XEC28DASD EC29 ATTACHED TO LARSSON EC29 R/W 0XEC29



Adding a DASD

root@larsson:~> modprobe dasd_mod dasd=ec27root@larsson:~> chccwdev -e ec27Setting device 0.0.ec27 onlineDoneroot@larsson:~> lsdasd Bus-ID Status Name Device Type BlkSz Size Blocks=======================================================================0.0.ec24 active dasda 94:0 ECKD 4096 7043MB 18030600.0.ec27 n/f dasdb 94:4 ECKD root@larsson:~> dmesg|tail|grep dasddasd(eckd): 0.0.ec27: 3390/0C(CU:3990/01) Cyl:10017 Head:15 Sec:224dasd(eckd): 0.0.ec27: volume analysis returned unformatted disk



DASD low level format:

root@larsson:~> dasdfmt -d cdl -b 4096 -f /dev/dasdb -pDrive Geometry: 10017 Cylinders * 15 Heads = 150255 Tracks

I am going to format the device /dev/dasdb in the following way: Device number of device : 0xec27 Labelling device : yes Disk label : VOL1 Disk identifier : 0XEC27 Extent start (trk no) : 0 Extent end (trk no) : 150254 Compatible Disk Layout : yes Blocksize : 4096

--->> ATTENTION! <<---All data of that device will be lost.Type "yes" to continue, no will leave the disk untouched: yesFormatting the device. This may take a while (get yourself a coffee).

cyl 385 of 3339 |#####---------------------------------------------| 11%

dasdfmt formats a DASD (ECKD) disk to prepare it for usage with Linux on System z



DASD: PartitioningCompared to other architectures, Linux on System z makes use of its own

partitioning tool for DASD devices. The common Linux tool fdisk can not be used in this environment!

Nevertheless the handling is similar.The system is limited to 3 partitions per disk when using DASD

root@larsson:~> fdasd /dev/dasdbreading volume label ..: VOL1reading vtoc ..........: ok

Command action m print this menu p print the partition table n add a new partition d delete a partition v change volume serial t change partition type r re-create VTOC and delete all partitions u re-create VTOC re-using existing partition sizes s show mapping (partition number - data set name) q quit without saving changes w write table to disk and exit

Command (m for help):



DASD: Partitioning (cont'd)

To create a partition:

root@larsson:~> fdasd /dev/dasdb[...]Command (m for help): nFirst track (1 track = 48 KByte) ([2]-150254): Using default value 2Last track or +size[c|k|M] (2-[150254]): Using default value 150254

Command (m for help): p

Disk /dev/dasdb: cylinders ............: 10017 tracks per cylinder ..: 15 blocks per track .....: 12 bytes per block ......: 4096 volume label .........: VOL1 volume serial ........: 0XEC27 max partitions .......: 3

------------------------------- tracks ------------------------------- Device start end length Id System /dev/dasdb1 2 150254 150253 1 Linux native



DASD: Partitioning (cont'd)

Your configuration is not completed before you write the changes to the disk

Now we have a new device partition (e.g. /dev/dasdb1) which can be used as any other Linux disk

root@larsson:~> fdasd /dev/dasdb[...]Command (m for help): wwriting VTOC...rereading partition table...

root@larsson:~> mke2fs -j /dev/dasdb1mke2fs 1.41.4 (27-Jan-2009)[...]Writing inode tables: done Creating journal (32768 blocks): doneWriting superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or180 days, whichever comes first. Use tune2fs -c or -i to override.

root@larsson:~> fdasd /dev/dasdb[...]Command (m for help): wwriting VTOC...rereading partition table...



Summary: Useful Commands

chccwdev e/denable/disable ccw device

dasdviewdisplay extended DASD information

dasdfmtlow level format for DASD (ECKD) devices

fdasdpartitioning tool for DASD

lsdasdlist DASD related device information



Logical Volumes

Physical Volumes

Logical Volume Management (LVM)

✱ Linux file system is basically inflexible

✱ It is difficult to modify partitions on a running system

✱ LVM provides a higherlevel view of the disk storage

✱ Gives you much more flexibility in allocating storage to applications and users

✱ You can resize and move logical volumes while partitions are still mounted and running

✱ Use LVM to manage logical volumes with names that make sense

Volume Group

*

**

/dev/vg1/lv2/dev/vg1/lv1



LVM Features

General:You can combine several hard disks or partitions

You can enlarge a logical volume when free space is exhausted

You can add hard disks to the volume group in a running system

You can add logical volumes in a running system

You can use several hard disks with improved performance in the RAID 0 (striping) mode

You can add up to 256 logical volumes

The Snapshot feature enables consistent backups

Benefits for Linux on System zMinidisks on z/VM cannot span more than one physical DASD volume.

Without a volume management system like LVM the size of a file system is limited to the size of a DASD volume.



root@larsson:~> pvcreate /dev/dasdb1pvcreate -- physical volume "/dev/dasdb1" successfully createdroot@larsson:~> pvcreate /dev/dasdc1pvcreate -- physical volume "/dev/dasdd1" successfully created

Physical Volume Setup

Define each Physical Volume PV

You can only add partitions and not a complete dasd (e.g. /dev/dasdd)

Check your setup

root@larsson:~> pvdisplay"/dev/dasdb1" is a new physical volume of "2.29 GB" --- NEW Physical volume --- PV Name /dev/dasdb1 VG Name PV Size 2.29 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID vwVK7v-WjGR-fdFh-GKOZ-WAtO-2cTy-r1gVs1 "/dev/dasdb1" is a new physical volume of "2.29 GB"[...]



Volume Group Setup

Create a Volume Group VG. This will be the „device“ which will contain the physical volume(s)

root@larsson:~> vgcreate vg1 /dev/dasdb1 /dev/dasdc1Volume group ”vg1" successfully createdroot@larsson:~> vgdisplay --- Volume group --- VG Name vg1 System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 4 VG Access read/write VG Status resizable

[…]

VG Size 4.58 GB PE Size 4.00 MB Total PE 1172 Alloc PE / Size 0 / 0 Free PE / Size 1172 / 4.58 GB VG UUID mQYKAf-i51q-N67F-jK2h-WL3C-hO7N-5kUWsG



Creating a Logical Volume

The following example creates a linear logical volume named lv1, based on the volume group vg1 with the size of 4.5 GigaByte

To create a striped logical volume, use i parameter

lvcreate -L 4.5G -i 2 -n lv1 vg1

root@larsson:~> lvcreate -L 4.5G -n lv1 vg1Logical volume "lv1" createdroot@larsson:~> lvdisplay --- Logical volume --- LV Name /dev/vg1/lv1 VG Name vg1 LV UUID 7Dso3A-ESEJ-mFEq-0TKu-cJcz-irox-XPsZeN LV Write Access read/write LV Status available # open 0 LV Size 4.50 GB Current LE 1152 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0



LVM – Simple Example (linear device)

Volume Group96MB

LV1 LE 056*4=24MB

LV2LE 078*4=32MB

LV3LE 0910*4=40MB

PV1PE 01112*4 = 48MB

PV2PE 01112*4 = 48MB

MappingLE>PV/PE

1PE = 1LE = 4MB (default size)

LogicalVolume (LV)

PhysicalVolume (PV)

e.g.LV3:LE7>PV2:PE9


© 2009 IBM Corporation

PhysicalVolume

PhysicalVolume

PhysicalVolume

1654

32

Logical Volume

Volume GroupVolume Group

LVM environment for striping

✱ Data evenly spread across

disks

✱ The I/O capacity of the disk

drives can be used in parallel

to access data on the logical

volume.

✱ Performance improvement

✱ No faulttolerance



LVM components at a glance

✱ A DASD partition is called a physical volume (PV), because that’s the volume where the data is physically stored.

✱ The PV is divided into several physical extents (PE) of the same size.

✱ The PEs are like blocks on the PV.

✱ Several PVs make up a volume group (VG), which becomes a pool of PEs available for the logical volume (LV).

✱ The LVs appear as normal devices in /dev/ directory.

✱ You can add or delete PVs to/from a VG, and increase/decrease your LVs.



How to create a filesystem on top of a Logical Volume

Use the same commands as for any other regular linux storage device

root@larsson:~> mke2fs -j /dev/vg1/lv1 mke2fs 1.41.4 (27-Jan-2009)Filesystem label=OS type: LinuxBlock size=4096 (log=2)Fragment size=4096 (log=2)65536 inodes, 262144 blocks13107 blocks (5.00%) reserved for the super userFirst data block=0Maximum filesystem blocks=2684354568 block groups32768 blocks per group, 32768 fragments per group8192 inodes per groupSuperblock backups stored on blocks:

32768, 98304, 163840, 229376

Writing inode tables: done Creating journal (8192 blocks): doneWriting superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 38 mounts or180 days, whichever comes first. Use tune2fs -c or -i to override.



Some more advanced LVM topics (cont.)

✱ Removing a volume group– Deactivate the volume group: vgchange -a n vg1– Now you actually remove the volume group: vgremove vg1

✱ Adding physical volumes to a volume group– use 'vgextend' to add an initialized physical volume to an existing volume group: vgextend vg1 /dev/dasdd1

✱ Removing physical volumes from a volume group– Then use 'vgreduce' to remove the physical volume: vgreduce vg1 /dev/dasdd1




✱ Removing a logical volumeA logical volume must be closed before it can be removed:

✱ Reducing a logical volume (only ext2,ext3 and reiserfs)Logical volumes can be reduced in size as well as increased. However, it is very important to

remember to reduce the size of the file system or whatever is residing in the volume before shrinking the volume itself, otherwise you risk losing data.

root@larsson:~> umount /dev/vg1/lv1root@larsson:~> lvremove /dev/vg1/lv1lvremove -- do you really want to remove "/dev/vg1/lv1"? [y/n]: ylvremove -- doing automatic backup of volume group "vg1"lvremove -- logical volume "/dev/vg1/lv1" successfully removed

root@larsson:~> umount /lv1root@larsson:~> resize2fs /dev/vg1/lv1 524288root@larsson:~> lvreduce -L-1G /dev/vg1/lv1root@larsson:~> mount /dev/vg1/lv1 /lv1




✱ Extending a logical volumeTo extend a logical volume you simply tell the lvextend command how much you want to

increase the size. You can specify how much to grow the volume, or how large you want it to grow to:

You can also specify a certain size to grow: lvextend -L+2.2G /dev/vg1/lv1

After you have extended the logical volume it is necessary to increase the file system size to match. how you do this depends on the file system you are using.

Reiserfs file systems can be resized using the following command: resize_reiserfs

root@larsson:~> lvextend -L6.7G /dev/vg1/lv1lvextend -- extending logical volume "/dev/vg1/lv1" to 6.7 GBlvextend -- doing automatic backup of volume group "vgg1"lvextend -- logical volume "/dev/vg1/lv1" successfully extended

root@larsson:~> umount /dev/vg1/lv1root@larsson:~> resize2fs /dev/vg1/lv1root@larsson:~> mount /dev/vg1/lv1 /lv1




✱ Activating of logical volumes after rebootUnless devices are enabled during boot time, any logical volumes must be activated later,

when the system is up and running. You do this with the following sequence of comands:

✱ For more Information about LVM please take a look at the HOWTO at

✱ http://tldp.org/HOWTO/LVMHOWTO/

root@larsson:~> chccwdev -e <device list>root@larsson:~> pvscanpvscan -- PV /dev/dasdb1 VG vg1 lvm2 [2.29 GB / 88.00 MB free][...]root@larsson:~> vgscanvgscan -- Reading all physical volumes. This may take a while...vgscan -- Found volume group "vg1" using metadata type lvm2root@larsson:~> lvscanlvscan -- inactive '/dev/vg1/lv1' [4.41 GB] inheritdroot@larsson:~> vgchange -ayvgchange -- 1 logical volume(s) in volume group "vg1" now active



✱ Create– {pv|vg|lv}create

✱ Display– {pv|vg|lv}display– {pv|vg|lv}s

✱ Remove– {pv|vg|lv}remove

✱ Scan– {pv|vg|lv}scan

LVM command overview

✱ Attributes / Status– {pv|vg|lv}change

✱ Organize– pvmove– {vg|lv}extend / {vg|lv}reduce– {vg|lv}rename– vgcfgbackup / vgcfgrestore– lvconvert



Linux LVM Architecture

✱ Logical Volume Management applications– dmsetup

low level logical volumemanagement

– LVM2latest version of LogicalVolume Manager

– Multipathmultipath configuration tool

MultipathLVM2dmsetup

libdevmapper

Device Mapper

kernel

User Space



✱ Libdevmapperlibrary for interaction between user and kernel device mapper

✱ Device Mapper– Device Mapper is a kernel driver that provides a generic framework for volume

management.– It provides a modular framework for constructing virtual block devices that map I/O to and

from existing devices according to a table of rules.target drivers like• Linear/striped target

• Mirror target

• Multipath target

– Responsibilities • Discover set of associated devices

• Create mapped devices (e.g. logical volume)

• Create mapping table containing configuration information

• Pass mapping table into kernel

• Possibly save mapping information

Linux LVM Architecture (cont.)

MultipathLVM2dmsetup

libdevmapper

Device Mapper



dmsetup options overview

✱ Manage

– create

– remove

– remove_all

– rename

– suspend / resume

– load / reload

✱ Info

– ls

– deps (ls tree)

– table

– status

– info (table v / status v)



Useful dmsetup commands

✱ List the current table for the device

✱ Give a list of (major, minor) pairs for devices referenced by the table

✱ List device names. tree displays dependencies between devices as a tree

– Major 253 is major number of devicemapper

hans@larsson:~> dmsetup table vg1-lv1: 0 4800512 linear 94:5 384vg1-lv1: 4800512 4636672 linear 94:9 384

hans@larsson:~> dmsetup depsvg1-lv1: 2 dependencies : (94, 9) (94, 5)

hans@larsson:~> dmsetup ls --tree vg1-lv1 (253:0) ├─ (94:9) └─ (94:5)



Outlook

Additional storage topics:

✱ PAV

✱ HyperPAV



Questions?

Documents

Disk Storage Setup with Linux on System z Susanne ... · Compared to other architectures, Linux on System z makes use of its own partitioning tool for DASD devices. The common Linux