Scott Gilliland

LVM and Raid

Why aren't flat partitions enough?

Redundancy

Speed

Live changes

Multipath I/O

Shared partitions

Raid Levels

►Raid-0: Block-level Striping

►Raid-1: Mirroring

►Raid-2: Byte striping

►Raid-3: Byte striping w/ parity disk

►Raid-4: Block-level striping with parity disk

►Raid-5: Block-level striping with distributed parity to cover the loss of any disk

►Raid-6: Block-level striping with distributed parity to cover the loss of any two disks

Raid Levels

►Raid-0: Block-level Striping

►Raid-1: Mirroring

►Raid-2: Byte striping

►Raid-3: Byte striping w/ parity disk

►Raid-4: Block-level striping with parity disk

►Raid-5: Block-level striping with distributed parity to cover the loss of any disk

►Raid-6: Block-level striping with distributed parity to cover the loss of any two disks

Raid 0

►Reads and writes are split between disks

♦ For n disks, rates go up by n

►Loss of any disk looses all data

♦ In theory, you still have n-1/n of your data, but really, you lose your data.

►Great if you have lots of disks, but don't care about your data

Raid 1

►N copies of all data

►Each write has to update all disks

♦ Ow, that's slow!

►Reads can be spread across all disks

♦ Like Raid 0, but without the data loss!

►Great if you only have two disks, and want redundancy♦ Like if you're building a machine with old drives

♦ For more n≥3, use Raid 5

Raid 5

►Distributes the parity information across all disks

►Can loose any one disk

►Reads use n-1/n disks

►Writes write to all n disks♦ Or worse – a read of the whole stripe may be needed

►Stripe sizes matter

♦ Especially for large numbers of small,disparate reads

►Usually a safe bet for 3 or more disks

Raid 6

►Usually defined as being Raid 5, but with twice the parity

►Can loose up to two disks

►Preferred over 'hot-spares'

♦ If you have an extra disk, you may as well use it

♦ How else will you know if your hot spare is bad?

Linux Software Raid

►Combines multiple block devices into one

♦ Disks or partitions

►Often called the MD system

♦ Stands for Multi Disk

►Modern arrays use a 'Superblock'

♦ Contains disk UID and array config● Allows for auto-detection

♦ Actually at end of disk

►Configured via mdadm

Linux Software Raid (cont)

►Status from /proc/mdstat

♦ And more modernly /sys/md

►You can even stack Raid sets

♦ Raid 0+1, Raid 0+5

►Can be 'reshaped' live

►Can be set to come up on boot

♦ But your bootloader will need a flat disk● Or it will need to 'understand' RAID

Typical Raid 1

►Two disks, one partition each

►Partitions mirrored as /dev/md0

►Each has a bootloader, set to boot with root=/dev/md0

►If boot disk fails, other disk can boot to the degraded array

scott@galileo:~$ cat /proc/mdstatPersonalities : [raid1]md0 : active raid1 hda1[0] hdc1[1] 78123968 blocks [2/2] [UU]

unused devices: <none>

Typical Raid 5

►A single or mirrored set of disks to boot

►A Raid 5 set as root, or as storage♦ The mirrored set often covers all disks, as same-

sized disks will all have the same extra space

♦ Here, the spare space is striped together as /tmp

-bash-3.2$ cat /proc/mdstatPersonalities : [raid6] [raid5] [raid4] [raid0]md1 : active raid0 sdc1[0] sdd1[1] sde1[2] sdf1[3] 10248192 blocks 256k chunks

md0 : active raid5 sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb3[1] sda3[0] 4870987520 blocks level 5, 256k chunk, algorithm 2 [6/6][UUUUUU]

mdadm

Seems to be the best way to change an array

Create:# mdadm --create /dev/md0 --level=5 --raid-devices=3 \ /dev/sdc1 /dev/sdd1 /dev/sde1

Set as faulty, remove, add:# mdadm --manage /dev/md0 --fail /dev/sdc1# mdadm --manage /dev/md0 --remove /dev/sdc1# mdadm --manage /dev/md0 --add /dev/sdc1

LVM (and LVM2)

►Can be considered an alternative to traditional partitioning♦ Can use any block device: partitions and full disks

►Allows for rearranging of storage, mirroring, striping, and snapshots

►With a daemon, can handle shared storage and multipath♦ Ex: two servers with a shared SAN disk between

them

►Does not do Raid 5 or 6.

♦ Or Raid 2, Raid 3, or Raid 4.

LVM Basics

►3 main types of entities:

♦ PV – Physical Volume● This is a disk you want to store data on.

♦ VG – Volume Group● A pool of PV's on a machine. You usually just want

one.

♦ LV – Logical Volume● An exposed block device. May be spanned,

striped, mirrored, or a 'snapshot'.

LVM Snapshots

►Allows you to take a full-disk image at a single point in time

►Uses Copy-on-write

♦ Only needs additional space when the device is written

♦ Needs physical space allocated for changes

►Useful for taking backups that are self-consistent

♦ Make snapshot, backup contents off-disk, erase snapshot

LVM configuration

►Command-line tools are simplest♦ All the details in LVM(8)

►System-config-lvm is the Redhat GUI tool♦ Only handles LVM, making MD+LVM configurations

difficult

►EVMS tool has an LVM2 module♦ Not actively maintained?

♦ Also has an MD module for software RAID.

►KVPM is a kde4-based manager♦ Seems kinda beta, but works

♦ Can parse MD devices

♦ Handles filesystems

Command-line LVM

Most of:[pv|vg|lv][change|convert|create|display|extend|reduce|remove|rename|resize|s|scan]

lvm> pvs PV VG Fmt Attr PSize PFree /dev/md0 test lvm2 a- 1.91G 1.62G /dev/sdc2 test lvm2 a- 1.44G 974.00M /dev/sde2 test lvm2 a- 1.39G 1.20Glvm> vgs VG #PV #LV #SN Attr VSize VFree test 3 2 0 wz--n- 4.75G 3.77Glvm> lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert mirrored test mwn-a- 300.00M mirrored_mlog 100.00 striped test -wn-a- 400.00M

Command-line LVM

PVS, VGS, and LVS can take additional fields

scott@bravil:/usr/src/kvpm-0.4.5b$ sudo pvs -o pv_fmt,pv_size,dev_size,pv_free,pv_used,pv_name,pv_attr,pv_pe_count,pv_pe_alloc_count,pv_tags,pvseg_start,pvseg_size,pe_start,lv_name Fmt PSize DevSize PFree Used PV Attr PE Alloc PV Tags Start SSize 1st PE LV lvm2 1.44G 1.44G 724.00M 750.00M /dev/sdc2 a- 737 375 0 100 192.00K striped lvm2 1.44G 1.44G 724.00M 750.00M /dev/sdc2 a- 737 375 100 150 192.00K mirrored lvm2 1.44G 1.44G 724.00M 750.00M /dev/sdc2 a- 737 375 250 125 192.00K striped lvm2 1.44G 1.44G 724.00M 750.00M /dev/sdc2 a- 737 375 375 362 192.00K lvm2 486.00M 486.31M 486.00M 0 /dev/sdd1 a- 243 0 0 243 192.00K lvm2 1.39G 1.40G 978.00M 450.00M /dev/sde2 a- 714 225 0 100 192.00K striped lvm2 1.39G 1.40G 978.00M 450.00M /dev/sde2 a- 714 225 100 1 192.00K lvm2 1.39G 1.40G 978.00M 450.00M /dev/sde2 a- 714 225 101 125 192.00K striped lvm2 1.39G 1.40G 978.00M 450.00M /dev/sde2 a- 714 225 226 488 192.00K

sudo lvs -o +segtype,stripes,stripesize,devices,seg_count,copy_percent LV VG Attr LSize Origin Snap% Move Log Copy% Convert Type #Str Stripe Devices #Seg Copy% mirrored test mwn-a- 300.00M mirrored_mlog 31.33 mirror 2 0 mirrored_mimage_0(0),mirrored_mimage_1(0) 1 31.33 striped test -wn-a- 900.00M striped 2 4.00K /dev/sdc2(0),/dev/sde2(0) 2 striped test -wn-a- 900.00M striped 2 4.00K /dev/sdc2(250),/dev/sde2(101) 2

KVPM

System-config-lvm