1. Solaris 10 Administration Topics Workshop 2 - Virtualization By Peter Baer GalvinFor Usenix Last Revision Apr 2009Copyright 2009 Peter Baer Galvin - All Rights ReservedSaturday, May 2, 2009

2. About the Speaker Peter Baer Galvin - 781 273 4100 pbg@cptech.com www.cptech.com peter@galvin.info My Blog: www.galvin.info Bio Peter Baer Galvin is the Chief Technologist for Corporate Technologies, Inc., a leading systems integrator and VAR, and was the Systems Manager for Brown Universitys Computer Science Department. He has written articles for Byte and other magazines. He was contributing editor of the Solaris Corner for SysAdmin Magazine , wrote Petes Wicked World, the security column for SunWorld magazine, and Petes Super Systems, the systems administration column there. He is now Sun columnist for the Usenix ;login: magazine. Peter is co-author of the Operating Systems Concepts and Applied Operating Systems Concepts texbooks. As a consultant and trainer, Mr. Galvin has taught tutorials in security and system administration and given talks at many conferences and institutions. Copyright 2008 Peter Baer Galvin - All Rights Reserved2Saturday, May 2, 2009 3. ObjectivesCover a wide variety of topics in Solaris 10Useful for experienced system administratorsSave timeAvoid (my) mistakesLearn about new stuffAnswer your questions about old stuffWont read the man pages to youWorkshop for hands-on experience and to reinforce conceptsNote Security covered in separate tutorialCopyright 2009 Peter Baer Galvin - All Rights Reserved 3Saturday, May 2, 2009 4. More ObjectivesWhat makes novice vs. advanced administrator? Bytes as well as bits, tactics and strategy Knows how to avoid trouble How to get out of it once in it How to not make it worse Has reasoned philosophy Has methodologyCopyright 2009 Peter Baer Galvin - All Rights Reserved 4Saturday, May 2, 2009 5. PrerequisitesRecommend at least a couple of years ofSolaris experience Or at least a few years of other Unix experienceBest is a few years of admin experience,mostly on Solaris Copyright 2009 Peter Baer Galvin - All Rights Reserved 5Saturday, May 2, 2009 6. About the TutorialEvery SysAdmin has a different knowledge setA lot to cover, but notes should make goodreference So some covered quickly, some in detail Setting base of knowledge Please ask questions But lets take off-topic off-line Solaris BOFCopyright 2009 Peter Baer Galvin - All Rights Reserved 6Saturday, May 2, 2009 7. Fair WarningSites varyCircumstances varyAdmin knowledge variesMy goals Provide information useful for each of you at your sites Provide opportunity for you to learn from each otherCopyright 2009 Peter Baer Galvin - All Rights Reserved 7Saturday, May 2, 2009 8. Why Listen to Me 20 Years of Sun experience Seen much as a consultant Hopefully, youve used:My Usenix ;login: columnThe Solaris Corner @ www.samag.comThe Solaris Security FAQSunWorld Petes Wicked WorldSunWorld Petes Super SystemsUnix Secure Programming FAQ (out of date)Operating System Concepts (The Dino Book), now 8th edApplied Operating System Concepts Copyright 2009 Peter Baer Galvin - All Rights Reserved 8Saturday, May 2, 2009 9. Slide OwnershipAs indicated per slide, some slidescopyright Sun MicrosystemsThanks to Jeff Victor for inputFeel free to share all the slides - as long asyou dont charge for them or teach fromthem for feeCopyright 2009 Peter Baer Galvin - All Rights Reserved 9Saturday, May 2, 2009 10. Overview Lay of the LandCopyright 2009 Peter Baer Galvin - All Rights ReservedSaturday, May 2, 2009 11. Schedule Times and BreaksCopyright 2009 Peter Baer Galvin - All Rights Reserved 11Saturday, May 2, 2009 12. CoverageSolaris 10+, with some Solaris 9 whereneededSelected topics that are new, different,confusing, underused, overused, etc Copyright 2009 Peter Baer Galvin - All Rights Reserved 12Saturday, May 2, 2009 13. OutlineOverviewObjectivesVirtualization choices in SolarisZones / ContainersLDOMS and DomainsVirtualboxXvm (aka Xen) Copyright 2009 Peter Baer Galvin - All Rights Reserved 13Saturday, May 2, 2009 14. Polling TimeSolaris releases in use? Plans to upgrade?Other OSes in use?Use of Solaris rising or falling? SPARC and x86 OpenSolaris?Copyright 2009 Peter Baer Galvin - All Rights Reserved 14Saturday, May 2, 2009 15. Your Objectives?Copyright 2009 Peter Baer Galvin - All Rights Reserved 15Saturday, May 2, 2009 16. Your Lab EnvironmentApple Macbook Pro 3GB memory Mac OS X 10.4.10VMware Fusion 1.0Solaris Nevada 50 Containers Copyright 2009 Peter Baer Galvin - All Rights Reserved 16Saturday, May 2, 2009 17. Lab PreparationHave device capable of telnet on theUSENIX network Or have a buddyLearn your magic numberTelnet to 131.106.62.100+magic numberUser root, password lisa Its all very secureCopyright 2009 Peter Baer Galvin - All Rights Reserved 17Saturday, May 2, 2009 18. Lab PreparationOr... Use virtualbox Use your own system Use a remote machine you have legit access toCopyright 2009 Peter Baer Galvin - All Rights Reserved 18Saturday, May 2, 2009 19. Lab PreparationOr... Use virtualbox Use your own system Use a remote machine you have legit access toCopyright 2009 Peter Baer Galvin - All Rights Reserved 19Saturday, May 2, 2009 20. Choosing Virtualization Technologies(See separate virtualization comparisondocument) Copyright 2009 Peter Baer Galvin - All Rights Reserved 20Saturday, May 2, 2009 21. !"#$%&()*"+,(-+*(.#&!/01*)"2/012(301$%$%4-, 5%1$"0#(!067%-,)*(5%1$"0#%80$%4- 9,4"16(!0-0.:-$!"#$%()*+,(*%-.#()*O1-2($4(B#D%P%#%$< O1-2($4(%,4#0$%4- C4.%60#(;4:0%-, *4#01%,(=4-$0%-1, *4#01%,(9,4"16; set physical=pnc0 zonecfg:app1:net> set address=192.168.118.140 zonecfg:app1:net> end zonecfg:app1> add fs zonecfg:app1:fs> set dir=/export/home zonecfg:app1:fs> set special=/export/home zonecfg:app1:fs> set type=lofs zonecfg:app1:fs> end zonecfg:app1> add inherit-pkg-dir zonecfg:app1:inherit-pkg-dir> set dir=/opt/sfw zonecfg:app1:inherit-pkg-dir> end zonecfg:app1> verify zonecfg:app1> commit zonecfg:app1> exitCopyright 2009 Peter Baer Galvin - All Rights Reserved 47Saturday, May 2, 2009 48. Zone Conguration - 3 # df -k Filesystemkbytesused avail capacity Mounted on /dev/dsk/c0d0s05678823 2689099 293293648% / /devices 000 0% /devices /dev/dsk/c0d0p0:boot 102961401889514% /boot proc 000 0% /proc mnttab 000 0% /etc/mnttab fd 000 0% /dev/fd swap600780 28 600752 1% /var/run swap600776 24 600752 1% /tmp /dev/dsk/c0d0s74030684 32853 3957525 1% /export/home # zoneadm -z app1 verify WARNING: /opt/zone/app1 does not exist, so it cannot be verified. When zoneadm install is run, install will try to create /opt/zone/app1, and verify will be tried again, but the verify may fail if: the parent directory of /opt/zone/app1 is group- or other-writable or /opt/zone/app1 overlaps with any other installed zones. could not verify net address=192.168.118.140 physical=pnc0: No such device or address zoneadm: zone app1 failed to verifyCopyright 2009 Peter Baer Galvin - All Rights Reserved 48Saturday, May 2, 2009 49. Zone Conguration - 4 # ls -l /opt/zone total 2 drwx------ 4 rootother 512 Aug 21 12:44 test # mkdir /opt/zone/app1 # chmod 700 /opt/zone/app1 # ls -l /opt/zone total 4 drwx------ 2 rootother 512 Sep 16 15:14 app1 drwx------ 4 rootother 512 Aug 21 12:44 test # zonadm -z app1 verify could not verify net address=192.168.118.140physical=pnc0: No such device or address zoneadm: zone app1 failed to verify # zonecfg -z app1 zonecfg:app1> info zonepath: /opt/zone/app1 autoboot: falseCopyright 2009 Peter Baer Galvin - All Rights Reserved 49Saturday, May 2, 2009 50. Zone Conguration - 5 net:address: 192.168.118.140physical: pnc0 zonecfg:app1> remove physical=pnc0 zonecfg:app1> add net zonecfg:app1:net> set physical=pcn0 zonecfg:app1:net> set address=192.168.118.140 zonecfg:app1:net> end zonecfg:app1> exit # zoneadm -z app1 verify # zoneadm -z app1 install Preparing to install zone . Creating list of files to copy from the global zone. Copying files to the zone. Initializing zone product registry. Determining zone package initialization order. Preparing to initialize packages on the zone. Initializing package of : percent complete: 0% . . . Copyright 2009 Peter Baer Galvin - All Rights Reserved 50Saturday, May 2, 2009 51. Zone Conguration -6 Zone is initialized. The file /zone/app1/root/var/sadm/system/logs/install_log> contains a log of the zone installation. # zoneadm list -vID NAME STATUSPATH 0 global running / 1 test running /opt/zone/test # df -k Filesystem kbytesused avail capacity Mounted on /dev/dsk/c0d0s0 5678823 2766177 285585850% / /devices0 0 0 0% /devices /dev/dsk/c0d0p0:boot102961401 8895 14% /boot proc 0 00 0% /proc mnttab 0 00 0% /etc/mnttab fd00 00% /dev/fd swap 594332 325943001% /var/run swap 594500 200 5943001% /tmp /dev/dsk/c0d0s7 4030684 32853 3957525 1% /export/home Copyright 2009 Peter Baer Galvin - All Rights Reserved51Saturday, May 2, 2009 52. Zone Conguration -7 # zoneadm -z app1 boot zoneadm: zone app1: WARNING: pcn0:2: no matching subnet found in netmasks(4) for 192.168.118.131; using default of 192.168.118.131. # zoneadm list -v ID NAME STATUS PATH0 global running/1 test running/opt/zone/test2 app1 running/opt/zone/app1 # telnet 192.168.118.140 Trying 192.168.118.140... telnet: Unable to connect to remote host: Connection refused # zlogin -C app1 [Connected to zone app1 console] Select a Locale 0. English (C - 7-bit ASCII) 1. U.S.A. (UTF-8) 2. Go Back to Previous Screen Please make a choice (0 - 2), or press h or ? for help: 0 . . . Copyright 2009 Peter Baer Galvin - All Rights Reserved 52Saturday, May 2, 2009 53. Zone Conguration -8 rebooting system due to change(s) in /etc/default/init [NOTICE: Zone rebooting] SunOS Release 5.10 Version s10_63 32-bit Copyright 1983-2004 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zone-app1 The system is coming up. Please wait. starting rpc services: rpcbind done. syslog service starting. Sep 16 15:48:24 zone-app1 sendmail[7567]: My unqualified hostname (zone-app1) unknown; sleeping for retry Sep 16 15:49:24 zone-app1 sendmail[7567]: unable to qualify myown domain name (zone-app1) -- using short name WARNING: local host name (zone-app1) is not qualified; see cf/README: WHO AM I? /etc/mail/aliases: 12 aliases, longest 10 bytes, 138 bytes totalCopyright 2009 Peter Baer Galvin - All Rights Reserved 53Saturday, May 2, 2009 54. Zone Conguration -9 Creating new rsa public/private host key pair Creating new dsa public/private host key pair The system is ready. zone-app1 console login: root Password: Sep 16 15:51:08 zone-app1 login: ROOT LOGIN /dev/console Sun Microsystems Inc. SunOS 5.10 s10_63 May 2004 # cat /etc/passwd root:x:0:1:Super-User:/:/sbin/sh daemon:x:1:1::/: bin:x:2:2::/usr/bin: . . . noaccess:x:60002:60002:No Access User:/: nobody4:x:65534:65534:SunOS 4.x NFS Anonymous AccessUser:/:Copyright 2009 Peter Baer Galvin - All Rights Reserved 54Saturday, May 2, 2009 55. Zone Conguration -10# useradd -u 101 -g 14 -d /export/home/pbg -s /bin/bash pbg# passwd pbgNew Password:Re-enter new Password:passwd: password successfully changed for pbg# zoneadm list -vID NAMESTATUSPATH 3 app1running /# exitzone-app1 console login: ~.[Connection to zone app1 console closed]Copyright 2009 Peter Baer Galvin - All Rights Reserved 55Saturday, May 2, 2009 56. Zone Conguration - 11# zoneadm list -v ID NAME STATUS PATH0 global running/1 test running/opt/zone/test3 app1 running/opt/zone/app1 # uptime 3:53pm up 5:14, 1 user, load average: 0.23, 0.34, 0.43 # telnet 192.168.118.140 Trying 192.168.118.140 Connected to 192.168.118.140. Escape character is ^]. Login: pbg Password:Copyright 2009 Peter Baer Galvin - All Rights Reserved 56Saturday, May 2, 2009 57. Zones and ZFSInstalling a zone with its root on ZFS is not supported asthe system then lacks the ability to be upgraded.Note that add fs can be used to add access to a ZFS lesystem to a zoneBeyond that, add dataset delegates a ZFS le system toa zone, removes it from the global zoneThe zone can manage the le system, except where managementwould effect other le systems / parent le systemFilesystem contents can still be seen from global zone via zonepath+mountpoint (i.e. /zones/zone00/zfs/zonefs/zone00) # zfs create zfs/zonefs/zone00 # zonecfg -z zone00 zonecfg:zone00> add dataset zonecfg:zone00:dataset> set name=zfs/zonefs/zone00 zonecfg:zone00:dataset> end Copyright 2009 Peter Baer Galvin - All Rights Reserved 57Saturday, May 2, 2009 58. Zone Script create -b set zonepath=/opt/zones/zone0 set autoboot=false add inherit-pkg-dir set dir=/lib end add inherit-pkg-dir set dir=/platform end add inherit-pkg-dir set dir=/sbin end Copyright 2009 Peter Baer Galvin - All Rights Reserved 58Saturday, May 2, 2009 59. Zone Script add inherit-pkg-dir set dir=/usr end add inherit-pkg-dir set dir=/opt/sfw end add net set address=192.168.128.200 set physical=pcn0 end add rctl set name=zone.cpu-shares add value (priv=privileged,limit=1,action=none) endCopyright 2009 Peter Baer Galvin - All Rights Reserved 59Saturday, May 2, 2009 60. Life in a Zone # ifconfig -a lo0: flags=1000849 mtu 8232 index 1inet 127.0.0.1 netmask ff000000 lo0:1: flags=1000849 mtu 8232 index 1zone testinet 127.0.0.1 netmask ff000000 lo0:2: flags=1000849 mtu 8232 index 1zone app1inet 127.0.0.1 netmask ff000000 pcn0: flags=1004843 mtu 1500 index 2inet 192.168.80.128 netmask ffffff00 broadcast 192.168.80.255ether 0:c:29:44:a9:df pcn0:1: flags=1000843 mtu 1500 index 2zone testinet 192.168.80.139 netmask ffffff00 broadcast 192.168.80.255 pcn0:2: flags=1000843 mtu 1500 index 2zone app1inet 192.168.80.140 netmask ffffff00 broadcast 192.168.80.255Copyright 2009 Peter Baer Galvin - All Rights Reserved60Saturday, May 2, 2009 61. Life in a Zone - 2 $ telnet 192.168.80.140 . . . $ df -k Filesystemkbytesusedavail capacity Mounted on /9515147 1894908 7525088 21% / /dev 9515147 1894908 7525088 21% /dev /export/home 10076926 10369 99657881% /export/home /lib 9515147 1894908 7525088 21% /lib /platform9515147 1894908 7525088 21% /platform /sbin9515147 1894908 7525088 21% /sbin /usr 9515147 1894908 7525088 21% /usr proc 00 0 0% /proc mnttab 00 0 0% /etc/mnttab fd 00 0 0% /dev/fd swap 1043072 16 1043056 1% /var/run swap 10430560 1043056 0% /tmp $ touch /usr/foo touch: /usr/foo cannot createNote that virtual memory (and therefore swap) are globalresourcesCopyright 2009 Peter Baer Galvin - All Rights Reserved 61Saturday, May 2, 2009 62. Life in a Zone - 3 $ ps -efUID PID PPID CSTIME TTYTIME CMD root 11120 111200 11:00:35 ?0:00 zschedpbg 11377 113470 11:01:28 pts/80:00 ps -ef root 11229 111200 11:00:40 ?0:00 /usr/sbin/cron root 11341 111200 11:00:46 ?0:00 /usr/sfw/sbin/snmpd root 11266 111200 11:00:41 ?0:00 /usr/lib/im/htt -port 9010 -s yslog -message_locale C root 11339 113360 11:00:46 ?0:00 /usr/lib/saf/ttymon root 11250 111200 11:00:41 ?0:00 /usr/lib/utmpd root 11264 112610 11:00:41 ?0:00 /usr/sadm/lib/smc/bin/smcboot root 11261 111200 11:00:41 ?0:00 /usr/sadm/lib/smc/bin/smcboot root 11227 111200 11:00:40 ?0:00 /usr/sbin/nscd root 11218 111200 11:00:40 ?0:00 /usr/lib/autofs/automountd root 11325 111200 11:00:45 ?0:00 /usr/lib/dmi/snmpXdmid -s zon e-app1 root 11239 111200 11:00:40 ?0:00 /usr/lib/sendmail -bd -q15m root 11265 112610 11:00:41 ?0:00 /usr/sadm/lib/smc/bin/smcboot root 11230 111200 11:00:40 ?0:00 /usr/sbin/inetd -s root 11273 112660 11:00:42 ?0:00 htt_server -port 9010 -syslog-message_locale C root 11129 111200 11:00:36 ?0:00 init Copyright 2009 Peter Baer Galvin - All Rights Reserved 62Saturday, May 2, 2009 63. Life in a Zone - 4 # mount -p / - / ufs - no rw,intr,largefiles,logging,xattr,onerror=panic /dev - /dev lofs - no zonedevfs /export/home - /export/home lofs - no /lib - /lib lofs - no ro,nodevices,nosub /platform - /platform lofs - no ro,nodevices,nosub /sbin - /sbin lofs - no ro,nodevices,nosub /usr - /usr lofs - no ro,nodevices,nosub proc - /proc proc - no nodevices,zone=app1 mnttab - /etc/mnttab mntfs - no nodevices,zone=app1 fd - /dev/fd fd - no rw,nodevices,zone=app1 swap - /var/run tmpfs - no nodevices,xattr,zone=app1 swap - /tmp tmpfs - no nodevices,xattr,zone=app1 # hostname zone-app1 # zonename app1Copyright 2009 Peter Baer Galvin - All Rights Reserved 63Saturday, May 2, 2009 64. Zone CloneAs of S10 8/07, zones are cloneableMuch faster than installing a zone As of 10/08 zones on ZFS -> ZFS clone - instantaneousUsable only if the zones of similar congsCongure a zone i.e. zone00Install the zoneCongure a new zone i.e. zone01Then rather than zoneadm install, with zone00 halted, do # zoneadm z zone01 clone m copy zone00 Copyright 2009 Peter Baer Galvin - All Rights Reserved 64Saturday, May 2, 2009 65. Zone Clone (cont)A cloned zone is uncongured and must beconguredWhen ZFS used as clone le system# zoneadm -z clone Can clone a zones previously-takensnapshot via# zoneadm -z clone -s Copyright 2009 Peter Baer Galvin - All Rights Reserved 65Saturday, May 2, 2009 66. Zone Clone (cont)So to clone zone1 to make zone2# zonecfg -z zone1 export -f configfileEdit congle to change zonepath and address (atleast)Create zone2 via zonecfg -z zone2 -fconfigfileHalt zone1 via zoneadm -z zone1 haltClone zone1 via zoneadm -z zone2 clone zone1 Use -m copy if zone1 on UFSBoot up both zones Check status via zoneadm list -iv Copyright 2009 Peter Baer Galvin - All Rights Reserved 66Saturday, May 2, 2009 67. Zone MigrationZones can be moved between like systemsAvailable S10 8/07Separate the zone from its current system # zoneadm z detachNote zone must be halted rstAttach a detached zone to a different system (assuming itsle system is now visible there, send a tarball, etc) # zoneadm z attach [-F]Note zone must be congured before this can workNote new system is validated to assure the zone can function thereTo create a cong for a zone that is detached rather thanhaving to zonecfg it from scratch # zonecfg z create -a zonepath Copyright 2009 Peter Baer Galvin - All Rights Reserved67Saturday, May 2, 2009 68. Zone Migration (cont)Can dry-run an attach / detach via the -n option tosee if the attach will workCan upgrade the attaching zone on the attachingsystem via -u but only if all packages on theattaching system are as new or newer than thedetaching systemCan force an attach if a detach could not be done(dead system for example)Best to save your zone cfg les for use on theattach system (or you have to recreate them)Copyright 2009 Peter Baer Galvin - All Rights Reserved 68Saturday, May 2, 2009 69. Other Cool Zone Stuffps Z shows zone in which each process is runningCan use resource manager with zonesZones can use global naming servicesUse features to enable or disable accounts per zoneInterzone networking executed via loopback forperformanceCopyright 2009 Peter Baer Galvin - All Rights Reserved 69Saturday, May 2, 2009 70. LabsCreate a simple zoneInstall itBoot itCongure itLook around in it - le systems, processes,resource use, users, etcHalt itCopyright 2009 Peter Baer Galvin - All Rights Reserved 70Saturday, May 2, 2009 71. Zones and DTraceZones can get some DTrace privileges (starting 11/06)# zonecfg -z my-zonezonecfg:my-zone> setlimitpriv="default,dtrace_proc,dtrace_user"zonecfg:my-zone> exitDTrace can use zonenames are predicates to lterresults# dtrace -n syscall:::/zonename==zone1/{@[probefunc]=count()} Copyright 2009 Peter Baer Galvin - All Rights Reserved 71Saturday, May 2, 2009 72. Fair-share SchedulingSolaris has many scheduler classes availableA thread has priority 0-169, user threads are 0-59The higher the priority, the sooner scheduled on CPUScheduler class decides how the priority is modied over timeDefault user-land is Time-sharing Time-sharing dynamically changes the priority of each thread based on its activity If a thread used it time quantum, its priority decreases (The quantum is the scheduling interval)Kernel uses sys classHave a look via ps -elfcCopyright 2009 Peter Baer Galvin - All Rights Reserved 72Saturday, May 2, 2009 73. !"#$%&"$(%&)(*+,($Fair-share Scheduling!"#$%&"$(%&)(*+,($ !"#$%&"$(%&)(*+,($2 22 1Bac upk AppSer er v 31 Bac up kBac up abas k Dat e31AppSer erDat ev abasWebAppSer erv3 Web Dat eabasWebDatabase gets 4 / 4+3+2+1= 40% of all CPU!!! 54 ! $ $!% 4$!%$ time available to container !""#"!"$# "% !""#"!"$# ! "%!4!"#$%&())*+#,%-,*.*/,#0/%$&$ $!%5 !""#"!"$# "%!"#$%&())*+#,%-,*.*/,#0/%$&!"#$%&()*+*,-.*$/()0(&-,1(+$2$3)0+(&45,$6778!"#$%&())*+#,%-,*.*/,#0/%$&!"#$%&()*+*,-.*$/()0(&-,1(+$2$3)0+(&45,$6778Copyright 2009 Peter Baer Galvin - All Rights Reserved 73 !"#$%&()*+*,-.*$/()0(&-,1(+$2$3)0+(&45,$6778Saturday, May 2, 2009 74. Zones and Fair Share SchedulingFSS allows all CPU to be used if needed, but overuse tobe limited based on shares given to CPU usersShares give to projects et al, and/or to containersLoad the fair share schedule as the default scheduleclassdispadmin d FSSMove all processes into the FSS classpriocntl -s -c FSS -i class TSGive the global zone some (2) sharesNote this is not persistent across reboots!prctl -n zone.cpu-shares -v 2 -r -i zoneglobalCopyright 2009 Peter Baer Galvin - All Rights Reserved 74Saturday, May 2, 2009 75. Zones and Fair-share scheduling (2)Check the shares of the global zone prctl -n zone.cpu-shares -i zone globalAdd a zone-wide resource control (1 share) to a zone(within zonecfg) (before S10U5) zonecfg:my-zone> add rctl zonecfg:my-zone:rctl> set name=zone.cpu- shares zonecfg:my-zone:rctl> add value(priv=privileged,limit=1,action=none) zonecfg:my-zone:rctl> endHow many total shares are given out on a givenmachine? Copyright 2009 Peter Baer Galvin - All Rights Reserved 75Saturday, May 2, 2009 76. FX SchedulerTime-share is heavy weight scheduler Has to calculate for every thread that ran in the last quantum, every quantum Plus decreases priority on CPU hogsInstead consider FX - xed scheduler class All priorities stay the same Light weight schedule can gain back a few percent of CPU Copyright 2009 Peter Baer Galvin - All Rights Reserved 76Saturday, May 2, 2009 77. !"#$%&()*+,-.*(/,,0+! 9-*&4#-:$,)$4()"0$5)*-#$(-*)"(-*$*"5$1*$3/;*$0()F-,>$3)#,1-(! G$3)#,1-($1#$C-$*,1,&1::+$1**&4#-=$,)$1#$-?&*,$H*51(-=I$0)):$J5-#$,5-$3)#,1-($C)),*" %":,&0:-$3)#,1-(*$1#$*51(-$,51,$0)):" !"5$1$3)#,1-($)#:+$"*-*$(-*)"(-*$J5-#$&,$&*$ ("##! G$3)#,1-($1#$C-$1**&4#-=$,)$1$,-.0)(1(+$0)):" /)):$)#:+$-?&*,*$J5&:-$3)#,1-($("#*" 951,$0)):$1##),$C-$*51(-=$J&,5$),5-($3)#,1-(*!"#$%&()*+*,-.*$/()0(&-,1(+$2$3)0+(&45,$6778 Copyright 2009 Peter Baer Galvin - All Rights Reserved 78Saturday, May 2, 2009 79. DRPsYou can make DRPs non-dynamic by not includinga variation in the range (i.e. 2 to 2 rather than 1 to 2)Probably preferred rather than real dynamicWith pools, interrupts and I/O only occur in thedefault poolThis can help pin a process to a set of CPUSCache stays hot, less context switchingSo consider a DRP cong with the kernel in thedefault pool and all apps in another poolCopyright 2009 Peter Baer Galvin - All Rights Reserved 79Saturday, May 2, 2009 80. Zones and Dynamic Resource PoolsAssign zones to dedicated CPU resources Used to assign zone to processor set Can be dynamically created, deleted, modiedCan be used with FSSCan be used to reduce Oracle (and other?) costs!Consider two DRPs, one with an email containerand one with 2 X web server containers (andglobal) (from http://www.sun.com/software/solaris/howtoguides/containersLowRes.jsp): Copyright 2009 Peter Baer Galvin - All Rights Reserved 80Saturday, May 2, 2009 81. Zones and DRPs (cont)Copyright 2009 Peter Baer Galvin - All Rights Reserved 81Saturday, May 2, 2009 82. Zones and DRPs (cont)Create a pool (from global zone) via # # enable DRPs # pooladm e # # save current config # pooladm s # # show current state, at start only pool_default exists global# pooladmsystem my_system string system.comment int system.version 1 boolean system.bind-default true int system.poold.pid 638 pool pool_defaultint pool.sys_id 0boolean pool.active trueboolean pool.default trueint pool.importance 1string pool.commentpset pset_defaultCopyright 2009 Peter Baer Galvin - All Rights Reserved 82Saturday, May 2, 2009 83. Zones and DRPs (cont) pset pset_default int pset.sys_id -1 boolean pset.default true uint pset.min 1 uint pset.max 65536 string pset.units population uint pset.load 7 uint pset.size 8 string pset.commentcpuint cpu.sys_id 1string cpu.commentstring cpu.status on-linecpuint cpu.sys_id 0string cpu.commentstring cpu.status on-linecpuint cpu.sys_id 3string cpu.commentstring cpu.status on-linecpuint cpu.sys_id 2string cpu.commentstring cpu.status on-line Copyright 2009 Peter Baer Galvin - All Rights Reserved 83Saturday, May 2, 2009 84. Zones and DRPs (cont) Create a new one-CPU processor set called email-pset # poolcfg -c create pset email-pset (uint pset.min=1; uint pset.max=1) Create a resource pool for the processor set # poolcfg -c create pool email-pool Link the pool to the processor set # poolcfg -c associate pool email-pool (pset email-pset) Set an objective (if including a range of processors (i.e. min max)# poolcfg -c modify pset email-pool (string pset.poold.objectives="wt-load")Activate the conguration# pooladm -c Copyright 2009 Peter Baer Galvin - All Rights Reserved 84Saturday, May 2, 2009 85. Zones and DRPs (cont)Check the cong # pooladmsystem my_systemstring system.commentint system.version 1boolean system.bind-default trueint system.poold.pid 638pool email-pool int pool.sys_id 1 boolean pool.active true boolean pool.default false int pool.importance 1 string pool.comment pset emailpool pool_default int pool.sys_id 0 boolean pool.active true boolean pool.default true int pool.importance 1 string pool.comment pset pset_defaultpset email-pset int pset.sys_id 1 boolean pset.default false uint pset.min 1 uint pset.max 1 string pset.units population uint pset.load 0 uint pset.size 1 string pset.comment cpuint cpu.sys_id 0string cpu.commentstring cpu.status on-line Copyright 2009 Peter Baer Galvin - All Rights Reserved 85Saturday, May 2, 2009 86. Zones and DRPs (cont)Check the cong pset pset_default int pset.sys_id -1 boolean pset.default true uint pset.min 1 uint pset.max 65536 string pset.units population uint pset.load 7 uint pset.size 7 string pset.commentcpu int cpu.sys_id 1 string cpu.comment string cpu.status on-linecpu int cpu.sys_id 3 string cpu.comment string cpu.status on-linecpu int cpu.sys_id 2 string cpu.comment string cpu.status on-lineCopyright 2009 Peter Baer Galvin - All Rights Reserved 86Saturday, May 2, 2009 87. DRPsNote that you can give ranges of CPUs tobe used in DRPsIf you do be sure to set an objective elsenothing will be dynamicNote that some software licenses allowlicensing of the app for only those CPUs inthe DRP that the zone is attached to (i.e.only pay for your DRP CPUs, not allCPUs)(!) Copyright 2009 Peter Baer Galvin - All Rights Reserved 87Saturday, May 2, 2009 88. Zones and DRPs (cont)Now enable FSS, make it default for pool_default# poolcfg -c modify pool pool_default (string pool.scheduler="FSS")Create an instance of the conguration# pooladm -cMove all the processes in the default pool and its associated zones under the FSS.# priocntl -s -c FSS -i class TS# priocntl -s -c FSS -i pid 1Now have the zones use the DRPs# zonecfg z email-zonezonecfg:email-zone> set pool=email-pool# zonecfg z Web1-zonezonecfg: Web1-zone> set pool=pool_defaultzonecfg:Web1-zone> add rctlzonecfg:Web1-zone:rctl> set name=zone.cpu-shareszonecfg:Web1-zone:rctl> add value (priv=privileged,limit=3,action=none)zonecfg:Web1-zone:rctl> end# zonecfg -z Web2-zonezonecfg:Web2-zone> set pool=pool_defaultzonecfg:Web2-zone> add rctlzonecfg:Web2-zone:rctl> set name=zone.cpu-shareszonecfg:Web2-zone:rctl> add value (priv=privileged,limit=2,action=none)zonecfg:Web2-zone:rtcl> endCopyright 2009 Peter Baer Galvin - All Rights Reserved 88Saturday, May 2, 2009 89. Zones, Resources, and S10 8/07Much simpler now if you just want a zone to have dedicatedCPUs, memory limits(From http://blogs.sun.com/jerrysblog/feed/entries/atom?cat=%2FSolaris)zonecfg:my-zone> set scheduling-class=FSSzonecfg:my-zone> add dedicated-cpuzonecfg:my-zone:dedicated-cpu> set ncpus=1-4zonecfg:my-zone:dedicated-cpu> set importance=10zonecfg:my-zone:dedicated-cpu> endzonecfg:my-zone> add capped-memoryzonecfg:my-zone:capped-memory> set physical=50mzonecfg:my-zone:capped-memory> set swap=128mzonecfg:my-zone:capped-memory> set locked=10mzonecfg:my-zone:capped-memory> endYou have to enable poold via svcadm if importanceusedStill use dispadmin to set system-wide schedulingCopyright 2009 Peter Baer Galvin - All Rights Reserved 89Saturday, May 2, 2009 90. Zones, Resources, and S10 8/07 (cont)Can use zonecfg for the global zone to persistentlyset resource management settings in globalNow can set other zone-wide resource limits easily zone.cpu-shares zone.max-locked-memory (locked property of the capped-memoryresource is preferred) zone.max-lwps zone.max-msg-ids zone.max-sem-ids zone.max-shm-ids zone.max-shm-memory zone.max-swap (The swap property of the capped-memory resourceis the preferred way to set this control)Copyright 2009 Peter Baer Galvin - All Rights Reserved 90Saturday, May 2, 2009 91. Zones and Networking S10 8/07Can now create exclusive-IP zones (i.e. dedicate an HBA port to a zone) known asIP InstancesNeed this if you want advanced networking features in a zone (rewalls, snooping,DHCP client, trafc shaping)Each zone get its own IP stack (and soon xVM will too)zonecfg:my-zone>set ip-type=exclusivezonecfg:my-zone> add netzonecfg:my-zone:net> set physical=e1000g1zonecfg:my-zone:net> endNow the zone can set its own IP address et al, can do IPMP within a zonezonecfg set physical= to one of the interfaces in an IPMP groupProject Crossbow will allow virtual NICs to be IP instance entity (no longer tying upEthernet port)Limited to Ethernet devices that use GLDv3 drivers (dladm show-link not reportinglegacy) Copyright 2009 Peter Baer Galvin - All Rights Reserved 91Saturday, May 2, 2009 92. Zones, Resources and 5/08CPU Caps Can limit the aggregated amount of CPU that a containers CPUs canaccumulateAlthough it is possible to use prctl(1M) command to manage CPU caps, the capctlPerl script that simplies it# capctl * -Pproj: Specify project id* -ppid: Specify pid* -Zzone: Specify zone name* -nname: Specify resource name* -vvalue: Specify resource valueFor example, to set a cap for project foo to 50% you can say:# capctl -P foo -v 50To change the cap to 80%:# capctl -P foo -v 80To see the cap value:# capctl -P fooTo remove the cap:# capctl -P foo -v 0Copyright 2009 Peter Baer Galvin - All Rights Reserved92Saturday, May 2, 2009 93. prctl vs zonecfgprctl can read resource settings in theglobal or child zones Not persistent for setting variables Cant set variables in the child zonezonecfg is persistent, but only runs inglobal zone Copyright 2009 Peter Baer Galvin - All Rights Reserved 93Saturday, May 2, 2009 94. Zone IssuesZone cannot reside on NFSBut zone can be NFS clientEach zone normally has a sparse installation of apackage, if package is from inherit-package-dir directorytreeBy default, a package installed in global zone is installed inall existing non-global zonesUnless the pkgadd G or Z options are usedSee also SUNW_PKG_ALLZONES and SUNW_PKG_HOLLOWpackage parametersPatches installed in global zone is installed in all non-globalzonesIf any zone does not match patch dependencies, patch notinstalled Copyright 2009 Peter Baer Galvin - All Rights Reserved 94Saturday, May 2, 2009 95. Zone issues - contUpgrading the global zone to a new Solaris releaseupgrades the non-global zones but depends on whichupgrade method is used (hint - use live upgrade)Best practice is to keep packages and patches syncedbetween global and all non-global zonesWatch out for giving users root in a zone couldviolate policy or regulationsFlash Archive (ar) can be used to capture systemcontaining zones and clone it, but only if zones arehalted.Details at http://www.opensolaris.org/os/community/zones/faq/flar_zonesCopyright 2009 Peter Baer Galvin - All Rights Reserved 95Saturday, May 2, 2009 96. Zones and Packages# pkgadd -d screen*The following packages are available:1 SMCscreenscreen (intel) 4.0.2Select package(s) you wish to process (or all to processall packages). (default: all) [?,??,q]:## Not processing zone : the zone is not running and cannot be booted## Booting non-running zone into administrative state## waiting for zone to enter single user mode...## Verifying package dependencies in zone ## Restoring state of global zone ## Booting non-running zone into administrative state## waiting for zone to enter single user mode.... . .## Booting non-running zone into administrative state## waiting for zone to enter single user mode...## waiting for zone to enter single user mode...## Installing package in zone Copyright 2009 Peter Baer Galvin - All Rights Reserved96Saturday, May 2, 2009 97. Sparse Zones vs. Whole Root ZonesWhen should you use sparse, when should you usewhole rootCheck per-application support and/or requirementssparse zones dont allow writes into /, /usr, etc by default, some appsdont like thatCan intermix sparse and whole-root on the same systemMake a sparse root into a whole root# zonecfg create -bIn the future, likely that the world will use whole rootzones and ZFS cloningBut zone roots on ZFS not supported until U6because not upgradeable Copyright 2009 Peter Baer Galvin - All Rights Reserved 97Saturday, May 2, 2009 98. Upgrading a System Containing ContainersSupported methods vary, depending onOS release being upgraded fromGenerally liveupgrade is best, but manydetails to considerWell documented at http://docs.sun.com/app/docs/doc/820-4041/gdzlc?a=view Copyright 2009 Peter Baer Galvin - All Rights Reserved 98Saturday, May 2, 2009 99. Zone Best PracticesNote that global zone root can copy les directly into zones via theirzonepath directoryConsider building at least one container per system Put all users and apps in there Fast to copy for testing Fast reboot Put it on shared storage for future attach / detach But watch out for limitsdtraceapp support in a zone Surprisingly, a global-zone mount within the zone le system is immediately seen in the zoneCopyright 2009 Peter Baer Galvin - All Rights Reserved 99Saturday, May 2, 2009 100. Zone Best Practices (2)Use zonecfg export to save each zonescong settings - store on a different systemFor every zone created, in its virgin state,create a clone of it and store it on adifferent systemPut zones on ZFS for best feature setConsider conguring child zones to sendsyslog output to central syslog serverCopyright 2009 Peter Baer Galvin - All Rights Reserved 100Saturday, May 2, 2009 101. Zones and /etc/systemFor variables no longer in /etc/system they can be set via the rctladm command,but only per project. This example is from the Sun installation guide for Weblogicon Solaris 10Modify /etc/project in each zone the app will run in to contain the followingadditions to the resource controls for user.root (assuming the application will runas root): bash-3.00# cat /etc/project system:0:::: user.root:1:::: process.max-file-descriptor=(privileged,1024,deny); process.max-sem-ops=(privileged,512,deny); process.max-sem-nsems=(privileged,512,deny); project.max-sem-ids=(privileged,1024,deny); project.max-shm-ids=(privileged,1024,deny); project.max-shm-memory=(privileged,4294967296,deny) noproject:2:::: default:3:::: group.staff:10::::Copyright 2009 Peter Baer Galvin - All Rights Reserved101Saturday, May 2, 2009 102. Zones and /etc/system (cont)Note that /etc/project is read at loginAlso to enable warnings via syslog if the resource limitsare approached execute the following commands oncein each zone the app will run in (they update the /etc/rctladm.conf le)Do this in the global zone, not persistent so script it: #rctladm-e syslog process.max-file-descriptor #rctladm-e syslog process.max-sem-ops #rctladm-e syslog process.max-sem-nsems #rctladm-e syslog process.max-sem-ids #rctladm-e syslog process.max-shm-ids #rctladm-e syslog process.max-shm-memoryCopyright 2009 Peter Baer Galvin - All Rights Reserved 102Saturday, May 2, 2009 103. Branded ZonesShipped in S10 8/07Allows native binary execution of bins from otheroperating systemsCentos rstInstall a brandz zone, install the guest OS, then installbinaries (RPMs et al) and run themCurrently limited to centos and other 2.4-based distrosResult - can use DTrace to analyze Linux perf problemsSee man pages for brands(5), lx(5)Copyright 2009 Peter Baer Galvin - All Rights Reserved 103Saturday, May 2, 2009 104. brandzExample install given at http://milek.blogspot.com/2006/10/brandz-integrated-into-snv49.html # zonecfg -z linux linux: No such zone configuredUse create to begin configuring a new zone.zonecfg:linux> create -t SUNWlxzonecfg:linux> set zonepath=/home/zones/linuxzonecfg:linux> add netzonecfg:linux:net> set address=192.168.1.10/24zonecfg:linux:net> set physical=bge0zonecfg:linux:net> endzonecfg:linux> add attrzonecfg:linux:attr> set name="audio"zonecfg:linux:attr> set type=booleanzonecfg:linux:attr> set value=truezonecfg:linux:attr> endzonecfg:linux> exit Copyright 2009 Peter Baer Galvin - All Rights Reserved104Saturday, May 2, 2009 105. brandz (cont) # zoneadm -z linux install -d /mnt/iso/centos_fs_image.tar.bz2A ZFS file system has been created for this zone.Installing zone linux at root directory /home/zones/linuxfrom archive /mnt/iso/centos_fs_image.tar.bz2This process may take several minutes.Setting up the initial lx brand environment.System configuration modifications complete!Setting up the initial lx brand environment.System configuration modifications complete!Installation of zone linux completed successfully.Details saved to log file:"/home/zones/linux/root/var/log/linux.install.10064.log" Copyright 2009 Peter Baer Galvin - All Rights Reserved 105Saturday, May 2, 2009 106. Solaris 8 and 9 ContainersNow available as a commercial product ($) from SunUses brandzCapture a Solaris 8 or Solaris 9 system via Archiver (akaP2V)Updater Tool, processes Solaris 8 image and prepares itfor new, virtualized environmentCreate it as a container under S10Apps think they are on S8 or S9Sun guarantees compatibilitySPARC onlyCopyright 2009 Peter Baer Galvin - All Rights Reserved 106Saturday, May 2, 2009 107. Solaris 8 and 9 Containers - conthttp://www.sun.com/software/solaris/pdf/solaris8and9containers_datasheet.pdf# zonecfg -z zone8zonecfg:zone8> create -t SUNWsolaris8zonecfg:zone8> set zonepath = /export/home/zones/zone8zonecfg:zone8> add netzonecfg:zone8:net> set address = zonecfg:zone8:net> set physical = e1000g1zonecfg:zone8:net> endzonecfg:zone8> verifyzonecfg:zone8> commitzonecfg:zone8> exit# zoneadm -z zone8 install -a {-u|-p}Try for 90 days via http://www.sun.com/software/solaris/containers/getit.jspCopyright 2009 Peter Baer Galvin - All Rights Reserved 107Saturday, May 2, 2009 108. zonestatTool to monitor entire system performance, including per-zoneMore information that prstat -ZDownload from http://opensolaris.org/os/project/zonestat/# ./zonestat |--Pool--|Pset|-------Memory-----|Zonename| IT|Size|Used| RAM| Shm| Lkd| VM|------------------------------------------global0D 20.1 556M0.0 0.0 331Mzone1 0D 20.0 26M 0.0 0.0 24M==TOTAL= === 20.1 608M0.0 0.0 355M# ./zonestat -l |----Pool-----|------CPU-------|----------------Memory----------------| |---|--Size---|-----Pset-------|---RAM---|---Shm---|---Lkd---|---VM---|Zonename| IT| Max| Cur| Cap|Used|Shr|S%| Cap|Used| Cap|Used| Cap|Used| Cap|Used-------------------------------------------------------------------------------global0D2 0.0 0.1 5 83556M 18E 0.0 18E 0.0 18E 331Mzone1 0D2 0.0 0.0 1 16 26M 18E 0.0 18E 0.0 18E 24M==TOTAL= --- ---- 2 ----0.1 --- -- 3.1G 608M 3.1G0.0 3.0G0.0 4.0G 355M Copyright 2009 Peter Baer Galvin - All Rights Reserved108Saturday, May 2, 2009 109. Zone FuturesLive migrationImproved networking via project crossbow Not just ip-exclusive. Virtual network stack for each containerS10 containers?Copyright 2009 Peter Baer Galvin - All Rights Reserved 109Saturday, May 2, 2009 110. LabsCreate a container with resource management (your choice)What is your view of le systems?What le systems are yours, what are shared?What do the le systems look like from the global zone?Test the resource management if possibleWhat does your networking look like?What is your life like in a zone?How are zones different from domains? From vmware?What scheduler is in use in your zone?If fair share, how many shares does your zone have?Copyright 2009 Peter Baer Galvin - All Rights Reserved 110Saturday, May 2, 2009 111. Labs (cont)If you are not fair share scheduled, turn iton and enable shares for your containerClone the zoneDetach and attach the zone (to the samesystem if necessary) Copyright 2009 Peter Baer Galvin - All Rights Reserved 111Saturday, May 2, 2009 112. LDOMSCopyright 2009 Peter Baer Galvin - All Rights Reserved 112Saturday, May 2, 2009 113. LDOMs Logical domains Released April 07 Only on Niagara and future CMT chips (Niagara II, Rock) Like enterprise-system domains but within one chip Slice the chip into multiple LDOMs, each with its own OS root, boot independently, etNow can run multiple OSes on 1 SPARC chip Copyright 2009 Peter Baer Galvin - All Rights Reserved 113Saturday, May 2, 2009 114. Copyright 2009 Peter Baer Galvin - All Rights Reserved 114Saturday, May 2, 2009 115. LDOMs - DetailsCan create up to 1 LDOM per thread(!) Best practice seems to be max one LDOM per corei.e. 8 LDOMs on Niagara I and IINice intro bloghttp://blogs.sun.com/ash/entry/ultrasparc_t2_launched_todayAnd nice ash demo http://www.sun.com/servers/coolthreads/ldoms/Community cookbooks http://wikis.sun.com/display/SolarisLogicalDomains/ LDoms+Community+CookbookCopyright 2009 Peter Baer Galvin - All Rights Reserved 115Saturday, May 2, 2009 116. LDOMS Introduction and Hands-On-Training Peter Baer GalvinWith Thanks to: Tom Gendron Chief Technologist SPARC Systems Technical Specialist Corporate Technologies Sun Microsystems 116 1Saturday, May 2, 2009 117. Agenda Virtualization Comparisons Concepts of LDOMs Requirements of LDOMs Examples Best Practices117Saturday, May 2, 2009 118. Single application per server The Data Center Today Server sprawl is hard to manageClientAppAppMailService Average server Developer Server Server Server Database DatabaseApplication Management NETWORK utilization between Data Center 5 to 15 %OS Server Energy costscontinue to riseStorage118Saturday, May 2, 2009 119. A widely understood problem119Saturday, May 2, 2009 120. Virtualization: Who and WhyInformationWeek: Feb 12, 2007 http://www.informationweek.com/news/showArticle.jhtml?articleID=197004875120Saturday, May 2, 2009 121. Server VirtualizationHard Partitions Virtual MachinesOS VirtualizationResource Mgmt. App Identity File WebMailCalendarWebSunRayAppServer Database Server Server Server ServerServer Database ServerServerDatabase ServerAppOS Server Multiple OSs Single OS> Very High RAS > Live OS migration > Very scalable and low> Very scalable and lowcapabilityoverhead overhead> Very Scalable> Improved Utilization> Single OS to manage> Single OS to manage> Mature Technology> Ability to run different OS > Cleanly divides system and > Fine grained resource> Ability to run different OS versions and typesapplication administration managementversions> De-couples OS and HW> Fine grained resource> Complete Isolationversionsmanagement121Saturday, May 2, 2009 122. Para vs. Full VirtualizationPara-Virtualization Para-virtualization:File Server WebServer MailServerApp > OS ported to special architecture> Uses generic virtual device driversOS> More efficient since it is hypervisoraware Server > almost native performanceFull Virtualization Full virtualization: FileServerWeb ServerMail Server App > OS has no idea it is running virtualized> Must emulate real i/o devicesOS> Can be slow/need help from hardware Control Domain > May use traps, emulation or rewriting Server122Saturday, May 2, 2009 123. What is an LDOM? It is a virtual server Has its own console and OBP instance A configurable allocation of CPU, FPU, Disk, Memory and I/O components Runs a unique OS/patch image and configuration Has the capability to stop, start and reboot independently Utilizes a Hypervisor to facilitate LDOMs123Saturday, May 2, 2009 124. Requirements for LDOMs Sun T-Series server> T1/2000 T5x20 rack servers> T6100, T6120 blade> Any future CMT based server Up to date Firmware on service processor http://sunsolve.sun.com/handbook_pub/validateUser.do?target=index minimum Solaris 10 11/06 on T1/2000, T6100 minimum Solaris 10 08/07 T5x20, T6120 Ldom Manager Software 1.0.1 + patches124Saturday, May 2, 2009 125. Hypervisor A thin interface between the Hardware and Solaris The interface is called sun4v Solaris calls the sunv4 interface to use hardwarespecific functions It is very simple and is implemented in firmware It allows for the creation of ldoms It creates communication channels between ldoms125Saturday, May 2, 2009 126. Key LDOMs componentsPrimary/Controlldom1 ldom2 The HypervisorUnallocatedResources Solaris 10 08/07 Solaris 10 11/06Solaris 10 08/07+app+patches +app+patches The Control Domain ldmddrd vntsd CPUCpu /dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service Domain CPUCpu CPU/dev/lofi/1 Cpu Mem Memvdisk0 vdisk1 Crypto CPUCpu CPUCpu Mem vol1Mem vnet0 vnet0 Multiple GuestControl & ServiceGuest GuestprimaryCPUCpu CPUCrypto ldom1 ldom2CpuMemMem Crypto vsw0Domains CPUCpu CPUCpuvnet0 vnet1 Mem CryptoMem Cryptoprimary-vds0 Virtualised devices Hypervisor primary-vsw0 CPUCpu CPU Cpu CPUCpuCPUCpuCPUCpu CPU CPUCpuCpu Hardware MemShared CPU,MemMemMem Crypto Mem Crypto Mem CryptoMemory & IOMemMem CryptoIO Devices72GBPCI-E Network126Saturday, May 2, 2009 127. LDOMs types Different Ldom Types - Control Domain- Hosts the Logical Domain Manager (LDM) - Service Domains - Provides virtual services to other domains - I/0 Domains - Has direct access to physical devices - Guest Domains - Used to run user environments Control, Service and I/O domains can be combined or separate> One of the I/O domains must be the control domain127Saturday, May 2, 2009 128. Key LDOMs components Primary/Control ldom1 ldom2 The HypervisorUnallocatedResourcesSolaris 10 08/07Solaris 10 11/06Solaris 10 08/07+app+patches +app+patches The Control DomainldmddrdvntsdCPUCpu /dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service Domain CPUCpu CPUZFS FSCpu Mem Memvdisk0 vdisk1 Crypto CPUCpu CPUCpu Mem vol1Mem vnet0 vnet0 Multiple GuestControl & ServiceGuest GuestprimaryCPUCpu CPUCrypto ldom1 ldom2CpuMemMem Crypto vsw0Domains CPUCpu CPUCpuvnet0 vnet1 Mem CryptoMem Cryptoprimary-vds0 Virtualised devices Hypervisor primary-vsw0CPU CpuCPU Cpu CPUCpuCPUCpuCPUCpu CPU CPUCpuCpu HardwareMem Shared CPU,MemMem MemCrypto Mem Crypto Mem CryptoMemory & IOMemMem CryptoIO Devices72GBPCI-E Network128Saturday, May 2, 2009 129. Control Domain Creates and manages other LDOMs Runs the LDOM Manager software Allows monitoring and reconfiguration of domains Recommendation:> Make this Domain as secure as possible129Saturday, May 2, 2009 130. Key LDOMs components Primary/Control ldom1 ldom2 The HypervisorUnallocatedResourcesSolaris 10 08/07Solaris 10 11/06Solaris 10 08/07+app+patches +app+patches The Control DomainldmddrdvntsdCPUCpu /dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service Domain CPUCpu CPUZFS FSCpu Mem Memvdisk0 vdisk1 Crypto CPUCpu CPUCpu Mem vol1Mem vnet0 vnet0 Multiple GuestControl & ServiceGuest GuestprimaryCPUCpu CPUCrypto ldom1 ldom2CpuMemMem Crypto vsw0Domains CPUCpu CPUCpuvnet0 vnet1 Mem CryptoMem Cryptoprimary-vds0 Virtualised devices Hypervisor primary-vsw0CPU CpuCPU Cpu CPUCpuCPUCpuCPUCpu CPU CPUCpuCpu HardwareMem Shared CPU,MemMem MemCrypto Mem Crypto Mem CryptoMemory & IOMemMem CryptoIO Devices72GBPCI-E Network130Saturday, May 2, 2009 131. Service Domain Provides services to other domains virtual network switch virtual disk service virtual console service Multiple Service domains can exist with shared orsole access to system facilities Allows for IO load separation and redundancywithin domains deployed on a platform Often Control and Service Domains are one andthe same131Saturday, May 2, 2009 132. IO Domain IO Domain has direct access to physical input andoutput devices. The number of IO domains is hardware dependent> currently limited to 2> limited by PCI-E switch configuration One IO domain must also be the control domain132Saturday, May 2, 2009 133. Key LDOMs components Primary/Control ldom1 ldom2 The HypervisorUnallocatedResourcesSolaris 10 08/07Solaris 10 11/06Solaris 10 08/07+app+patches +app+patches The Control DomainldmddrdvntsdCPUCpu /dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service Domain CPUCpu CPUZFS FSCpu Mem Memvdisk0 vdisk1 Crypto CPUCpu CPUCpu Mem vol1Mem vnet0 vnet0Control & ServiceGuest Guest Multiple Guestprimaryvsw0 CPUCpu CPUCpuMemCryptoMem Crypto ldom1 ldom2Domains CPUCpu CPUCpuvnet0 vnet1 Mem CryptoMem Cryptoprimary-vds0 Virtualised devices Hypervisor primary-vsw0CPU CpuCPU Cpu CPUCpuCPUCpuCPUCpu CPU CPUCpuCpu HardwareMem Shared CPU,MemMem MemCrypto Mem Crypto Mem CryptoMemory & IOMemMem CryptoIO Devices72GBPCI-E Network133Saturday, May 2, 2009 134. Guest Domains Contain the targeted applications the LDOMs werecreated to service. Multiple Guest domains can exist> Constrained only by hardware limitations May use one or more Service domains to obtain IO> Various redundancy mechanisms can be used Can be independently powered and rebooted andwithout affecting other domains134Saturday, May 2, 2009 135. Key LDOMs components Primary/Control ldom1 ldom2 The HypervisorUnallocatedResourcesSolaris 10 08/07Solaris 10 11/06Solaris 10 08/07+app+patches +app+patches The Control DomainldmddrdvntsdCPUCpu /dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service Domain CPUCpu CPUZFS FSCpu Mem Memvdisk0 vdisk1 Crypto CPUCpu CPUCpu Mem vol1Mem vnet0 vnet0Control & ServiceGuest Guest Multiple Guestprimaryvsw0 CPUCpu CPUCpuMemCryptoMem Crypto ldom1 ldom2Domains CPUCpu CPUCpuvnet0 vnet1 Mem CryptoMem Cryptoprimary-vds0 Virtualised devices Hypervisor primary-vsw0CPU CpuCPU Cpu CPUCpuCPUCpuCPUCpu CPU CPUCpuCpu HardwareMem Shared CPU,MemMem MemCrypto Mem Crypto Mem CryptoMemory & IOMemMem CryptoIO Devices72GBPCI-E Network135Saturday, May 2, 2009 136. Virtual devices Virtual devices are hardware resources abstracted by the hypervisorand made available for use by the other domains Virtual devices are :> CPUs - VCPU> Memory -> Crypto cores - MAU> Network switches - VSW> NICs - VNET> Disk servers - VDSDEV> Disks - VDISK> Consoles - VCONS136Saturday, May 2, 2009 137. Example 1 Install Ldom Manager &Setting up the Control Domain137Saturday, May 2, 2009 138. Example 1 steps Update firmware to latest release Install Supported version of Solaris Install Logical Domain Manager (LDM) software Configure the control domain Save initial domain config Reboot Solaris138Saturday, May 2, 2009 139. A note on system interfaces Provide out-of-band management Two types (iLOM and ALOM) T1/2000 uses ALOM interface T5x20 uses iLOM iLOM CLI has a ALOM compatibility shell> ALOM shell used in the examples A web based interface available (SC = system controller, SP = system processor)> essentially the same thing.139Saturday, May 2, 2009 140. Web based iLOM interface140Saturday, May 2, 2009 141. ALOM compatibility shell login to SP as root/changeme -> create /SP/users/admin -> set /SP/users/admin role=Administrator -> set /SP/users/admin cli_mode=alom Creating user ... Enter new password: ******** Enter new password again: ******** Created /SP/users/admin exit login as admin141Saturday, May 2, 2009 142. Step 1Firmware verification and update142Saturday, May 2, 2009 143. System Identification and Update Check the Service Processor of your system for firmware levels using alom mode (showhost not available in bui)sc> showhost Sun System Firmware 7.0.1 2007/09/14 16:31Host flash versions: Hypervisor 1.5.1 2007/09/14 16:11 OBP 4.27.1 2007/09/14 15:17Check SC Firmware POST 4.27.1 2007/09/14 15:43version 7.0.1 Upgrade your system firmware if needed...> flashupdate command> sysfwdownload (via Solaris on platform)> BUI143Saturday, May 2, 2009 144. Firmware update examplesc> showkeyswitchKeyswitch is in the NORMAL position.sc> flashupdate -s 10.8.66.15 -f /incoming//Sun_System_Firmware-6_4_6-Sun_Fire_T2000.binUsername: tgendronPassword: ********SC Alert: System poweron is disabled.Update complete. Reset device to use new software.sc>sc> resetsctelnet and login back in once up.sc> showhostSun-Fire-T2000 System Firmware 6.5.5 2007/10/28 23:09144Saturday, May 2, 2009 145. Firmware update example 2Step 1:From Solaris running on T5120 with the SP to updateDownload the patch from Sun Solve 127580-05.zipStep 2:unzip and cd into 127580-05Step 3:run sysfwdownload [image].pkgStep 4:reboot solarissc> resetsc145Saturday, May 2, 2009 146. Installing LDOM manager software T5x20 requires Solaris 10 8/07 or greater T1/2000 requires Solaris 10 11/06 or greater +* 124921-02 at a minimum* 125043-01 at a minimum* 118833-36 at a minimum 11/06 is minimum for guests ldm 1.0.2 is current> includes Solaris Security Toolkit (optional)146Saturday, May 2, 2009 147. Install the LDM Software Unzip and install w/installation script Security of Control Domain is important> Recommend selecting the JASS secure configuration Once complete entire system is one LDOM LDOM software installed in /opt/SUNWldm# [cmt1/root] ldm listNAMESTATE FLAGS CONS VCPU MEMORY UTIL UPTIMEprimaryactive-n-cv SP64 8064M 0.0% 3h 19m[cmt1/root]All the system resource are in domain primary * Follow the Administration Guide to install required OS and patches147Saturday, May 2, 2009 148. Flag Definitions# ldm listNAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIMEprimary active -n-cvSP 32 32640M 0.1% 6d 20h 24m# -placeholder ccontrol domain ddelayed reconfiguration nnormal sstarting or stopping ttransition vvirtual I/O domain148Saturday, May 2, 2009 149. Example 1 Part 2Setting up the Control Domain149Saturday, May 2, 2009 150. On naming things... Choose LDOM component names carefully> Names are used to manage the devices> Bad choices can be very confusing later on...> Keep names short and specific... You need names for ...> Disk Servers, and disk device instances> Network Virtual Switches, and network device instances> Domains Service and device names are only known to theControl and Service domains Guest domains just see virtual devices.150Saturday, May 2, 2009 151. Control/Service Domain On our Primary Domain do the following ... In this example Control and Service are combined Primary > Control domain runs the LDM CPUCpuUnallocatedResources > Service domain has these services set up: Solaris 10 08/07 CPUCpuldmd CPUCpu CPUCpuMemMem drd Set up the basic services needed. vntsdCPUCpu CPUCpuMem MemMem Mem> vds - virtual disk service CPUCpu CPUCpuMemMem CPUCrypto CryptoCpu CPU> vcc - virtual console concentrator vcc0vds0Control & ServiceCpuMemMemprimaryCPU CPU Mem CryptoCpuCpu Mem Crypto> vsw - virtual network switch vsw0 CPUCpu CPU Mem CryptoCpu Mem CryptoCrypto Crypto The service names in this example are below: primary-vdsHypervisorprimary-vsw> primary-vds0CPU CpuCPU Cpu Hardware> primary-vcc0Mem Shared CPU, MemCrypto> primary-vsw0Memory & IOIO Devices Allocate resourcesPCI-E> CPU, Memory, Crypto, IO devices Network 72GB 72GB151Saturday, May 2, 2009 152. Control/Service Domain set-up (1) # Add services to the control domain # The mac address taken from a physical interface, e.g., e1000g0. ldm add-vds primary-vds0 primary ldm add-vcc port-range=5000-5100 primary-vcc0 primary ldm add-vsw mac-addr=0:14:4f:6a:9e:dc net-dev=e1000g0 primary-vsw0 primary # Activate the virtual network terminal server svcadm enable vntsd # Allocate resources to the control domain and save ldm set-mau 1 primary ldm set-vcpu 8 primary ldm set-memory 2G primary ldm add-spconfig my-initial # Reboot required to have the configuraiton take effect. init 6152Saturday, May 2, 2009 153. Crypto NoteNoteIf you have any cryptographic devices in the control domain, youcannot dynamically reconfigure CPUs. So if you are not usingcryptographic devices, set-mau to 0.153Saturday, May 2, 2009 154. Control/Service Domain set-up (2)# Verify the primary domain configurationldm list-domainNAMESTATE FLAGS CONS VCPU MEMORY UTIL UPTIMEprimaryactive-n-cv SP82G6.3% 6m# Enable Networkingifconfig vsw0 plumbifconfig e1000g0 down unplumbifconfig vsw0 10.8.66.208 netmask 255.255.255.0 broadcast + upifconfig -alo0: flags=2001000849 mtu 8232 index 1inet 127.0.0.1 netmask ff000000vsw0: flags=1000843 mtu 1500 index 3inet 10.8.66.208 netmask ffffff00 broadcast 10.8.66.255ether 0:14:4f:6a:9e:dc154Saturday, May 2, 2009 155. Ldom Service details155Saturday, May 2, 2009 156. Reconfiguration Dynamic reconfiguration> Resource changes that take effect w/out reboot of domain Delayed reconfiguration> Resource changes that take effect after a reboot Resource examples:> VCPU, Memory, IO devices Currently only VCPUs are dynamic156Saturday, May 2, 2009 157. Virtual Disk Server device (vds) DelayedReconfiguration VDS runs in a service domain Performs disk I/O on corresponding raw devices Device types can be> A entire physical disk or LUN (can be san based)> Single slice of disk or LUN> Disk image in a filesystem (e.g. ufs, zfs)> Disk volumes (zfs, svm, VxVM)> lofi devices NOT supported Virtual Disk Client (vdc drivers)> Requests standard block IO via the VDS> Classic client/server architecture157Saturday, May 2, 2009 158. Virtual Disk devices Physical LUNS perform best Disk image files efficient use of space ZFS snapshots and clones give rapid provisioning Network install not supported with> zfs volumes> single slice Network install requries> entire disk> disk image file158Saturday, May 2, 2009 159. Virtual Network Switch servicesDelayed (vswitch) Reconfiguration Implements a layer-2 network switch Connects virtual network devices to > To the physical network > or to each other (internal private network) vswitch not automatically used by service domain > must be plumbed159Saturday, May 2, 2009 160. Virtual Console ConcentratorDelayed (vcc)Reconfiguration Provides console access to LDoms Service domain VCC driver communicates with all guest consoledrivers over the Hypervisor> No changes required in guest console drivers (qcn) Makes each console available as a tty device on the Control/Service domain usage: telnet local host 160Saturday, May 2, 2009 161. Virtual Network Terminal Server Delayed daemon (vntsd)Reconfiguration VCC implemented by vntsd Runs in the Control/Service domain Aggregates the VCC tty devices and makes them available overnetwork sockets> Accessible once a domain is configured and bound> Attach prior to domain start to watch domain OBP boot sequence Only one user at a time can view a serial console Flexible support of port groups, IPs, port numbers etc> Not visible outside the Control/Service domain by default161Saturday, May 2, 2009 162. Example 2Setting up the Guest Domain162Saturday, May 2, 2009 163. Primary ldm1Guest DomainUnallocatedResourcesSolaris 10 11/06Solaris 10 08/07 ldmdCPUCpu CPUCpuMemMem+app+patchesdrd In the control domain:vntsd CPUCpu CPUCpuMemMem/dev/dsk/c0d0s0 CPUCpu CPU /dev/c0t1d0s0CpuMemMem ldm1-vdisk1 T2000ldm1- vol1Control & Service CPUCpu CPUCpuMemMemCrypto CryptoGuest ldm add-domain ldm1/dev/e1000g0 primary CPUCpu CPUCpuMem CryptoMem Cryptoldom1vsw0 CPUCpu CPUvnet0 ldm add-mau 1 ldm1CpuMem CryptoMem Crypto ldm add-vcpu 4 ldm1 primary-vds0Hypervisorprimary-vsw0 ldm add-memory 4G ldm1 CPU CpuCPU CpuHardware CPUCpu CPU Cpu Shared CPU, ldm add-vnet vnet0 primary-vsw0 ldm1 Mem Mem Crypto Memory & IO MemMem CryptoIO Devices ldm add-vdsdev /dev/dsk/c0t1d0s2 ldm1-vol1@primary-vds0 PCI-E ldm add-vdisk ldm1-vdisk1 ldm1-vol1@primary-vds0 ldm1 72GB 72GB Network ldm set-var auto-boot?=false ldm1 ldm set-var boot-device=vdisk ldm1 ldm set-var nvramrc-devalias vnet0 /virtual-devices@100/channel- devices@200/network@0 ldm1 ldm bind-domain ldm1 Watch the console of ldom1 using ...> telnet localhost 5000 ldm start-domain ldm1163Saturday, May 2, 2009 164. Disk Service Setup Primaryldm1 Establish a Virtual Disk Service primary-vds Solaris 10 11/06 Solaris 10 11/06+app+patches ldmd/dev/dsk/c0d0s0 Associate it with some form of vntsddrdmedia. ldm1-vdisk1 A real device or slice /dev/dsk//dev/c0t1d0s2 c0t1d0s0 or ldm1-vol1 or a disk image e.g. / Control & Service primary Guest ldom1 ldmzpool/ldg1 Create disk server deviceinstance to be exported to guest primary-vdsHypervisordomains ldm1-vol1@primary-vds CPUCpu CPU CpuHardwareCPU CpuCPU Cpu Shared CPU,ldm add-vdsdev /dev/dsk/c0t1d0s2MemMem Crypto Memory & IOMem MemCryptoldm1-vol1@primary-vds0 IO Devices ldm add-vdisk ldm1-vdisk1 ldm1-vol1@primary-vds0 ldm172GB 72GBPCI-E (The disk device name can vary - find itvia ok show-devs)Network164Saturday, May 2, 2009 165. Virtual Disk Client (vdc)Delayed Reconfiguration vdcs are the objects passed to OBP and the Operating System inguest systems Guest domain OBP and Solaris sees normal SCSI devices Domain administrators may setup devaliases or use raw vdiskdevices vdcs provide Guest domains with virtual disk devices (vdisks) viadevice instances from Virtual Disk Servers running in the ServiceDomains(s) A future release will provide virtualised access to DVD/CD-ROM inservice domains165Saturday, May 2, 2009 166. Network SetupPrimaryldom1 Establish a Virtual Network Switch Solaris 10 08/07 Solaris 10 11/06 Services ldmd+app+patches primary-vsw0 vntsddrd > Automatically associated with avsw device instance vsw0 Control & Service Guestvnet0 primaryprimary-ldom1 May or may not choose tovsw0 vnet0@ldm1 associate it with media. e1000g0 a real NIC primary-vswHypervisor or no NIC . in memoryCPU CpuCPU Cpu CPUCpuCPUCpu Create a network device instance Hardware to provide to guest domainsMem MemCrypto Shared CPU, Memory & IO MemMemCrypto vnet0@ldm1 IO Devices e1000g072GB PCI-E Network166Saturday, May 2, 2009 167. Virtual Network Device (vnet)Delayed Reconfiguration Implements an ethernet device in a domain> Communicates with other vnets or the outside world over vswitchdevices If the vSwitch is suitably configured, packets can be routed out ofthe server. vnet exports a GLDv3 interface> A simple virtual Ethernet NIC> Enumerates as a vnetx device> For domain-domain transfers, vnets connect directly.167Saturday, May 2, 2009 168. Memory Delayed Reconfiguration Memory is configured through the Control Domain Minimum allocatable chunk is 8kB> Minimum size is 12MB (for OBP)> Though most OS deployments will need > 512M If memory is added over time to a domain> Memory device bindings within a domain may appear to show thatmemory fragmentation is occuring> Not a problem, all handled in HW by the MMU> No performance penalty168Saturday, May 2, 2009 169. vCPUsImmediate Reconfiguration Each UltraSPARC T1 has up to 8 physical cores with 4 threads each> Each thread is considered a vCPU, so up to 32 vCPUs or Domains Each UltraSPARC T2 has up to 8 physical cores with 8 threads each> Each thread is considered a vCPU, so up to 64 vCPUs or Domains Maximum Granularity is 1 vCPU per domain vCPUs can only be allocated to one Domain at a time. Can be dynamically allocated with the Domain running,> Take care if removing a vcpu from a running domain, will there beenough compute power left in the domain ?169Saturday, May 2, 2009 170. Example 3Guest Domains and ZFS170Saturday, May 2, 2009 171. Using ZFS (1) setup zfs1. Remove the disk from the service domainldm stop-domain ldm1LDom ldm1 stoppedldm unbind-domain ldm1ldm remove-vdsdev ldm1-vol1@primary-vds02. Create a zpoolroot@cmt1 > zfs create mypool/ldomsroot@cmt1 > zfs create mypool/ldoms/ldm1root@cmt1 > cd /export/ldoms/ldm1root@cmt1 > lsroot@cmt1 > mkfile 12G `pwd`/rootdisk171Saturday, May 2, 2009 172. Using ZFS (2) setup guest domain3. Configure the guest domainroot@cmt1 > ldm add-domain ldm1root@cmt1 > ldm add-vcpu 8 ldm1root@cmt1 > ldm add-memory 1G ldm1root@cmt1 > ldm add-vnet vnet0 primary-vsw0 ldm1root@cmt1 > ldm add-vdsdev /export/ldoms/ldm1/rootdisk ldm1-vol1@primary-vds0root@cmt1 > ldm add-vdisk ldm1-vdisk1 ldm1-vol1@primary-vds0 ldm1root@cmt1 > ldm set-var auto-boot?=false ldm1root@cmt1 > ldm set-var boot-device=ldm1-vdisk1 ldm1root@cmt1 > ldm set-var nvramrc-devalias vnet0 /virtual-devices@100/channel-devices@200/network@0 ldm1172Saturday, May 2, 2009 173. Using ZFS (3) setup guest domain 4. Start the guest domain root@cmt1 > ldm bind-domain ldm1 root@cmt1 > ldm start-domain ldm1 LDom ldm1 started 5. Inspect the domain root@cmt1 > ldm list-domain NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cvSP 8 2G 0.7% 17h 12m ldm1 active -t--- 5000 8 1G 13% 7s telnet localhost 5000 {ok} boot vnet0 - install installation goes forward173Saturday, May 2, 2009 174. Provision the guest6. Set up for jumpstartDetermine the mac addressroot@cmt1 > ldm list-bindings ldm1[snip]NETWORKNAME SERVICEDEVICE MACvnet0 primary-vsw0@primary network@0 00:14:4f:f8:2a:c4PEERMACprimary-vsw0@primary 00:14:4f:46:41:b4telnet localhost 5000{0} ok bannerSPARC Enterprise T5120, No Keyboard[snip]Ethernet address 0:14:4f:fb:7:42, Host ID: 83fb0742.174Saturday, May 2, 2009 175. Provision the guest (2) {0} ok boot vnet0 - install Boot device: /virtual-devices@100/channel-devices@200/network@0ile and args: - install Requesting Internet Address for 0:14:4f:f8:2a:c4 SunOS Release 5.10 Version Generic_120011-14 64-bit ...How to breaktelnet> send brkDebugging requested; hardware watchdog suspended.c)ontinue, s)ync, r)eboot, h)alt? rResetting...{0} ok175Saturday, May 2, 2009 176. Guest Domain (zfs) login{0} ok bootBoot device: ldm1-vdisk1 File and args:SunOS Release 5.10 Version Generic_120011-14 64-bitCopyright 1983-2007 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.Hostname: ldm1ldm1 console login:176Saturday, May 2, 2009 177. Using ZFS (2) cloning domainsSnapshot and Clone the installed boot disktgendron@cmt1 > zfs listNAME USED AVAIL REFER MOUNTPOINTmypool 12.0G 54.9G 27.5K /exportmypool/ldoms 12.0G 54.9G 25.5K /export/ldomsmypool/ldoms/ldm1 12.0G 54.9G 12.0G /export/ldoms/ldm1root@cmt1 > zfs snapshot mypool/ldoms/ldm1@initialCreate the clonesroot@cmt1 > zfs snapshot mypool/ldoms/ldm1@initialroot@cmt1 > zfs clone mypool/ldoms/ldm1@initial mypool/ldoms/ldm2root@cmt1 > zfs clone mypool/ldoms/ldm1@initial mypool/ldoms/ldm3root@cmt1 > zfs clone mypool/ldoms/ldm1@initial mypool/ldoms/ldm4root@cmt1 > zfs clone mypool/ldoms/ldm1@initial mypool/ldoms/ldm5177Saturday, May 2, 2009 178. Using ZFS (2) Leverage the clones4. Create the new guest domains (should be easily to script this)ldm add-domain ldm2ldm add-vcpu 8 ldm2ldm add-memory 1G ldm2ldm add-vnet vnet0 primary-vsw0 ldm2ldm add-vdsdev /export/ldoms/ldm2/rootdisk ldm2-vol1@primary-vds0ldm add-vdisk ldm2-vdisk1 ldm2-vol1@primary-vds0 ldm2ldm set-var auto-boot?=false ldm2ldm set-var boot-device=vdisk ldm2ldm set-var nvramrc-devalias vnet0 /virtual-devices@100/channel-devices@200/network@0 ldm2ldm bind-domain ldm2ldm start-domain ldm2178Saturday, May 2, 2009 179. Boot the cloned ldom{0} ok bootBoot device: vdisk File and args:SunOS Release 5.10 Version Generic_120011-14 64-bitCopyright 1983-2007 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.WARNING: vnet0 has duplicate address 010.030.019.178 (in use by 00:14:4f:f8:2a:c4); disabledFeb 13 19:55:29 svc.startd[7]: svc:/network/physical:default:Method "/lib/svc/method/net-physical" failed with exit status 96.Feb 13 19:55:29 svc.startd[7]: network/physical:default misconfigured:transitioned to maintenance (see svcs -xv for details)Hostname: ldm1...179Saturday, May 2, 2009 180. Example 4Split Service Domains180Saturday, May 2, 2009 181. Sun Fire T2000 Block Diagram181Saturday, May 2, 2009 182. Split IO Example Setting up a second Service domain with split PCI busses...-bash-3.00# ldm list-bindings primaryName: primary...IO: pci@780 (bus_a)pci@7c0 (bus_b)...-bash-3.00# df // (/dev/dsk/c1t0d0s0 ):28233648 blocks 3450076 files-bash-3.00# ls -l /dev/dsk/c1t0d0s0lrwxrwxrwx 1 rootroot65 Apr 11 13:25 /dev/dsk/c1t0d0s0 -> ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0:a-bash-3.00# grep e1000g /etc/path_to_inst"/pci@780/pci@0/pci@1/network@0" 0 "e1000g""/pci@780/pci@0/pci@1/network@0,1" 1 "e1000g""/pci@7c0/pci@0/pci@2/network@0" 2 "e1000g""/pci@7c0/pci@0/pci@2/network@0,1" 3 "e1000g"-bash-3.00# ldm remove-io pci@780 primary..-bash-3.00# shutdown -i6 -y -g0..-bash-3.00# ldm add-io pci@780 second-svrc-dom-bash-3.00# ldm start second-srvc-dom-bash-3.00# ldm list-bindings..-bash-3.00#Check which PCI bus ports we own and are currently using and be sure to only give awayunused ones... i.e need to retain the Control Domain boot disk controller and network device...Providing a PCI bus to a Guest makes the selected Domain a Service domain, by definition access to physical IO = Service Domain.182Saturday, May 2, 2009 183. Sun Fire T5x20 Block Diagram16 x FB-DIMMs Disk Chassis1RU 2RU/8 SSI x8 FPGA10GbE 10GbE x4 PCI-ELSI x4Switch x8MPC885SAS linksx41068EILOM PLX 8533Service DVD x4 Processor PCI-Ex4x8 Switch USB2.0 PCI-Eto IDE PCI-E x1Switch PLX 8533tox810GbE2.0USBPLX 8517x4 x4x4 SerDes BCM8704USB 2.0 x4x4 Hub 10GbE2RU Only USB Intel IntelCu PHY2.0 BCMxxxx DualDual GbE GbEXFPFront Panel 10GbEFibre 0 1 2 3 Plugin USB Quad GbEPCI-E PCI-E PCI-E PCI-EPCI-E PCI-ESerial NetworkPOSIX Rear Panel 183 Connectors x16 x8x8x8 x8x8 Mgt Mgt Serial DB-9Saturday, May 2, 2009 184. MPxIO considerations MPxIO can be used in the Service/Control domain Very straightforward to configure with defaults...> Ensure you have two FC-AL HBAs in a single service domain attached to the the same SAN array> Check that you have two paths to the same SAN devices (ls /dev/dsk/)> Enable MPxIO by running the command stmsboot -e and rebooting the control/ service domain> Check that you now have only a single path to the SAN devices...184Saturday, May 2, 2009 185. IPMP considerations IPMP has several options for configurations > Refer to the Admininstration Guide for worked examples... > Options are Multipathing in the Service Domain or Multipathing in the Guest Domain185Saturday, May 2, 2009 186. Ldom 1.0.1Best Practice Guidence186Saturday, May 2, 2009 187. Ldom Best Practice (1) Control Domain> Runs LDM daemon processes> Must have adequate CPU and memory> Start w/ 1 core (4 or 8 threads) 1GB Memory> Make this domain as secure as possible187Saturday, May 2, 2009 188. Ldom Best Practice (2) I/O and service domains> Runs IO for other domains> Resources will be sized based on IO load> Start w/ 1 core and 1GB memory> 4GB of memory if zfs used for virtual disks images> Add complete cores as heavier I/O loads188Saturday, May 2, 2009 189. Ldom Best Practice (3) Core/Thread Affinity> Core resources are shared by threads> E.g. L1 cache and MAU, FPU Best to avoid allocating the threads of a core toseparate domains Create larger Ldoms first using complete cores Smaller domains last189Saturday, May 2, 2009 190. Ldom Best Practice (4)Delayed Reconfiguration Cypto Units Each T1/T2 physical CPU Core has a Crypto Unit> 8 in total on a 8 core system> referred to as (MAU) Crypto cores can only be allocated to domains that have at leastone vcpu(thread) on the same physical Core as the crypto unit Crypto cores cannot be shared, they are owned by exactly one (orno) Domain Probably best to allocate all four/eight threads on a Core to adomain that wants to use the Crypto core190Saturday, May 2, 2009 191. More on Crypto Units For example we define three domains in order of LDOM1 LDOM2 LDOM3LDOM1 then LDOM2 then LDOM 3... LDOM1 has 3 threads (vCPUs) on Core 0> Only has access to MAU0 since it only has threadson Core 0T1 Core 0 T1 Core 1 T1 Core 2 LDOM2 has 6 vCPUs spread across Cores 0, 1 & 2> Potentially has access to MAUs 0,1 & 2> BUT.. LDOM1 already binds MAU0> So only can take MAU1 and MAU2MAU MAUMAU LDOM3 has 3 vCPUs on Core 201 2> But cant access any MAUs since LDOM2 has already taken MAU2 Adding and removing vCPUs can cause access to previously accessible MAUs to belost, currently you cant elect specific vCPUs, framework does that itself When MAUs are allocated to Domains, vCPUs become delayed reconfigurationproperties in those domains191Saturday, May 2, 2009 192. Ldom Best Practice (5) Plan your LDOM configuration carefully, reconfiguration may become awkward Use easy to understand names> Try not to overload vds, vsw, ldom, vdisk,vnic etc... Use MPxIO or VxVM, VxFS, Sun Cluster on service domains (only VxFS in Guests) for resilient storage devices Use IPMP on Guest or Service Domains for resilient network connections192Saturday, May 2, 2009 193. Ldom Best Practice (6) For hi-speed inter-domain comms use device-less/in-memory VSW configs For high disk performance, allocate a whole real device via a dedicated, properlysized Virtual Disk Server and Service domain Look at the server architecture when configuring devices to ensure you get thebandwidth you expect For critical applications consider hot/warm standby domains across multiplephysical servers, never rely on multiple instances within a single server.193Saturday, May 2, 2009 194. LDOMs v1.0.1 Notes All domains can be Stopped and Started independently > Beware, Guest domains attempting to perform IO using a rebooting Service domain will stall until the Service domain returns. LDOM SNMP MIB available now with traps and requests to the LDOM framework MAC address on banner different from what is raprd for jumpstart Only vcpus can be dynamically reallocated > BUT... if the domain has crypto cores this becomes a delayed reconfiguration > You cannot choose which vCPUs are allocated to a domain By default the Control/Service domain cannot network with Guest domains > Plumb the vSwitch vsw device to enable communications > Give the vsw device the e1000g devices MAC address Check you have the latest versions of the documents, Software & Firmware194Saturday, May 2, 2009 195. SVM, VxVM, ZFS Volume managers SVM, VxVM and ZFS volumes can be exported from a Service Domain to Guestdomains and appear as virtual disks to the Guest Domains> Always appear as a disk with only one s0 slice> Cant be used as Solaris Install targets...yet, just use for data storage Can export a disk image file placed in one of these volumes as a full disk image toGuest domains> Allows use of the disk as Solaris Install Target> Doing this with ZFS allows very efficient re-use of images using ZFSSnapshotting and Cloning and Compression> Invisibly bestows the benefits of the underlying Volume manager on the disksavailable to the Guest domains> Using SVM allows either Guest or Service domain to access the disk image,allowing for off-linemaintenance of the guest domain filesystems (only one at a time can mount the filesystem) VxVM can only be used in the Service domain, not Guest domains195Saturday, May 2, 2009 196. Solaris Cluster 3.2 Support Sun Cluster 3.2 is now supported in IO Domains> i.e domains with real physical devices, PCI busses or NIU devices Please check the web site here for more infom on deployment scenarios> http://blogs.sun.com/SC/entry/announcing_solaris_cluster_support_in196Saturday, May 2, 2009 197. Logical Domains (LDoms) Roadmap LDoms 1.0> LDoms 1.0.1 - CURRENT> Niagara support Niagara2 support> Up to 32 LDOMs per system, guest domain I/O domain reboot supportmay be rebooted independently Control domain minimization> Virtualized console, ethernet, disk & SNMP MIBcryptographic acceleration Web management tool> Live re-configuration of virtual CPUs (freeware/unsupported)> FMA diagnosis for each domain> Control domain hardening* Requiring new Solaris 10 update197Saturday, May 2, 2009 198. References for further information http://www.sun.com/ldoms Sun Blueprints relating to LDOMs http://www.sun.com/blueprints/0207/820-0832.html http://www.sun.com/blueprints/0807/820-3023.html SDLC Release of LDOMs http://www.sun.com/download/products.xml?id=46e5ba66 Official Documentation for the SDLC release http://www.sun.com/servers/coolthreads/ldoms/get.jsp LDOMs Blogs http://blogs.sun.com/hlsu/entry/logincal_domains_1_0_1 OpenSolaris LDOMs community http://www.opensolaris.org/os/community/ldoms/198Saturday, May 2, 2009 199. LDOMS IntroductionandHands-On-Training Peter Baer GalvinWith Thanks to: Tom Gendron Chief Technologist SPARC Systems Technical Specialist Corporate Technologies Sun Microsystems 199 1Saturday, May 2, 2009 200. Agenda Virtualization Comparisons Concepts of LDOMs Requirements of LDOMs Examples Best Practices200Saturday, May 2, 2009 201. Single applicationper serverThe Data Center TodayServer sprawl is hardto manageClient AppAppMailService Average server Server Server Server Database DatabaseDeveloper Application utilization between ManagementNETWORK Data Center 5 to 15 %OSServer Energy costs continueto riseStorage201Saturday, May 2, 2009 202. A widely understood problem202Saturday, May 2, 2009 203. Virtualization: Who and WhyInformationWeek: Feb 12, 2007 http://www.informationweek.com/news/showArticle.jhtml?articleID=197004875203Saturday, May 2, 2009 204. Server VirtualizationHard Partitions Virtual Machines OS VirtualizationResource Mgmt. App IdentityFile Web Mail CalendarWeb SunRay AppServer Database ServerServer ServerServer Server Database Server ServerDatabaseServerApp OSServer Multiple OSs Single OS> Very High RAS > Live OS migration capability> Very scalable and low overhead > Very scalable and low overhead> Very Scalable > Improved Utilization> Single OS to manage> Single OS to manage> Mature Technology > Ability to run different OS > Cleanly divides system and > Fine grained resourceversions and typesapplication administration management> Ability to run different OSversions> De-couples OS and HW> Fine grained resourceversionsmanagement> Complete Isolation204Saturday, May 2, 2009 205. Para vs. Full VirtualizationPara-Virtualization Para-virtualization:App > OS ported to special architecture FileWeb MailServerServerServer> Uses generic virtual device driversOS> More efficient since it is hypervisor aware> almost native performance Server Full virtualization:Full Virtualization > OS has no idea it is running virtualized FileServerWeb ServerMail Server App > Must emulate real i/o devices> Can be slow/need help from hardwareOS> May use traps, emulation or rewriting Control Domain Server205Saturday, May 2, 2009 206. What is an LDOM? It is a virtual server Has its own console and OBP instance A configurable allocation of CPU, FPU, Disk, Memory and I/O components Runs a unique OS/patch image and configuration Has the capability to stop, start and reboot independently Utilizes a Hypervisor to facilitate LDOMs206Saturday, May 2, 2009 207. Requirements for LDOMs Sun T-Series server> T1/2000 T5x20 rack servers> T6100, T6120 blade> Any future CMT based server Up to date Firmware on service processor http://sunsolve.sun.com/handbook_pub/validateUser.do?target=index minimum Solaris 10 11/06 on T1/2000, T6100 minimum Solaris 10 08/07 T5x20, T6120 Ldom Manager Software 1.0.1 + patches207Saturday, May 2, 2009 208. Hypervisor A thin interface between the Hardware and Solaris The interface is called sun4v Solaris calls the sunv4 interface to use hardwarespecific functions It is very simple and is implemented in firmware It allows for the creation of ldoms It creates communication channels between ldoms208Saturday, May 2, 2009 209. Key LDOMs components Primary/Control ldom1 ldom2 The HypervisorUnallocatedResourcesSolaris 10 11/06Solaris 10 08/07Solaris 10 08/07+app+patches +app+patches The Control Domainldmd vntsddrd CPUCpu/dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service Domain/dev/lofi/1CPUCpu CPU CPUCpuCpu CPUMem Mem Cryptovdisk0 vdisk1Cpu Mem vol1Multiple Guest MemControl & Servicevnet0GuestGuestvnet0 primary CPUCpu CPUCpu Mem CryptoCryptoldom1ldom2 MemDomains vsw0 CPUCpu CPU vnet0vnet1Cpu MemCrypto MemCrypto Virtualised devicesprimary-vds0Hypervisorprimary-vsw0 CPUCpu CPU CpuCPU CpuCPU CpuCPU CpuCPU CPU Cpu Cpu HardwareMem Shared CPU,MemMem MemCryptoMem Crypto MemCryptoMemory & IO Mem Mem Crypto IO Devices72GB PCI-ENetwork209Saturday, May 2, 2009 210. LDOMs types Different Ldom Types - Control Domain - Hosts the Logical Domain Manager (LDM) - Service Domains- Provides virtual services to other domains - I/0 Domains- Has direct access to physical devices - Guest Domains- Used to run user environments Control, Service and I/O domains can be combined or separate> One of the I/O domains must be the control domain210Saturday, May 2, 2009 211. Key LDOMs components Primary/Control ldom1 ldom2 The HypervisorUnallocatedResourcesSolaris 10 11/06Solaris 10 08/07 Solaris 10 08/07+app+patches +app+patches The Control Domainldmd vntsddrd CPUCpu/dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service DomainZFS FS CPUCpu CPU CPUCpuCpu CPUMem Mem Cryptovdisk0 vdisk1Cpu Memvol1Multiple Guest Mem Control & Service vnet0GuestGuestvnet0primaryCPUCpu CPUCpu Mem CryptoCryptoldom1ldom2 MemDomainsvsw0 CPUCpu CPU vnet0vnet1Cpu MemCrypto MemCrypto Virtualised devicesprimary-vds0Hypervisorprimary-vsw0 CPUCpuCPUCpu CPU CpuCPU CpuCPU CpuCPU CPU Cpu Cpu Hardware MemShared CPU,MemMemMemCrypto Mem Crypto MemCryptoMemory & IO Mem Mem Crypto IO Devices72GB PCI-ENetwork211Saturday, May 2, 2009 212. Control Domain Creates and manages other LDOMs Runs the LDOM Manager software Allows monitoring and reconfiguration of domains Recommendation:> Make this Domain as secure as possible212Saturday, May 2, 2009 213. Key LDOMs components Primary/Control ldom1 ldom2 The HypervisorUnallocatedResourcesSolaris 10 11/06Solaris 10 08/07 Solaris 10 08/07+app+patches +app+patches The Control Domainldmd vntsddrd CPUCpu/dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service DomainZFS FS CPUCpu CPU CPUCpuCpu CPUMem Mem Cryptovdisk0 vdisk1Cpu Memvol1Multiple Guest Mem Control & Service vnet0GuestGuestvnet0primaryCPUCpu CPUCpu Mem CryptoCryptoldom1ldom2 MemDomainsvsw0 CPUCpu CPU vnet0vnet1Cpu MemCrypto MemCrypto Virtualised devicesprimary-vds0Hypervisorprimary-vsw0 CPUCpuCPUCpu CPU CpuCPU CpuCPU CpuCPU CPU Cpu Cpu Hardware MemShared CPU,MemMemMemCrypto Mem Crypto MemCryptoMemory & IO Mem Mem Crypto IO Devices72GB PCI-ENetwork213Saturday, May 2, 2009 214. Service Domain Provides services to other domains virtual network switch virtual disk service virtual console service Multiple Service domains can exist with shared or soleaccess to system facilities Allows for IO load separation and redundancy withindomains deployed on a platform Often Control and Service Domains are one and thesame214Saturday, May 2, 2009 215. IO Domain IO Domain has direct access to physical input andoutput devices. The number of IO domains is hardware dependent> currently limited to 2> limited by PCI-E switch configuration One IO domain must also be the control domain215Saturday, May 2, 2009 216. Key LDOMs components Primary/Control ldom1 ldom2 The HypervisorUnallocatedResourcesSolaris 10 11/06Solaris 10 08/07 Solaris 10 08/07+app+patches +app+patches The Control Domainldmd vntsddrd CPUCpu/dev/dsk/c0d0s0 /dev/dsk/c0d0s0 The Service DomainZFS FS CPUCpu CPU CPUCpuCpu CPUMem Mem Cryptovdisk0 vdisk1Cpu Memvol1Multiple Guest Mem Control & Service vnet0GuestGuestvnet0primaryCPUCpu CPU Crypto ldom1ldom2Cpu Mem MemCryptovsw0DomainsCPUCpu CPUCpu Mem Mem CryptoCrypto vnet0vnet1 Virtualised devicesprimary-vds0Hypervisorprimary-vsw0 CPUCpuCPUCpu CPU CpuCPU CpuCPU CpuCPU CPU Cpu Cpu Hardware MemShared CPU,MemMemMemCrypto Mem Crypto MemCryptoMemory & IO Mem Mem Crypto IO Devices72GB PCI-ENetwork216Saturday, May 2, 2009 217. Guest Domains Contain the targeted applications the LDOMs werecreated to service. Multiple Guest domains can exist> Constrained only by hardware limitations May use one or more Service domains to obtain IO> Various redundancy mechanisms can be used Can be independently powered and rebooted andwithout affecting other domains217Saturday, May 2, 2009 218. Key LDOMs components Primary/Control ldom1ldom2 The HypervisorUnallocatedResourcesSolaris 10 11/06 Solaris 10 08/07 Solaris 10 08/07+app+patches+app+patches The Control Domainldmd vntsddrd CPUCpu/dev/dsk/c0d0s0/dev/dsk/c0d0s0 The Service DomainZFS FS CPUCpu CPU CPUCpuCpu CPUMem Mem Cryptovdisk0vdisk1Cpu Memvol1Multiple Guest Mem Control & Service vnet0Guest Guestvnet0primaryCPUCpu CPU Crypto ldom1 ldom2Cpu Mem MemCryptovsw0DomainsCPUCpu CPUCpu Mem Mem CryptoCrypto vnet0vnet1 Virtualised devicesprimary-vds0Hypervisorprimary-vsw0 CPUCpuCPUCpu CPU Cpu CPUCpuCPUCpu CPU CPUCpuCpu Hardware MemShared CPU,Mem MemMemCrypto MemCrypto MemCryptoMemory & IOMemMem Crypto IO Devices72GBPCI-ENetwork218Saturday, May 2, 2009 219. Virtual devices Virtual devices are hardware resources abstracted by the hypervisor andmade available for use by the other domains Virtual devices are :> CPUs - VCPU> Memory -> Crypto cores - MAU> Network switches - VSW> NICs - VNET> Disk servers - VDSDEV> Disks - VDISK> Consoles - VCONS219Saturday, May 2, 2009 220. Example 1 Install Ldom Manager &Setting up the Control Domain220Saturday, May 2, 2009 221. Example 1 steps Update firmware to latest release Install Supported version of Solaris Install Logical Domain Manager (LDM) software Configure the control domain Save initial domain config Reboot Solaris221Saturday, May 2, 2009 222. A note on system interfaces Provide out-of-band management Two types (iLOM and ALOM) T1/2000 uses ALOM interface T5x20 uses iLOM iLOM CLI has a ALOM compatibility shell> ALOM shell used in the examples A web based interface available (SC = system controller, SP = system processor)> essentially the same thing.222Saturday, May 2, 2009 223. Web based iLOM interface223Saturday, May 2, 2009 224. ALOM compatibility shell login to SP as root/changeme -> create /SP/users/admin -> set /SP/users/admin role=Administrator -> set /SP/users/admin cli_mode=alom Creating user ... Enter new password: ******** Enter new password again: ******** Created /SP/users/admin exit login as admin224Saturday, May 2, 2009 225. Step 1Firmware verification and update225Saturday, May 2, 2009 226. System Identification and Update Check the Service Processor of your system for firmware levels using alom mode (showhost not available in bui) sc> showhostSun System Firmware 7.0.1 2007/09/14 16:31 Host flash versions:Hypervisor 1.5.1 2007/09/14 16:11OBP 4.27.1 2007/09/14 15:17 Check SC FirmwarePOST 4.27.1 2007/09/14 15:43version 7.0.1 Upgrade your system firmware if needed...> flashupdate command> sysfwdownload (via Solaris on platform)> BUI226Saturday, May 2, 2009 227. Firmware update examplesc> showkeyswitchKeyswitch is in the NORMAL position.sc> flashupdate -s 10.8.66.15 -f /incoming//Sun_System_Firmware-6_4_6-Sun_Fire_T2000.binUsername: tgendronPassword: ********SC Alert: System poweron is disabled.Update complete. Reset device to use new software.sc>sc> resetsctelnet and login back in once up.sc> showhostSun-Fire-T2000 System Firmware 6.5.5 2007/10/28 23:09227Saturday, May 2, 2009 228. Firmware update example 2Step 1:From Solaris running on T5120 with the SP to updateDownload the patch from Sun Solve 127580-05.zipStep 2:unzip and cd into 127580-05Step 3:run sysfwdownload [image].pkgStep 4:reboot solarissc> resetsc228Saturday, May 2, 2009 229. Installing LDOM manager software T5x20 requires Solaris 10 8/07 or greater T1/2000 requires Solaris 10 11/06 or greater + * 124921-02 at a minimum * 125043-01 at a minimum * 118833-36 at a minimum 11/06 is minimum for guests ldm 1.0.2 is current> includes Solaris Security Toolkit (optional)229Saturday, May 2, 2009 230. Install the LDM Software Unzip and install w/installation script Security of Control Domain is important> Recommend selecting the JASS secure configuration Once complete entire system is one LDOM LDOM software installed in /opt/SUNWldm# [cmt1/root] ldm listNAMESTATE FLAGS CONS VCPU MEMORY UTIL UPTIMEprimaryactive-n-cv SP64 8064M 0.0% 3h 19m[cmt1/root]All the system resource are in domain primary* Follow the Administration Guide to install required OS and patches230Saturday, May 2, 2009 231. Flag Definitions# ldm listNAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIMEprim