HP Backup Exchange

8/8/2019 HP Backup Exchange

1/24

Backup and recovery best practices for Microsoft ExchangeServer 2007 with HP servers and HP StorageWorks arrayand tape products

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Solution configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Testing objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 NTBackup with Exchange ESE streaming backup API . . . . . . . . . . . . 7

Backing up using 4 concurrent streams to 4 disks . . . . . . . . . . . 8 Backing up using 4 concurrent streams to 4 tape drives . . . . . . . . . 9Backing up using 16 concurrent streams to 16 disks . . . . . . . . . . 10 Backing up using 16 concurrent streams to 4 disks . . . . . . . . . . . 11 Backing up using 32 concurrent streams to 32 disks . . . . . . . . . . 12 Backing up using 10 concurrent streams to 10 tape drives . . . . . . . . 13Testing summary . . . . . . . . . . . . . . . . . . . . . . . . . 14

Exchange 2007 backups and VSS . . . . . . . . . . . . . . . . . . . 15 Exchange 2007 continuous replication and backups . . . . . . . . . . 15

Incremental backups . . . . . . . . . . . . . . . . . . . . . 16 Passive database backups . . . . . . . . . . . . . . . . . . . 16 Off-host database backups . . . . . . . . . . . . . . . . . . . 16

VSS and streaming backups . . . . . . . . . . . . . . . . . . . . 16 Best practices and results . . . . . . . . . . . . . . . . . . . . . . . . . 18

Monitor server workload . . . . . . . . . . . . . . . . . . . . . . . . 18 Keep mailbox databases to a manageable size . . . . . . . . . . . . . . 18 Use storage groups to gain a proper concurrency setting for backups . . . . . 18Know the impact of multiplexing backups . . . . . . . . . . . . . . . . . 18 Be careful with VSS concurrency . . . . . . . . . . . . . . . . . . . . 18 Perform incremental backups to save time and space . . . . . . . . . . . . 19 Three important things to back up . . . . . . . . . . . . . . . . . . . . 19 Minimize LUN presentation for backup volumes . . . . . . . . . . . . . . 19 Benefits of using streaming backups . . . . . . . . . . . . . . . . . . . 19

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Appendix A Bill of materials

. . . . . . . . . . . . . . . . . . . . . . . .

21

For more information . . . . . . . . . . . . . . . . . . . . . . . . . . . 23


2/24

OverviewBackupandrecoveryoperationsareacriticalactivitywithrespect toMicrosoftExchangeServer2007.Withmessagingcommonlyviewedasamissioncriticalapplication inmostcompanies, theability tobackupallorpartofyourExchangedatawhenrequiredandwithinthespecifiedbackupwindowisvital. MicrosoftisencouragingExchange2007administratorstobackupbyutilizingtheinewembeddedhost-basedVolumeShadowcopyService(VSS)technology. However,manyuserswichoose tocontinue todeployonlinestreamingbackupseitherasafirststepinimplementation,orasavalidmethod tosimplifyprocessesor leverageconcurrency.Thispaperprovidesusersinenterpriseenvironments thedata toeffectivelyunderstandtheoptionsaswellas the limitationsofimplementingonlinestreamingbackupsinto theirinfrastructure. Host-basedVSSbackupsarealsoaddressedandwillbe tested indepthata laterpoint.ThispaperisprimarilydirectedtoHPusersinanenterpriseenvironmentcurrentlyrunningorplanningtorunExchange2007. Thispaperprovides: Bestpracticerecommendationsbasedonextensive lab testresults;encompassingconfiguration,

design, process,andconsiderations Backupand recoveryrecommendationsforonlinestreamingbackupusers, focusingonboth

diskand tapeas targets (D2D2TandD2T) AcomparisonofperformancewithExchange2003 Supportingconfigurationrecommendations forHPserversand forHPStorageWorks8000

EnterpriseVirtualArray (EVA8000) Recommendationsfor theuseofmultiplestreamsandstoragegroupconfigurations toshorten

backupwindows AdiscussionofbackupswithVSS, includingadvantagesandnewfeatures Adiscussionof theadvantagesanddisadvantagesofutilizingstreamingversushost-based

VSS for backupsBy leveraging therecommendationsandbestpractices,administratorshave theopportunity toshortenbackupwindowsandefficiently load theserver, therebyreducingcostandmaximizinghardware and personnel resources.

2


3/24

SolutionconfigurationThetestingenvironmentconsistedofMicrosoftExchange2007utilizingHPProLiantBladeServersandtheHPStorageWorksEVA8000. TheHPStorageWorks6000EnterpriseVirtualArray(EVA6000)witFibreattachedtechnology(FATA)drives,andHPStorageWorksEnterpriseSystemsLibrary(ESL)e-Serie712e tape librarywerebothusedasdedicatedbackupandrestoredevices. Configurationof theservers

and

SAN-attached

devices

was

based

on

Microsoft

Exchange

2007

planning

and

architecturdocuments,HPStorageWorks4x00/6x00/8x00EnterpriseVirtualArrayconfigurationbestpractice

whitepaper,availableonhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdf,andHP StorageWorks Enterprise Backup Solution design guide, available onhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdf .Fora listofhardwareandsoftwareusedin thisproject,seeBillofmaterialsonpage21.During this testcycle,WindowsServer2003NTBackupwas theonlyreleasedbackupapplicationsupportingExchange2007. NTBackupwasused foronlinebackupswith thegoalofreducingbackupwindows,allowingmore time foronlinemaintenanceanddefragmentation.Areductionin restore timesasa resultof thesesolutionscanallowmore time forrecoveryoperations, thusincreasing thepotential tomeet tighterRecoveryTimeObjectives(RTOs).Note

AlthoughNTBackupwasusedtoconductthe testingdiscussedin thispaper,thebestpracticesdeveloped through this testingcanbeapplied tootherbackupapplications toreducebackupwindowsandrecovery time.

ExchangeMailboxServersTheMicrosoftExchange2007serversconsistedofHPProLiantBL480cG1BladeServersrunningMicrosoftWindowsServer2003R2x64EnterpriseEdition. Eachserverwasconfiguredwith twodual-portFibreChannelmezzaninecards. TheFibreChannelcardswerebasedonEmulexLP1105Seriesadaptersandwere installedusing theHPhostbusadapter(HBA)SmartComponentdrivers.TheHBAdriversusedwere theHPStorportmini-portdriversalongwith theMicrosoftWindows2003Storportdriver. Storportdriverswereused tocreateacompatibleconfiguration inwhichdiskand tapecanbeaccessed through thesameHBA.Formore tapecompatibility information,see theEnterpriseBackupSolution(EBS)compatibilitymatrixand theHPStorageWorksEnterpriseBackupSolutiondesignguide.Thec7000Bladeenclosureprovided theFibreChannelandEthernetconnections to theProLiantc-ClassBladeServers. TheenclosurealsoprovidedredundantpowerandcoolingfeaturesalongwithIntegratedLightsOut(iLO)connectivity. TheiLOconnectivitymadeitpossibletodeployandmanageremoteserversfromtheirconsoles.Allmanagementfeaturesinthec-Classenclosurewere interfacedthrough thec-ClassOnboardAdministrator (OA).Inorder tosize theExchangeenvironment, theHPSizingandConfigurationTool forMicrosoftExchangeServer2007andtheMicrosoftTechNetarticleMicrosoftExchangeServer2007(availablonhttp://technet.microsoft.com/en-us/library/bb124558.aspx )providedguidanceonserverCPUandmemorysizing.

3
http://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdfhttp://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdf


4/24

Storage arraysTheEVA8000storagearraywassetup ina2C12Dconfigurationwith146300-GB10KFibreChanneldiskdrives,whichwereusedfor theExchangestores.AnEVA6000wassetup ina2C8Dconfigurationwith80 500-GB 10KFATA disk drivesandwasusedasa target forD2D backups.Thevirtualdiskscreatedon theEVA8000werebuiltwitharedundancyofVraid1and thevirtualdiskscreatedon theEVA6000werebuiltwitharedundancyofVraid0.AllExchangedatabaseand logdatawasstoredon theEVA.Toverify theExchangestoragesize,weused theMicrosoftExchange2007StoragePlanningCalculator and HP Exchange 2007 Storage Calculator (http://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.html).Tape libraryTheHPStorageWorksESLe-Series712etape librarywasusedastheNearlinestoragedevice. TheESL was configured with 16 HP StorageWorks Ultrium960 (LTO3) Fibre Channel tape drives, 4e2400-FC4-GbFibreChannelInterfaceControllers,andoneInterfaceManager.

Backup applicationserverNTBackupwasrunontheExchangeservers. IneverytestcasebackupswererunfromtheExchangeserver to the SAN attached storage. Note

Adedicatedbackupserverisrecommendedwhenusinga third-partybackupapplication.

StoragemanagementTheHPStorageWorksCommandViewgeneralpurposeserverrunningonanHPProLiantDL580G1serverwasconfigured withMicrosoftWindows Server2003 Enterprise EditionSP1. HPStorageWorksCommandViewEVA6.0andCommandViewTL1.80wereused tomanage theEVAand tapedevices, respectively.SAN infrastructureTwoindependentfabricsconsistedofHPStorageWorksEdgeSwitch4/32B-Seriesswitchesrunningfirmwareversion5.1.0b.Alllinkstoandfromserversandstoragedeviceswere4Gb.DomaincontrollersAsingleActiveDirectory(AD)EnterpriseutilizingadomaincontrollerandbackupdomaincontrollerconsistedofHPProLiantBL20pserversrunningMicrosoftWindowsServer2003R2EnterpriseEditioandwereconfiguredasAD,DNS,DHCP,andGlobalcatalogservers.

Application loadsimulationMicrosoftExchangeLoadGenerator(LoadGen)wasusedasthe toolforcreatingtheADusersandpopulating theirmailboxes. LoadGenwasrunon theExchangeservers themselves. Userswere

4
http://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.html


5/24

initializedwith100MBand1GBmailboxsizes. MailboxsizeswerecreatedbyeditingtheLoadGeXMLconfigurationfileforaspecificdistributionandweightof testmessages.Databasestorage layoutForthistestingtherewere twoExchange2007mailboxservers. Thefirstserverhosted5,000userswith100-MBmailboxesand thesecondserverhosted5,000userswith1-GBmailboxes. ForbothserversonedatabaseLUNandonetransactionlogLUNwerepresentedforeverystoragegroup.Theserverhostingthe5,000100-MBmailboxeswasconfigured with 32storagegroupswithonemailboxdatabaseperstoragegroup. Each100-MBusermailboxdatabasehosted156usersandhadatotalmailboxdatabasesizeofapproximately16GB.Totalserverstoragewasapproximately520 GB.Theserverhosting the5,0001-GBmailboxeswasconfiguredwith50storagegroups,withonemailboxdatabaseperstoragegroup. Each1-GBusermailboxdatabasehosted100usersandhadatotalmailboxdatabasesizeofapproximately100GB.Totalserverstoragewasapproximately5TB.

5


6/24

Figure 1. Test environment configuration

1. ExchangeLoadGennetworkclients 6. EVA6000 (FATA)2. Managementserver 7. EVA80003. Exchangeservers (BL480c) 8. ESLe-Series712e4.Activedirectoryandbackupservers 9. Localareanetwork5. 32-Port4GbB-SeriesFibreChannelswitches

6


7/24

TestingTesting objectivesExchange2007offersmanynewfeaturesthatdirectlyandindirectlysupporthighperformanceandfeaturerichbackupandrecoverysolutions. Sinceonlinestreamingbackupsthrough theExtensibleStorageEngine(ESE)APIwerede-emphasizedbyMicrosoft,muchworkhasbeendonebybackupapplicationvendors tosupportbackingupand restoringExchange2007usingVSS.VSSoffersaway tocreateaconsistentpoint-in-timecopyof thedatabase. Now thatClusterContinuousReplication(CCR)orLocalContinuousReplication(LCR)providesstandbycopiesof theExchangedatabase,betterandsimplerExchangebackupsarepossible.Someof themore indirectenhancements inExchange2007 thatsupportbackupandrecoveryinclude: The64-bitoperatingenvironment The number of storage groups available

Exchange

management

console

Recovery storage groups Database portability Increasedretention in thedeleted itemsdumpsterCustomersusingExchange2007areexpected toraisemailboxquotas,whichwillcauseExchangeservers to host more data. Because there will be more data to back up and no more time in which toperform thebackups,it is important toconsiderways toacceleratetheprocess.ThispaperfocusesontestingperformedwithExchange2007inanHPserverandstorageend-to-endbackupsolution. Thegoalsof the testingwere to: Demonstrate ways of minimizing the backup windows through the use of storage group

configurationswithstreamingbackups. Developbackupandrecoverybestpractices.NTBackupwasused to testExchangestreamingbackups.NTBackupwithExchangeESEstreamingbackupAPISinceMicrosofthasde-emphasizedtheirsupportoftheESEstreamingbackupAPI,newfeaturesandfunctionalitywillnotbeavailableforthisbackupmethod. ESEstreamingbackuphaschangedverylittle in itssupportofExchange2007andworks inmuch thesameway itdoeswithExchange2003topulldataoutof theExchangedatabaseandmove it tobackupmedia.Weconductedthefollowing tests tounderstand theeffectofconcurrencyonbackupperformance:1. Backingupusing4concurrentstreams to4disks2. Backingupusing4concurrentstreams to4 tapedrives3. Backingupusing16concurrentstreams to16disks4. Backingupusing16concurrentstreams to4disks

7


8/24

5. Backingupusing32concurrentstreams to32disks6. Backingupusing10concurrentstreams to10 tapedrivesEach test is analyzedseparately.Backingupusing4concurrentstreamsto4disksWeusedNTBackuptobackupExchange inordertoallowcomparisonbetweenpreviousExchange2003 testdataand thenewExchange2007 testdata. Exchange2003 imposedamaximumoffourstoragegroups. Typically,astoragegroup inExchange isassigned toavolumepartitionontheserver. If therewerefourpartitionsfor thestoragegroups then itwouldbepossibletobackupusinguptofourconcurrentstreams. Toenableafaircomparison,Exchange2007wasconfiguredwith5,000users,eachwitha100-MBmailbox,splitamongfourstoragegroupsand fourmailboxdatabasesperstoragegroup,aswasExchange2003 in theprevious tests.InpreviousbackuptestingusingdedicateddiskbackuppartitionsonanEVA6000withFATAdrives,Exchange2003wasable toachieveaperformanceof283GB/houror520GB in1hourand50minutes.WhenperformingExchange2007backups,using thesameconfiguration,similarperformancewasobserved,thereforedemonstratingthatnotmuchhaschangedinthestreamingAPforExchange. Performancedidnotincreasewiththechangetoa4-Gbstorageareanetwork(SAN)infrastructureandtwoadditionalFibreChannelhostbusadaptersaswellasanadditional28GBomemory. Figure2highlights resultsof the test.

Figure 2. 4streams todisk (D2D)

8


9/24

Note thepagepoolmemoryutilizationandprocessorutilization inFigure2. Knowinghowhardtheserver isworking toperformbackups isakey indicatorofExchangeserverefficiencyandcanindicatepotentialroom for improvement.Withonly5%processorutilizationand less than70MBofpagepoolmemory inuse, thisserverhasplentyofroomfora largerbackupworkload,assumingbackupsaredoneduringscheduledbackupwindowswherethereisminimaluseractivity.The lack of performance improvementbetweenExchange2003andExchange2007, with itsadditionalmemoryand fasterstoragenetworking infrastructure, indicates that theperformancebottleneckmustliesomewhereinthedeliveryofthedatafromExchangeto thebackupmedia. Itispossible that thestreamingbackupAPI limits thespeedatwhichdata isdelivered to thebackupmedia,due tooperationssuchasconsistencychecksandsmallreadblocksizesof64KB.NoteHP recommends thatyou to use thefile un-buffered switch with NTBackup jobs to disk, even on 64-bit Windows Server2003. Notusing thisswitchwillcauseprocessor andmemory tobe over-utilized resulting in poorbackupperformanceandconsequentlyextendedbackupwindows. Formore information,see theMicrosoftsupportwebsite:http://support.microsoft.com/kb/839272/.

Backingup

using

4

concurrent

streams

to

4

tape

drives

Latergenerationtapetechnologyhasproventobemoreefficientthandiskforhighcapacitybackupdue to the tapedevicesembeddedmemorybuffersandhardwarecompression. Therefore,wetestedExchange2007backups to tape inorder toprovidedata forcomparison. Leveraging thesame testconfigurationasthepreviousdisk todisk testingwithfourstreams,andusingLTO3 tapedevices insteadof theFATAdrives in theEVA,should improveperformancegiven thememoryandcompression of the tapedrive.Figure3showsthatperformanceusingtapewithafourstream,fourstoragegroupconfigurationwa15%better thanwhenusingdisks. Thesame520-GBdatabasewasbackedup to tape in1hourand30minutes. Notethattheincreasedperformancecorrespondstohigherprocessorandmemoryutilization than thatobservedwhenusingdisks in theprevious test.

9
http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/http://support.microsoft.com/kb/839272/


10/24

Figure 3. 4 streams to tape

Backingupusing16concurrentstreamsto16disksAlthough theExchangeonlinestreamingbackupAPIhasnothelped to improve theperformanceofExchange2007backupswithNTBackup, thereareotherfeatures inExchange2007 thatmightimprovebackupperformance. Theprimaryfocusofthistestingwastoincreasethenumberofstoraggroups,resulting inan increase in thenumberofconcurrentstreams tobackupmedia, inorder toobserve theeffectonbackup timesandsystems.Exchange 2007 was configuredtospread5,000100-MBusermailboxesacross32storagegroupsandonemailboxdatabaseperstoragegroup. Thisallowedfor theconfigurationofNTBackup tosend16concurrentstreams to theEVA6000fordisk todisk(D2D)backups. Eachbackupstreammoved twostoragegroups to thebackup target. Figure4showsthattheserverachieveda220%performanceincreaseoverfourstreambackups,withatransferrateofapproximately624GB/hour.

10


11/24

Figure 4. 16 streams todisk

Notice thattheserverusedmoreCPUandmemorytoperformthe16streambackupsversusthefoustreambackups. Oneof thebenefits of using 64-bit Windows Server is the ability to use more thantheprevious32-bit limitationof4GBofmemory. Because thisExchange200764-bitserverhad32GBofmemory,kernelmemory,orpoolpagebytes,no longerhas the32-bit,Exchange2003limitation of 180 MB. Even though the Exchange 2007 server did not go above this limit, pool pagebytesservesasan indicationofhowhard theserverhadtowork togetthebackupdone. CPUdidnotgo above 50% processor time.Backingupusing16concurrentstreamsto4disksThenumberofD2D targetdiskpartitionswasreducedtofourfrom16tosee ifthenumberof targetdiskshadaneffectonoverall throughputof thebackup job. Thiswillmake iteasier forstorageadministrators tomanage theproliferationof logicalunits (LUNs) forExchangebyreducing thenumberofLUNsrequiredforbackup. Figure5showsthatwhenthetargetbackupdiskswerereducedto four,performancestayed thesame;however, theserverhad toworka littleharder tomaintainthebackupfile imageson therespectivebackupdiskLUNs. Notehowservermemorygraduallyincreasedasbackupfilesizesgrewon the limitednumberofbackupdisks.

11


12/24

Figure 5. 16 streams to4disks

Evenwith theslightgrowth inmemoryandCPUutilization in this testcompared to thepreviousone, theworkloaddidnotstrain theserverandperformancewasmaintained. This testshows theadvantages inusingdisk-basedbackup technologyover tape;multiplestreamscouldbesent toasingledisk targetwithout theneedformultiplexingor interleavingdatastreams. Multiplexing isaconcern forsomebackupadministratorsbecauseof thepotential for increased restore timeasdata is de-multiplexed.Backingupusing32concurrentstreamsto32disksInthistest,thenumberofstoragegroupswasincreasedto32,eachwithasinglemailboxdatabaseEachof the32backupstreamstargetedadedicatedbackupLUNon theEVA6000. ManagingsomanyLUNsandpartitionsbecame increasinglycomplex. Figure6shows thedecrease inbackupperformancewhichresultedfrom theserverhaving tomanagesomanystreams.

12


13/24

Figure 6. 32 streams todisk

BecauseNTBackupmaintainsasinglejobandgraphicalinterfaceforeachstream, thedesktoponthisserverhosted32applicationsrunningconcurrently,resulting inan increasedsystemmemoryload. Backupperformancedroppedby29% from624GB/hour in the16streams tests to445GB/hour. Notethatpagepoolbytesincreaseddramaticallyfromthe100150MBrangeinprevioutests, to just over 400 MB. Although the upper limit for page pool bytes can be much higher on a 64-bitoperatingsystem,thechangedemonstratestheamountofworktheserverhadtodotomanagethemultitudeofbackupstreams.Asshown inFigure6,serverloadhasadirect impactonbackupperformance. In thistest,theperformancediminishedsomewherebetween16and32streams.Backingupusing10concurrentstreamsto10tapedrivesDueto theefficiencyofphysical tapedevices like theLTO3,highcapacitybackupjobscanachievehigh throughput. TheExchange2007databasewasconfiguredwith5,0001-GBmailboxusersacross50storage groups. Tenbackup streams were configuredwithNTBackup,withfivestoragegroupsperstreambeingsentserially to the tapedevice. Becausephysical tapedevicesofferimprovedperformanceduetohardwarecompressionanddatabuffering,performanceofthissolutiowasapproximately1,330GB/hourwhichrepresentsanincreaseinperformanceof470%over thefourstreambackups. Ittook3hoursand50minutestobackupthe5-TBdatabase. Figure7showsthemoderate impacton theserverresources.

13


14/24

Figure 7. 10 streams to tape

The test demonstrates the benefit of using tape for high concurrencybackups. Even higherperformancewouldbeexpectedifmoretapeswereusedtoenablemorestreamswiththisworkloadTesting summaryThe testdatashows thatusingmorestoragegroups in Exchange2007 willallow forhigherbackupconcurrencywhich, in turn,shouldprovideforhigherperformancebackups. SeeFigure8.Understandingserverutilizationduringbackupshelpstoidentifyopportunitiesforraisingconcurrencandbackupperformance.Asseen inFigure6,32concurrentstreams to32disks,performancediminishedastheserverkernelmemorybecameover-utilized. ToreducethenumberofbackupLUNrequiredtoachieveproperconcurrencyrememberthatmultiplestreamscanbesenttoasinglediskvolume,withouttherequirementtointerleave thedata,whichwouldcauserestorestotakelonger.

14


15/24

Figure 8. NumberofstoragegroupsversesperformancewithD2Dbackups

Exchange2007backupsandVSSVSShasprovenuseful inenabling thecreationofapoint-in-timecopyofopenfilesanddatabases.Microsofthas improved theVSSframework,making iteasierforstorage,serversandapplicationstowork together. PreviousVSSsolutionswerecomplexandcostlybecause theyrequireddetailedknowledgeofhowtheVSSstorageproviders,ExchangeVSSwriters,andbackupapplicationVSSrequestorsworkedtogether. FewpopularbackupapplicationssupportedExchange2007when itwasreleasedalthoughsomevendorshadcommittedtosupportlaterin theyear.Withthegrowth inVSSandwithMicrosoftsdecision tomoveExchangebackups towardsaVSS-basedsolution, thesituationhasimproved;backupapplicationprovidersarenowtakingtheirVSSintegrationseriously.BecauseVSSisaserialframeworkandconcurrentbackupsofthesamestoragegroupareprohibitedthe backup jobs must be configuredcarefully. BecauseVSS isalsoavolume levelapplication,administratorsshouldbecautiouswhenrunningconcurrentbackupsofstoragegroups locatedonthesamepartition. Becausetherewillbe limitationsonconcurrency, it is important toconfigurethebackupapplicationssoas tooffset the jobsthusoffsetting theVSScreationprocesses.ThenewExchangeStoreConsistencyCheckReferencecapability (chksgfiles.dll) allows the backupapplicationtoprogrammaticallyverifythe integrityofExchangestoragegroupsandmailboxdatabases. Thisenablesthebackupapplication tobalance thespeedoftheintegritycheckagainsttheworkloadplacedon theserver.Withearlierversions,VSSbackupscouldnotberestored toExchangeRecoverystoragegroups.WithExchange2007,VSSbackupscannowberestoredtoanExchangeRecoveryStorageGrouponanyExchangeserver in theorganization.Exchange2007continuousreplicationandbackupsContinuousreplicationcapabilities,whichprovide logshippingservices toremotestorageorremotehosts,enhanceavailabilityinMicrosoftExchange2007. LocalContinuousReplication(LCR)providesfor thedatabase tobereplicated toalternatestorageon thesamehost,with logreplaykeepingthepassivedatabaseup todate. ClusterContinuousReplication (CCR)provides for logs tobeshippedtoapassiveclusternodewherethereplicatedlogsarereplayed into thepassivedatabase.Although thebenefitsforquickrecoveryareapparent,continuousreplicationalsoprovidesforsomeimportantpossibilitieswithrespecttootherbackupandrecoveryoperations. Threecapabilitieswhiccontribute tobackupandrecoveryarediscussed further: Ability toperformdaily incrementalbackupswithweeklyfullbackups

15


16/24

Ability tominimize the relianceofbackupwindowsbybackingup thepassiveLCRorCCRcopy of the database

Ability toperformoff-hostbackupswith thepassiveclusternode inCCRIncremental backupsIn theabsenceofcontinuousreplication,daily fullbackupsof theExchangedatabasearehighlyrecommendedbecausemerging incrementalwithfullbackupsextendsrecovery timeandexposestheorganizationto theriskoflosingdata. Continuousreplicationreducestheriskofdatalossandprovidesforrecoverywithouthavingtorestorefrombackups. Duetothereducedrelianceonbackupas the recoverysource, incrementalsarenowarecommendedstrategywhenusingcontinuousreplicationwhich in turnwillhelp tominimizebackupwindows.Passive databasebackupsUsingapassiveratherthanactivecopyofadatabaseasthebackupdatasourcereducescontentioforserverandstorageresources,assuming thepassivecopyofthedatabaseisonadifferentsetofdisks.VSSmustbeusedtobackupthedatabasereplicasbecauseMicrosofthasaddednewfunctionalitytoExchange2007buthasnotenhancedtheESEstreamingbackupAPI.TherearenowtwoVSSwriterthatcanberesponsibleforExchange in theVSSframework: theStorewriterand theReplicationwriter. Ineitherpassiveoractivedatabasebackups,restoreswillalwaysbewritten to theactivedatabaseor inotherwords, to theactiveserverrunning thestoreprocess. TheVSS integration iscritical inmaintainingdatabaseconsistency through theuseofreplicationservicesandchecksumverificationof theVSScopyof theactiveorreplicadatabases. Iffullor incrementalbackupsof thedatabasearedonebyusing thecontinuousreplicaasasource,thereplicationservicewillkeep thestoreheaderup todate inboth theactiveandpassivecopiesof thedatabase toensuredatabasedivergencedoesnotoccur. Thereplicationservicealsoplaysakeyrole inhowmany logswillbetruncatedasaresultofafullorincrementalbackup. Log truncationon theactivedatabaserequiresthatthe logfileshavebeenreplayedintotheactiveandpassivedatabasesandhavebeenbackedupusingeithera fullor incrementalbackup.Off-hostdatabase backupsWhenusingCCR,apassivedatabaseclusternodecan takeresponsibility forExchangebackupsso that theactivedatabasenodecanreserveresources for theuserand replication loads. Theoverhead incurredon thepassivenodeshouldhave little tono impacton theactiveserver. Thissolutionmayalleviatepressuresonenterprisebackupwindowsor time fordatabasemaintenanceactivities.Aswithpassivedatabasebackups,restoresmustbedone to theactivestore. Incrementalscan also beusedwithCCRoff-hostsolutions.VSSandstreamingbackupsVSS isdefining thewayservers,storage,andapplicationswork together toprotectdata. Thistechnology iscriticalforapplications thatrequireconsistentpoint-in-timecopies. Theflexibility thatVSSsolutionsnowprovidewithrespecttooff-hostbackupsandfastsnapshotssetsanewstandardfoExchangebackups.VSShasmanybenefits. However,itisarelativelynewtechnologyandhassomelimitations.VSSlimitsinrunningconcurrentstreamsand itsdependenceondataat thevolumelevelhaveanegativeimpactonperformance. EarlytestinghasdemonstratedVSSlimitsatapproximately8concurrent jobs. Offsetting thestart times forVSSbackupsso thatmultiple jobsdonotstartat

16


17/24

thesame timecanmitigatebackup failurescausedbyVSS timeouts. Inaddition,becauseVSS isdependentondataatavolume level,runningconcurrent jobsfromdatabasesorstoragegroupson the same partitioncancausebackup failuresandVSSsnapshotcreation timeouts.AsVSSbecomesamorematurebackupframework,itisexpectedthatmanyoftheseissueswillberesolvedMeanwhile,streamingbackups forExchange2007shouldbeconsideredasawayaround theconcurrency limitations imposedbyVSS.AlthoughnotasfastasVSSinsinglestreamapplications,streamingbackup isastrongalternativewhenconcurrency is involved.

17


18/24

BestpracticesandresultsMonitorserver workload ServeradministratorsshouldnotehowExchangeserversperformduringbackups. Serversmayhaveroom to reconfigurebackup jobs toallowformoreconcurrencywhich, in turn,shouldprovide forbetterperformance. Inbackingup32concurrentstreamsusingNTBackup,400MBofpagepoolmemorywasused. Thisindicatesthattheserverwasover-taxedcausingadecline inperformance.Keepingpagepoolmemorybelow180MBappears tocontribute toefficiencyandperformance.Keepmailboxdatabases toamanageablesizeBecauseExchange2007allowsformanymoremailboxdatabases, itispossible toallocatefeweruserstoeachmailboxdatabase. Keepingmailboxdatabasestoamanageablesizewillenablefasterestorationandwilldecreaseserverworkload. Microsoftrecommendskeepingmailboxdatabasesizesbelow100GBifcontinuousreplicationisnotbeingused. Microsoftsrecommendationisbasedon the time it takes toperformafullrecovery.A fullrecovery involvesrecovering thefiles thatwerebackedup,and thenreplaying logfilesintothedatabasetobringitto thepointin timeatwhich itwasbackedup. Thesmaller thedatabasefile,thefastertherecoveryofadatabasewillbe.UsestoragegroupstogainaproperconcurrencysettingforbackupsUsebackup jobconcurrency tomovemoredata. Typically,concurrency isappliedat thevolumelevel,meaning thatmostbackupapplicationsworkwellwithconcurrentstreamswheneachstreamiscoming fromadifferentdiskvolume. Exchange2007storagegroups typically resideonadedicated LUN and provide a good way for Exchange to take part in backup concurrency. WhetheusingstreamingbackupsorVSSbackups,morestoragegroupsareavailableforhigherdegreesofconcurrency. Thedata included in thisreportsuggests that16concurrentstreamsprovidesforthefastestbackups. Forsuchhighlevelsofconcurrencyconsiderusingdisk-to-diskbackupsas therewould

need

to

be

16

available

tape

drives

to

be

able

to

have

16

concurrent

backup

streams

to

tape

Know the impactofmultiplexingbackupsItiscommontosendmultiplebackupstreamstothesametapedeviceinordertokeepthetapedevicestreamingquicklyandkeepconcurrency levelshighwhenonlyafew tapedrivesareavailable. Inmostcasesmultiplexingcan improvebackupperformance,butcancauserestores to takemuchlonger. Forexample,whenfour25-GBdatabasefilesarebackedupin4streamsto thesametapedeviceusingmultiplexing, thebackupfileon tapewillbea100-GBbackupfile. Inorder torestoreasingle25-GBdatabasefile,all100GBofthebackupfilemustberestored. Iffastbackupsarearequirementandonlyafew tapedrivesareavailable,considerstaging theprimarybackups toadiskvolume,and thenmoving thedisk-basedbackups to tape. Note that therecanbemultipleconcurrentstreams todisk,withoutmultiplexing thedatastreams.BecarefulwithVSSconcurrencyCreatingVSSpoint-in-timecopiesismostlyaserialprocess. BeawareofhowbackupjobsaresetupwhenusingVSSintegratedbackups. Iftoomanybackupjobsarestartingatthesame time,theVSSprocesscantimeout. IfaVSSjob timesout,thebackup jobcanfailandwillneed toberestarted.Sincemanybackupswilluse theVSSpoint-in-timecopyasamethodforbackingup thedatabase

18


19/24

todiskor tape, thepoint-in- timecopycreationwillbefollowedby thebulkdatamovement to thebackupdestination. Byoffsettingbackup jobswithpauses(forexample,5minutes)inbetween, theVSSpoint-in-timecopycreationcantakeplaceonitsown,whilethebulkdatamovementwillhappeinamoreconcurrentfashion.VSSconcurrencyappearstobecomeproblematicataround8streamsFinally, iftherearemultipleExchangestoragegroupsperLUN,besurethatstoragegroupsfrom thesameLUNsarenot backed up concurrently as VSS is a volume level service; backing up two storagegroups from thesamevolumeat thesame timecouldcauseconflicts.NoteMicrosofthasreleasedanupdaterolluppackage thatresolvessomeVSSsnapshotissuesinWindowsServer2003. Formoreinformation,seehttp://support.microsoft.com/?kbid=940349 .

Perform incrementalbackupstosave timeandspaceIfExchange2007continuousreplication isbeingused,incrementalbackupscanbeused insteadofdailyfullbackups. Incrementalbackupshelp tomaintain transaction logtruncation,whilereducingtheamountofrepeateddatabackedupeverynight. Ifusing incrementalbackups,expectalongerrecovery time torestore the fullbackupaswellasevery incremental inorder toget thedatabasetoapoint-in-timerecovery.Also,plan forenough time toperform logreplayafter thefileshavebeen restored.Three important things tobackupAlthoughSystemStateand theExchange2007databaseare theobviousrequirementsforbackuppolicies,consideralsobackinguptheEVAconfigurationdataasapartoftheregularbackupitemsEVAconfigurationdatacanbecollectedbyrunningHPStorageWorksStorageSystemScriptingUtility,which is includedinCommandViewEVAorcanbedownloadedasaseparateupdatefromHP.Saving thisconfigurationdatafrom theEVAwillallow forarapid replacementof thearray,should

the

entire

array

need

to

be

replaced.

MinimizeLUNpresentationforbackupvolumesAlthoughitisimportanttohavealargernumberofLUNsfortheExchangestoragegroupstobackupusingconcurrentstreams, thedisksthatarebeingbackedupcanbefewerthan the totalnumberofstorage group volumes. Although fewer target volumes can be used, the server will have to work harderas itmanagesmultiple, large,growingfilesonafilesystem.BenefitsofusingstreamingbackupsStreamingbackupswillbenefit from usingconcurrency to improveExchangeserverbackups.Although therearesome limitationsonhow fastdatacanbestreamedwhenusing thisbackupmethod,theabilitytoperformconcurrentbackupsoutweighsanyperformancelimitations. Streamingbackupsarebeingde-emphasizedbyMicrosoft,however, theycanbe lesscomplicated thanVSSintegrated backups.

19
http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349http://support.microsoft.com/?kbid=940349


20/24

ConclusionThispaperpresentedresultsandbestpracticesderivedfromonlinestreamingbackupandrecoverytesting,andreportedVSSbackup testingobservations. BothonlineandVSSbackupscanhaveavalidplace inExchange2007backupdesign.Thespecificrecommendationsresultingfromthisworkdemonstrate thatearlyplanningcanhavealargeimpacton theabilitytoperformefficient,timely,andeffectivebackupandrestorations.Planning considerations: Serverworkload is important tomonitor. If theworkload is low,consider reconfiguringyour

backup jobsformoreconcurrency, leading tobetterperformance. Largerdatabasesizesmightnotbebetter. Mailboxsizeneeds tobemanageable for faster

restorationanddecreasedworkload. Leveragestoragegroupstomaximizeconcurrency,configuringstoragegroupsondedicatedLUNsFindings: Basedonfourstoragegroups(thelimitforExchange2003),noperformance improvementwas

observed fordisk-to-diskbackupbetweenExchange2003andExchange2007. Disk-to-tapeshowedanapproximate15%performance improvement.

By takingadvantageof thenew Exchange2007storagegroup limits, userscan increaseconcurrentstreams tobackupmedia. Testingsupports that16concurrentstreamsprovide thefastest backups.

Byreferencing the lab-provenresultsfrom thispaperandgivingconsideration to thederivedbestpracticesandrecommendations,usersaregiven thesupporting informationnecessarytoplanandimplementaneffectiveandefficientExchange2007backupandrecoverystrategy tomeet theirspecific requirements.

20


21/24

AppendixABillofmaterialsThefollowingtabledescribesthespecificenvironmentutilizedintestingandisincludedforreferencepurposes.BL480CServersQuantity Partnumber Description1 AF002A HPUniversalRack 10642G2ShockRack1 AF002A 001 Factory Express Base Racking1 403321-B21 HP BLc7000 1 PH 2 PS 4 Fan Full ICDC Kit 2 406740-B21 HPBLc1GbEnetPassThruModOptKit2 412138-B21 HPBLc7000EnclPwrSply IEC320Option2 412140-B21 HPBLc7000EnclSingleFanOption1 412142-B21 HPBLc7000EnclMgmtModuleOption1 H4396A NoAdditionalSupport4 416669-B21 HP ProLiant BL480cG151604G2PSvr4 U2426A Microsoft & Novell OE 4 416669-B21 HP ProLiant BL480c G1 5160 4G 2P Svr24 397413-B21 HP4GBFBDPC2-5300 2x2GBKit12 375861-B21 HP 72GB 10K SAS 2.5 Hot Plug Hard Drive8 403621-B21 HPBLcEmulexLPe1105FCHBAOptKit4 U2426A Microsoft & Novell OE 1 AF062A HP10KG2600WStabilizerKit1 AF062AB01 Includewithcompletesystem1 AF054A HP 10642 G2 Sidepanel Kit 1 AF054A0D1 Factory integrated2 252663-D75 HP40AHVCoreOnly Corded PDU2 252663-D750D1 Factory integrated2 AE371A BrocadeBladeSystem 4/24SANSwtPowrPkEVA6000AbackuparrayQuantity Partnumber Description1 AD522A EVA8000 2C12D80 364437-B23 FATA Drives, 500GBFC10K2 T4256D HPEVA4K/6K/8K6.0ControllerMediaKit1 AF002A HPUniversalRack10642G2ShockRack2 T3724C HPCommandViewEVAv5.0MediaKit2 T3732A HPCVEVA5000/8000UnlimuseperEVALTU

21


22/24

EVA8000BprimaryarrayQuantity Partnumber Description1 AD522A EVA8000 2C12D168 364622-B23 HDD,300GBFC110KFactALL168 364622-B23 HDD, 146GB FC 1 10K FactALL2 T4256D HPEVA4K/6K/8K6.0ControllerMediaKit1 AF002A HPUniversalRack10642G2ShockRack2 T3724C HPCommandViewEVAv5.0MediaKit2 T3732A HPCVEVA5000/8000UnlimuseperEVALTUTapestorageQuantity Partnumber SAN1 AA934C Zerodrive,712LTOcartridgebase library4 AA938A ESL E-Series Drive Cluster16 AD595A ESL E-Series Ultrium960FCDriveUpgradeKit4 AD576A ESLE-Seriese2400-FC4G InterfaceController1 InterfaceManager (comeswithEnterpriseLibrary)InfrastructureQuantity Partnumber SAN2 A7393A HPStorageWorksSANSwitch4/3264 A6515A HPShortWaveOpticalTransceivers64 50micronSWFiberPatchcordsClient-ware

Exchangeclienthardware18 8 DC5000 Desktops, 6 DL320 Servers, 4 DL360 Servers 1 CVEVAandCVgeneralpurposeserver

Networking4 J4897A HP ProCurve 2724 Gig-E switch

22


23/24

Formore informationThissection listsreferencesand theironline locations.HPLinksNoteSomeofthefollowinglinksaresecurewebsites thatrequireanHPPassportregistration. HPPassportisasingle loginservicethat letsyouregisterwithHPPassport-enabledwebsitesusingasingleuseridentifierandpasswordofyourchoice.

HP Passport registrationhttps://passport2.hp.com/hpp/newuser.do

HPExchange2007StorageCalculatorhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.html

HP Sizing and ConfigurationToolforMicrosoftExchangeServer2007http://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.html

HPExchange2007ProLiantSizerhttp://www27.compaq.com/sb/Exchange2007/index.asp

CustomerFocusedTesting solutionshttp://www.hp.com/go/hpcft

EnterpriseBackupSolution(EBS).FromtheEBSpage,clickaquicklinkorselectEBS123.http://www.hp.com/go/ebs

HPSANdocumentationhttp://www.compaq.com/products/storageworks/san/documentation.html

HPStorageWorksEnterpriseBackupSolutionnearonlinebackup-restoresolution implementationguidehttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pd

HPStorageWorksEnterpriseBackupSolutiondesignguidehttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdf

EnterpriseBackupSolutionHardware/SoftwareCompatabilityMatrixwww.hp.com/go/ebs

HPsupportdocumentation (performanceand troubleshooting)http://www.hp.com/support/pat

HPStorageWorksSANdesignguidehttp://www.hp.com/go/sandesign

HP IT resourcecenter (see the forums)

23
https://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/go/sandesignhttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://www.hp.com/support/pathttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://c/Documents%20and%20Settings/kretzca/My%20Documents/CFT/oracle%20final/Oracle%2010g%20Grid%20Control%20final_for%20xml/www.hp.com/go/ebshttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdfhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.compaq.com/products/storageworks/san/documentation.htmlhttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/ebshttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www.hp.com/go/hpcfthttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://www27.compaq.com/sb/Exchange2007/index.asphttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttp://h71019.www7.hp.com/activeanswers/Secure/511755-0-0-0-121.htmlhttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.dohttps://passport2.hp.com/hpp/newuser.do


24/24

http://itrc.hp.com/MicrosoftLinks: NTBackupandExchange2007

http://technet.microsoft.com/en-us/library/aa998870.aspx Exchange2007Backups described

http://technet.microsoft.com/en-us/library/bb124515.aspx NTBackup and VSS

http://msexchangeteam.com/archive/2004/06/25/166104.aspx ExchangeStoreConsistencyCheckReference

http://msdn2.microsoft.com/en-us/library/bb204219.aspx Exchange 2007 TechNet article

http://technet.microsoft.com/en-us/library/bb124558.aspx YouhadmeatEHLOMicrosoftExchangeTeamBlog

http://blogs.msdn.com/exchange TechNetWebcast: HighAvailabilityandClustering inExchangeServer2007

https://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-us Backup ProcessUsedwithClusteredExchangeServer2003Servers

http://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-

8A07E02CF9BF&displaylang=en

2007Hewlett-Packard DevelopmentCompany,L.P.Theinformationcontainedhereinissubject tochangewithoutnotice. Theonlywarranties forHPproductsandservicesaresetforth in theexpresswarrantystatementsaccompanyingsuchproductsandservices. Nothinghereinshouldbeconstruedasconstitutinganadditionalwarranty. HPshallnotbe liable for technicaloreditorialerrorsoromissionscontainedherein.MicrosoftandWindowsareU.S.registered trademarksofMicrosoftCorporation.4AA1-6077ENW, October2007
http://itrc.hp.com/http://itrc.hp.com/http://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A0-8A07E02CF9BF&displaylang=enhttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttps://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-ushttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttp://blogs.msdn.com/exchangehttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://technet.microsoft.com/en-us/library/bb124558.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msdn2.microsoft.com/en-us/library/bb204219.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://msexchangeteam.com/archive/2004/06/25/166104.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/bb124515.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://technet.microsoft.com/en-us/library/aa998870.aspxhttp://itrc.hp.com/http://itrc.hp.com/http://itrc.hp.com/http://itrc.hp.com/http://itrc.hp.com/http://itrc.hp.com/http://itrc.hp.com/http://itrc.hp.com/