Transcript
Page 1: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

W H I T E PA P E R

m e t a l o g i x . c o m

INTRODUCTION............................................................................................. 2

REPLICATOR PERFORMANCE AND SCALABILITY FEATURES .................. 4

Replicator Multi-Engine Deployment ................................................. 7

Multi-Threaded Replication Queue Architecture ............................. 7

Customized Replication Interval....................................................... 8

ConfigurablePackageEventCount................................................8

ConfigurablePackageEventProcessingDuration..........................9

ReplicationPackageCompression..................................................9

HardwareNetworkCompressionSupport.....................................9

SoftwarePackageCompression...................................................10

RemoteDifferentialCompression(Rdc)........................................10

IndependentOutboundEventAndInboundEvent

ProcessingControls.......................................................................11

SelectiveStructureReplication.....................................................11

Rule-BasedItem-LevelContentReplication.................................11

SelectableReplicationEvents.......................................................12

SharedReplicationPackages.......................................................12

PackageDatabaseCaching..........................................................13

ConfigurableReplicationMonitorUpdateLevel...........................13

ImprovedInitialReplicationSupport.............................................14

MICROSOFT TECHNOLOGY CENTER PERFORMANCE

ANDSCALABILITYBENCHMARK...............................................................14

METALOGIX NETWORK TEST LAB PERFORMANCE BENCHMARKING...16

UNDERSTANDING REPLICATOR PERFORMANCE AND SCALABILITY.......................................................................17

MetalogixReplicatorPipeline.......................................17

ReplicatorPerformance...........................................18

OutboundEventProcessing..................................19

PackageTransfer......................................................20

InboundEventProcessing......................................21

Performance Monitoring .........................................22

Replicator Pipeline Performance Features .............24

ReplicatorScalability..............................................25

APPENDIX A – ABOUT WINDOWS REMOTE DIFFERENTIAL COMPRESSION ................................................................ 25

Copyright © 2012 Metalogix International GmbH. All rights reserved. Metalogix is a trademark of Metalogix International GmbH. StoragePoint is a registered trademark of BlueThread Technologies, Inc. Microsoft, Exchange Server, Microsoft Office, SharePoint, and SQL Server are registered trademarks of Microsoft Corporation.

Metalogix Replicatorfor SharePointENTERPRISE PERFORMANCE AND SCALABILITYVersion4.1June2012

CONTENTS

Page 2: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

MetalogixReplicatorisanenterprise-classsolutionforreplicatingcontent,metadata,structure,security,look&feelandbusinessprocessesfromoneSharePointwebapplicationtoantherwebapplication.Replicatorversion4supportsSharePointFoundation2010,SharePointServer2010,OfficeSharePointServer2007andWindowsSharePointServices3.0.

MetalogixReplicatorsupportsthereplicationneedsofMicrosoft’s largest(andsmallest)SharePointcustomers.Version4providesadditionalperformanceandscalabilitycapabilitiesthesecustomerswillneedastheycontinuetogrowtheirDistributedSharePointenvironments.

The release ofMetalogix Replicator version 4 represents a new performance and scalabilitymilestone forSharePointreplicationsolutions.Version4includesnewfeaturesforsupportingmassivelyscalableSharePointreplicationnetworks:

SharedReplicationPackages,

PackageDatabaseCaching,

Improved Initial Replication Support, and

ImprovedRemoteDifferentialCompression(RDC).

Together,thesefeaturesbuildontheperformancefeaturesthathavebeenbuiltintoReplicatorovertheprevious3major releasesand7yearsofdevelopmenteffort.TheReplicatorversion4performance featuresetnowincludesthefollowingcomponents:

1.MultipleReplicationEngineDeployment

2. Multi-threaded Replication Queue Architecture

3.CustomizableReplicationInterval

4.ConfigurablePackageEventCount

5.ConfigurablePackageEventProcessingDuration

6.ReplicationPackageCompression

7.HardwareNetworkCompressionSupport

8.SoftwarePackageCompression

9.RemoteDifferentialCompression(RDC)

10.IndependentControlsforInboundProcessingandOutboundEventProcessing

11.SelectiveStructureReplication

INTRODUCTION

2

Page 3: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

3

BENCHMARKING AT THE MICROSOFT TECHNOLOGY CENTER

12.Rule-basedItem-levelContentReplication

13.SelectableReplicationEvents

14.SharedReplicationPackages

15.PackageDatabaseCaching

16.ConfigurableReplicationMonitorUpdateLevel

Thisdocumentprovidesadditionaldetailed informationon theReplicatorperformance featureset.Thefirstsectiondescribeseachcomponentof theReplicatorperformance featureset.Thesecondsection includesin-depthdiscussionsofeachfeatureaswellasnewbenchmarktestingperformedattheMicrosoftTechnologyCenter inNewYork,NY and theMetalogixNetwork Test Lab. The final section of the document analysesReplicator from an operational point-of-view.

InAugust2010,withthesupportoftheMicrosoftTechnologyCenterinNewYork,NY,theMetalogixMassiveScalabilityTeamexecutedseveralperformanceteststoexerciseapre-releaseversionofMetalogixReplicatorversion4.TheScalabilityTeamconfiguredthelargestknownDistributedSharePoint2010environmentintheworldforthesetests:

61SharePointfarmsconnectedintoasinglereplicationnetwork

Microsoft Hyper-V server virtualization technology

65SharePoint2010Windows2008virtualservers

16Windows2008R2hostphysicalservers

128GBofphysicalRAM

1.2TBofphysicaldiskstorage

Page 4: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

4

TheReplicator version4performance feature set builds uponmany components that havebeenbuilt intoReplicator over the previous 3major releases and 7 years of development effort. The new or significantlyimprovedversion4featuresaremarkedwithanasterisk.

Figure1.MicrosoftTechnologyCenter,NewYorkBenchmark:MetalogixConnectfor Replicator Visualization

 

REPLICATOR PERFORMANCE AND SCALABILITY FEATURES

Apre-releaseversionofMetalogixConnectforReplicatorwasusedtovisualizeandreportonthisconfiguration.Thefullydeployed61SharePointfarmenvironmentisdepictedinFigure1.

Page 5: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

5

Feature DescriptionReplicator Multi-EngineDeployment

ReplicatorEnterpriseEditionenablestheReplicatorEnginetobedeployedonmultiplewebfront-endsineachfarm.Withmultiplewebfront-endshostingReplicatorEngineinafarm,Replicatorprovidesahigheravailabilityandhigherperformancereplicationsolution;inadditiontosupportingincreasedscalability.

Multi-Threaded Replica-tion Queue Architecture

Replicator’sReplicationEnginearchitectureisbasedonamulti-threadedreplica-tionqueuearchitecturethatsupportsparallelReplicationPackageprocessing.

CustomizableReplicationInterval

Replicatorsupportsseveralreplicationschedulingpolicies:Immediately,Interval,Daily,Weekly,MonthlyandManual.

ConfigurablePackageEvent Count

Bydefault,ReplicatorincludestheabilitytogrouporbatchmultipleReplicationEventsintoasingleReplicationPackageforprocessingandtransfertotheTargetWebAp-plication.ThisConfigurablePackageEventCountdeterminesthemaximumnumberofEventsthatcanbepackagedintoasingleReplicationPackage.ThisenablesReplica-tortobetunedfordifferentlevelsofreal-timereplication,wideareanetworkperfor-mancecharacteristicsandavailableInboundandOutboundEventProcessingmemoryand CPU processing resources.

ConfigurablePackageEvent Processing Duration

SimilartotheConfigurablePackageEventCountfeature,theConfigurablePack-age Event Processing Duration allows control over how many Replication Events are groupedorbatchedtogetherintoasingleReplicationPackagebasedonelapsedOutboundEventProcessingtime.ThissettingenablesReplicatorlocalserverresourcerequirementstobetunedforoptimalperformance.

ReplicationPackageCompression

Replicator supports several forms of software compression in addition to support for andcompatibilitywithvarietyofhardwarenetworkcompressiondevices.Replicatorcan use either software or hardware compression to reduce the amount of Replication Packagedatatransferredoverawideareanetwork.

SoftwarePackageCompression

ReplicatorsupportsZIPsoftwarepackagecompressionandacustomimplementationofMicrosoft’sRemoteDifferentialCompression(RDC)thatishighlyoptimizedfortheone-wayandbi-directionalreplicationofSharePointdata.

HardwareNetworkCompression Support

Whenhardwarenetworkcompressiondevices,bandwidthoptimizationappliances,ornetworkacceleratorsolutionsareavailable(suchastheRiverbed®Steelhead®Appli-ance),Replicatorcanbeconfiguredtominimizetheserverresourcesusedforsoftwarecompressionandmaximizetheeffectiveofthenetworkcompressiondeviceorappli-cation.

RemoteDifferentialCompression(RDC)*

Metalogix Replicator supports a custom implementation of Microsoft’s RemoteDifferentialCompression(RDC)technologythatMetalogixhasoptimizedtoreducethetotalamountofReplicationPackagedatathatneedstobetransferredbe-tweentheSourceWebApplicationtotheTargetWebApplicationduringreplication.

IndependentOutboundEventandInboundEventProcessing Controls

ReplicatorsupportsindependentcontrolsforOutboundEventProcessingandIn-boundEventProcessingattheWebApplicationandReplicationConnectionlevelsforaSourceWebApplicationandTargetWebApplication.Inaddition,replicationforaparticularWebApplicationcanbeconfiguredasone-wayorbi-directional.

Page 6: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

6

Feature DescriptionSelective Structure Replication

Bydefault,manycustomersusewebapplication-to-webapplicationreplication – replicating the entire content of each site collection inonewebapplicationtothesecondwebapplication.SelectivestructurereplicationenablestheSharePointAdministratortoselectaspecificsubsetoftheSourceWebApplicationstructuretobereplicatedtotheTargetWebApplication.

Rule-basedItem-levelContent Replication

Rule-baseditem-levelcontentreplicationusestheMetalogixReplicatorRulesEngineto process custom rule sets to determine if an item in a particular list or document libraryshouldbereplicated.

SelectableReplicationEvents

ReplicatorsupportsselectablereplicationofchangesthatoccurinaSharePointWebApplication,SiteCollection,WebSite,ListorDocumentLibrary.Thedifferenttypesofindividual changes are called Replication Events. Replication Events are categorized intohigher-levelEventGroups.ByselectingwhichEventsneedtobereplicatedandwhichEventsdon’tneedtobereplicated,theSelectableReplicationEventsfeatureprovidestheSharePointAdministratorwithfine-grainedcontrolovertheEventsprocessingduringInboundEventProcessing,PackageTransferandOutboundEventProcessing; which in turn help improve overall Replicator performance.

Shared Replication Packages*

InReplicatorversion3,aseparateReplicationPackagewascreatedforeachoutboundReplicationConnection.TheSharedReplicationPackagefeatureinReplicatorversion4eliminatesthetimerequiredtocreateaseparatePackageforeachoutboundReplicationConnection–asinglesharedPackagecontainingthebatchofReplicationEventsiscreated.DependingonthenumberofEventsinthepackage,thesizeandtypeoftheSharePointchange,andthenumberofoutboundReplicationCon-nections,thiscansignificantlyreduceOutboundEventProcessingtimeandresources.

PackageDatabaseCaching*

InReplicatorversion4,uponacceptanceofthetransferofanInboundReplicationPackage,ReplicatorimmediatelycachesthemetadataforeachReplicationEventintheReplicatorConfigurationdatabase.ThisimprovestheperformanceoftheReplica-torEnginebyeliminatingtheneedtorepeatedlyaccesstheserializedPackagedata;especially in SharePoint farms that host multiple Replicator Engines.

ConfigurableReplicationMonitorUpdateLevel*

DuringnormalInboundEventProcessing,Replicator,runningonaTargetWebApplication,willmakeseveralcallstotheReplicatorWebServicerunningontheSourceWebApplicationtokeeptheMonitorReplicationreportsontheSourceWebApplicationascurrentaspossible.TohelpminimizetheQueuedItemupdatenet-worktrafficbetweentheSourceandTargetWebApplications,Replicatorversion4.1supportsanewReplicationConnectionpropertycalled“quietmode”.

Improved Initial ReplicationSupport*

ThenewBackupModefeatureprovidesamoreefficientmethodofreplicatingalargesitecollectionorhierarchyofwebsites.UsingBackupMode,ReplicatorusestheSharePoint import and export operations to create a single archive of the entire group ofwebsites,andqueuesthereplicationofthisarchiveasasingleReplicationEvent.

*NeworsignificantlyimprovedReplicatorversion4feature

Page 7: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

7

ReplicatorEnterpriseEditionenablestheReplicatorEnginetobedeployedonmultiplewebfront-endsineachfarm.Withmultiplewebfront-endshostingReplicatorEngineinafarm,Replicatorprovidesahigheravailabilityandhigherperformance replicationsolution; inaddition tosupporting increasedscalability. Ifoneengine isstopped, the other engines are still operational.

Thefollowingisbasedonaversion4benchmarkperformedattheMicrosoftTechnologyCenterinNewYorkusingahub-and-spoketopologyconnecting61SharePoint2010farms.Thefigureshows:

Multiplereplicationenginesprocessingandtransferringmultiplepackagessimultaneously

MultiplethreadsperenginesupportingparallelReplicationPackageprocessingandtransfer

Smooth scaling of the replication processing under heavy load

Replicator’sReplicationEnginearchitectureisbasedonamulti-threadedreplicationqueuearchitecturethat supports parallelReplicationPackageprocessing.ReplicatorStandardEdition supports 4 threadsperreplicationenginewithasingleactiveenginerunningonasinglewebfront-endperSharePointfarm.ReplicatorEnterpriseEditionsupports10toanunlimitednumberofthreadsrunningonasinglereplicationenginerunningonanunlimitednumberofwebfront-endsperSharePointfarm.

Figure2.MultipleEngine,MultipleThreadPerformanceandScalability showsReplicatorEnterpriseEditionrunningon4webfront-endsconfiguredtoperformparallelReplicationPackageprocessingandtransfer.

Figure2.MultipleEngine,MultipleThreadPerformanceandScalability

REPLICATOR MULTI-ENGINE DEPLOYMENT

MULTI-THREADED REPLICATION QUEUE ARCHITECTURE

 

Page 8: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

8

CUSTOMIZED REPLICATION INTERVAL

CONFIGURABLE PACKAGE EVENT COUNT

Replicatorsupportsseveraldifferentreplicationschedulingpolicies:

Immediately(defaultreal-time,event-drivenreplicationpolicy)

Interval(seconds,minutes,hours)

Daily

Weekly

Monthly

Manual

Bydefault,ReplicatorincludestheabilitytogrouporbatchmultipleReplicationEventsintoasingleReplicationPackageforprocessingandtransfertotheTargetWebApplication.ThisConfigurablePackageEventCountdetermines themaximumnumberofEvents that canbepackaged into a singleReplicationPackage. ThisenablesReplicator to be tuned for different levels of real-time replication,wide area network performancecharacteristicsandavailableInboundandOutboundEventProcessingmemoryandCPUprocessingresources.

AhigherPackageEventCountwillresultinfewer,largerPackagesbeingcreatedwhichcanincreaseoverallperformance.TheWindowsBackground IntelligentTransferProcess (BITS)protocol usedbyReplicator forPackageTransferisdesignedforhighperformanceandreliability(restartable)withlargefiledownloadsaswellassmallerdownloads.Largerpackages,somewhatsurprisingly,canresultinbetterthroughputandperformanceinslowandunreliablewideareanetworkscenarios.

ThesesettingsenablefinegrainedcontroloverwhenOutboundEventProcessingandPackageTransferwilloccur.This in turnsallows theSharePointAdministrator tobettermanageSharePointserverandwideareanetworkresources.

Figure3.MapFamilyReplicationSchedule

 

Page 9: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

9

SimilartotheConfigurablePackageEventCountfeature,theConfigurablePackageEventProcessingDurationallowscontroloverhowmanyReplicationEventsaregroupedorbatchedtogether intoasingleReplicationPackage based on elapsedOutbound Event Processing time. This setting enables Replicator local serverresourcerequirementstobetunedforoptimalperformance.

Replicatorsupportsseveral formsofsoftwarecompression inadditiontosupportforandcompatibilitywithvarietyofhardwarenetworkcompressiondevices.ReplicatorcanuseeithersoftwareorhardwarecompressiontoreducetheamountofReplicationPackagedatatransferredoverawideareanetwork.

ThefollowingsectionsdescribethebenefitsofReplicator’ssupportforhardwarenetworkcompressionandsoftwarepackagecompression.Replicator’shighlyoptimizedimplementationofMicrosoft’sRemoteDifferentialCompression(RDC)isalsodescribed.

When hardware network compression devices, bandwidth optimization appliances, or network acceleratorsolutions are available (such as the Riverbed® Steelhead® Appliance), Replicator can be configured tominimizetheserverresourcesusedforsoftwarecompressionandmaximizetheeffectivenessofthenetworkcompression device or application. A sample deployment is illustrated in Figure 4. Metalogix Replicator and Riverbed®Steelhead®ApplianceCompressedReplicationPackageSolution.

Figure4.MetalogixReplicatorandRiverbed®Steelhead®ApplianceCompressedReplicationPackageSolution

CONFIGURABLE PACKAGE EVENT PROCESSING DURATION

REPLICATION PACKAGE COMPRESSION

HARDWARE NETWORK COMPRESSION SUPPORT

BranchOffice

LAN

WAN

Metalogix ReplicatorData Center

Metalogix Replicator on RSP

Page 10: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

10

ConfiguringReplicatortosupporthardwarenetworkcompressiondevicesavoidstheredundantandpotentiallyexpensiveoverheadofcompressingpackagesinsoftwarewhenhardwarecompressionisavailable.

SOFTWARE PACKAGE COMPRESSION

REMOTEDIFFERENTIALCOMPRESSION(RDC)

ReplicatorsupportsZIPsoftwarepackagecompressionandacustomimplementationofMicrosoft’sRemoteDifferential Compression (RDC) that is highly optimized for the one-way and bi-directional replication ofSharePoint data.

MetalogixReplicatorsupportsacustomimplementationofMicrosoft’sRemoteDifferentialCompression(RDC)technologythatMetalogixhasoptimizedtoreducethetotalamountofReplicationPackagedatathatneedstobetransferredbetweentheSourceWebApplicationtotheTargetWebApplicationduringreplication.

“Remote Differential Compression (RDC) allows data to be synchronized with a remote sourceusingcompressiontechniquestominimizetheamountofdatasentacrossthenetwork.RDCissuitableforapplicationsthatmovedataacrossawideareanetwork(WAN)wherethedatatransmissioncostsoutweightheCPUcostofsignaturecomputation.RDCcanalsobeusedonfasternetworksiftheamountofdatatobetransferredisrelativelylargeandthechangestothedataaretypicallysmall.”1MorespecificdetailsonRDCcanbefoundinAppendixA–AboutWindowsRemoteDifferentialCompression.

Metalogix’sDynamicDifferentialCacheimprovementstoMicrosoftRDCuseserver-sidepre-processingandpost-processingtosignificantlyreducetheamountofSharePointdatathatistransferredoverthewideareanetwork.

DuringperformancetestingintheMetalogixNetworkTestLab,usingreal-lifeandsimulatedtestdata(SharePointtasks,multi-leveldocumentlibraries,Office2003documents,Office2007documents,Office2010documentsaswellasotherdesktopfileformatssuchasAutoCADdrawings,photosandimages),MetalogixReplicatorRDCreducedthetotalnetworktrafficbymorethan90%comparedtoZIPcompression.

Page 11: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

INDEPENDENT OUTBOUND EVENT AND INBOUND EVENT PROCESSING CONTROLS

SELECTIVE STRUCTURE REPLICATION

RULE-BASED ITEM-LEVEL CONTENT REPLICATION

ReplicatorsupportsindependentcontrolsforOutboundEventProcessingandInboundEventProcessingattheWebApplicationandReplicationConnectionlevelsforaSourceWebApplicationandTargetWebApplication.Inaddition,replicationforaparticularWebApplicationcanbeconfiguredasone-wayorbi-directional.

Whenonlyone-wayreplication isrequired,performance isoptimizedwhenEventprocessing isdisabledontheappropriateWebApplication.Inthebi-directionalreplicationscenario,separateReplicationConnectionsenableseparatereplicationpoliciestobeusedfortheoutboundandinboundconnections.

Bydefault,manycustomersusewebapplication-to-webapplicationreplication–replicatingtheentirecontentofeachsitecollection inonewebapplicationtothesecondwebapplication.SelectivestructurereplicationenablestheSharePointAdministratortoselectaspecificsubsetoftheSourceWebApplicationstructuretobereplicatedtotheTargetWebApplication.

Forexample:

One or more site collections

Rootwebsite

Anyindividualchildwebsite

Anychildwebsiteandsubhierarchyofthechildwebsite’schildwebsites

Individuallistsand/ordocumentlibrariesinawebsite.

Selectivestructurereplicationcanbeusedtosignificantlyreducethevolumeofreplicationdatabyselectingthespecificsitecollectionsandwebsitestobereplicated.

Rule-based item-levelcontent replicationuses theMetalogixReplicatorRulesEngine tooptionallyprocesscustomrulesetstodetermineifaniteminaparticularlistordocumentlibraryshouldbereplicated.

Althoughrules-basedcontentreplicationisnormallyconsideredasecurityandcontentpromotionfeature,rulesetscanalsobeusedtoblockthereplicationoflargerfilesorotherfilesbasedoneachitem’smetadatavaluesand improve performance.

11

1Source:http://msdn.microsoft.com/en-us/library/aa372948(v=VS.85).aspx)

Page 12: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

SELECTABLE REPLICATION EVENTS

SHARED REPLICATION PACKAGES

12

Replicator supports selectable replication of changes that occur in a SharePoint Web Application, SiteCollection,WebSite,ListorDocumentLibrary.ThedifferenttypesofindividualchangesarecalledReplicationEvents. Replication Events are categorized into higher-level Event Groups. By selecting which Events need tobereplicatedandwhichEventsdon’tneedtobereplicated,theSelectableReplicationEventsfeature provides the SharePoint Administrator with fine-grained control over the Events processing duringInboundEventProcessing, Package Transfer andOutboundEventProcessing;which in turn help improveoverall Replicator performance.

Figure5.MetalogixReplicatorReplicationEventshighlightstheReplicationEventssectionof theConfigureMapFamilywebpagefromReplicatorCentralAdministration.EntireEventGroupsorindividualEventscanbeenabledordisabled.Thisfigureshowsthedefaultsettings.

InReplicatorversion3,aseparateReplicationPackagewascreatedforeachoutboundReplicationConnection.The Shared Replication Package feature in Replicator version 4 eliminates the time required to create aseparatePackageforeachoutboundReplicationConnection–asinglesharedPackagecontainingthebatchofReplicationEvents iscreated.Dependingon thenumberofEvents in thepackage, thesizeand typeoftheSharePointchange,andthenumberofoutboundReplicationConnections, thiscansignificantly reduceOutboundEventProcessingtimeandresources.

Figure6.MicrosoftTechnologyCenterBenchmark:SharedReplicationPackageOutboundProcessingTimeillustrateshowtheelapsedOutboundProcessingtimeforthefirstPackage(greenportionoffirstbarinthechart)wassignificantlyreducedfortheremaining59instancesofthePackagereplicatedtoTargetWebApplication2through60.

Replication version4.1 also includesnewWebSiteReplicationEvents for replicating SharePoint 2010’ssocialnetworkingfeatures:ratings,commentsandkeywordtags.

Figure 5. Metalogix Replicator Replication Events

 

Page 13: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

PACKAGE DATABASE CACHING

CONFIGURABLE REPLICATION MONITOR UPDATE LEVEL

In Replicator version 4, upon acceptance of the transfer of an Inbound Replication Package, ReplicatorimmediatelycachesthemetadataforeachReplicationEvent in theReplicatorConfigurationdatabase.ThisimprovestheperformanceoftheReplicatorEnginebyeliminatingtheneedtorepeatedlyaccesstheserializedPackagedata;especiallyinSharePointfarmsthathostmultipleReplicatorEngines.

PackageDatabaseCachingspecificallyappliestoInboundEventProcessingontheTargetWebApplicationServerandimprovesoverallInboundEventProcessingbyreducingthenumberoffilesystemaccesses,reducingthevolumeofdatareadfromthefilesystemandloweredfiledecompressionCPUusage.

DuringnormalInboundEventProcessing,Replicator,runningonaTargetWebApplication,willmakeseveralcallstotheReplicatorWebServicerunningontheSourceWebApplicationtokeeptheMonitorReplicationreportsontheSourceWebApplicationascurrentaspossible.

TohelpminimizetheQueuedItemupdatenetworktrafficbetweentheSourceandTargetWebApplications,Replicatorversion4.1supportsanewReplicationConnectionpropertycalled“quietmode”.Itcanbeconfigured through theConfigureReplicationConnectionadministrationpagewhere it is referred toas theConfigurableReplicationMonitorUpdate Level. This new setting has 2 values:Normal andMinimal (quitemode).

DuringperformancetestingintheMetalogixNetworkTestLabreplicating10001KBfiles,thenetworktrafficwasreducedby55%andthenumberofHTTPPUToperationswasreducedby60%.

Figure6.MicrosoftTechnologyCenterBenchmark:SharedReplicationPackageOutboundProcessingTime

13

 

Page 14: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

ThenewBackupModefeatureprovidesamoreefficientmethodofreplicatingalargesitecollectionorhierarchyofwebsites.UsingBackupMode,ReplicatorusestheSharePointimportandexportoperationstocreateasinglearchiveoftheentiregroupofwebsites,andqueuesthereplicationofthisarchiveasasingleReplicationEvent.

BackupMode significantly improves the performance and reliability of a large initial replication: instead ofcreatingandtransferring10,000sofqueueditemsandpackagesinaverylargeSharePointenvironment,onlyafew(large)Packagesneedtobepackagedandtransferred.

Inaddition,ifRDCcompressionisenabled,theamountofdatatransferredoverawideareanetworkwillbereduceddramatically.Whenreplicatingtomultiplelocations,onlyonePackageiscreatedforalldestinationswiththeversion4SharedReplicationPackagefeature.

Figure 7. Microsoft Technology Center Windows Server 2008 R2 Hyper-V Host Environment: MetalogixReplicatorBenchmark

InAugust2010,MetalogixwasinvitedtotheMicrosoftTechnologyCenter(MTC)inNewYork,NYtomeasuretheperformanceandscalabilityofanearlyreleaseofMetalogixReplicatorforSharePointversion4.0.

WorkingcloselywiththeMTCtechnicalstaff,theMetalogixMassiveScalabilityTeamdeployedandconfiguredtheworld’slargestSharePoint2010DistributedSharePointenvironment,measuredintermsofthenumberof independent SharePoint farms. Eleven large scale physical servers from Dell and HP running Microsoft WindowsServer2008R2wereusedtodeployandconfigure85WindowsServer2008

R2 virtual machines. Microsoft Windows Hyper-V was used for the operating system virtualization solution. The entireconfigurationwasdeployedandmanagedusingMicrosoftWindowsVirtualMachineManager.

IMPROVED INITIAL REPLICATION SUPPORT

MICROSOFT TECHNOLOGY CENTER PERFORMANCE AND SCALABILITY BENCHMARK

14

HUB Servers

64GB

x 16

Core

s

Hype

r-V H

osts

3 x 200GB LUNs 200GB LUN 200GB LUN 200GB LUN

x 8

32GB

x 16

Core

s

32GB

x 16

Core

s

32GB

x 16

Core

s

VPN VMM RouterAD/DNS LabWorkstations

TARGET Servers TARGET Servers TARGET Servers

Page 15: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

15

Fivevirtualmachinesrunningon16processorcoreswithaccessto64GBofRAMand600GBofdiskstoragewereusedtohost4WebFront-EndServersand1SQLServer2008databaseserverintheHubSharePoint2010farm.Eighty(80)additionalvirtualmachinesrunningon96processorcoreswithaccessto320GMofRAMand2TBofdiskstoragewereusedtohost60single-serverSharePoint2010farms.ThespecificconfigurationoftheWindowsServer2008R2physicalhostscanbefoundinTable1.MicrosoftTechnologyCenterWindowsServer2008R2Hyper-VHostEnvironment:VirtualMachineConfiguration.

Table1.MicrosoftTechnologyCenterWindowsServer2008R2Hyper-VHostEnvironment:VirtualMachineConfiguration

Thefollowingperformancemonitoringtoolswereusedduringthebenchmark:

Windows Performance Monitor

Metalogix Replicator Replication Monitor

Metalogix Replicator Replication Status

Metalogix Connect for Replicator

Custom Replication Data Collection PowerShell Scripts

CustomReplicatorDeployment,Configuration,andManagementPowerShellScripts

CustomSharePointBenchmarkingPowerShellScripts

HostType

No.Hosts

Cores RAM(GB)

No.LUN

LUNSize(GB)

TotalCores

TotalRAM

TotalStorage(GB)

VMRAM

No. VM

Hub 1 16 64 3 200 16 64 600 - 5

Targets 1 16 32 1 200 16 32 200 4 8

Targets 1 16 32 1 200 16 32 200 4 8

Targets 8 8 32 8 200 64 256 1600 4 64

Totals 11 112 384 2600 12 85

Page 16: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

DuringthedesignanddevelopmentofMetalogixReplicatorversion4,eachnewperformancebuildwastestedandmeasuredalong40dimensions,including:

CPU usage

Availablememory

Workingset

Diskaccess

Networkaccess

Networkutilization

HTTPWebServerperformance

Theconfigurationof theMetalogixNetworkTest Lab is depicted in Figure 8.MetalogixNetworkTest Lab:PerformanceBenchmarking.IncontrastwithvirtualizedMicrosoftTechnologyCenterenvironmentusedforthemassivescalabilityandperformancebenchmarking,theMetalogixNetworkTestLabrunsentirelyonphysicalhardware.

Figure8.MetalogixNetworkTestLab:PerformanceBenchmarking

METALOGIX NETWORK TEST LAB PERFORMANCE BENCHMARKING

16

RDC1 SQL Server

Phys

ical H

osts

RDC1 WFE

WAN Simulator

AD/DNS Lab Workstations

RDC2 WFE RDC2 SQL Server

Page 17: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

ThekeytounderstandingReplicatorperformanceandscalability fromaday-to-dayoperationalperspectiveinvolvesunderstandinghowthebasiccomponentsoftheReplicatorPipelinecaptureandpackageReplicationEvents,transferpackagesfromtheSourceWebApplicationtotheTargetWebApplicationandfinallyupdatetheTargetWebApplicationwiththeneworchangedcontent.

OutboundEventProcessing is responsible forcapturingandrecordingReplicationEvents thatoccur in theSourceWebApplication.Forexample,whenausercreatesanewcontactinacontactlistinaSharePointWebsiteoreditsandsavesaMicrosoftWorddocumentthatisstoredinaSharePointdocumentlibrary,eachoftheseis captured and recorded as a Replication Event. Replication Events are captured and stored in a set of SQL ServertablesinaReplicatordatabase.OutboundEventProcessingiscontrolledbyReplicationMapswhichdeterminewhatEventsneedtobecaptured,packagedandtransferredtotheTargetWebApplication.GroupsofReplicationEventsarepackaged into two typesofmessagesorobjects:Queued ItemsandReplicationPackages.

Figure9.MetalogixReplicatorPipeline

The Replicator Pipeline is a detailed representation of how Replicator divides the overall replication process into 3activities:

CapturesachangeonaSourceWebApplication,packagesagroupofReplicationEventsandassociatedSharePointcontentintoaQueuedItemandcorrespondingReplicationPackage,

TransferstheQueuedItemandPackagefromtheSourcetotheTargetWebApplication,and

WhenreceivedandacceptedontheTargetWebApplication,appliesthechangesrepresentedbytheQueuedItemandPackagetotheTargetWebApplication.

These3activities,OutboundEventProcessing,PackageTransferandInboundEventProcessingareshowninthefollowingfigure.

UNDERSTANDING REPLICATOR PERFORMANCE AND SCALABILITY

METALOGIX REPLICATOR PIPELINE

 

17

Page 18: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

AQueued Item is a unit ofwork to be transferred to a TargetWebApplication for remote execution. TheReplicatorWebServiceonaTargetWebApplication iscalledtopushaQueuedItemfromtheSourceWebApplicationtoaTargetWebApplication.

A Replication Package is a collection of one or more Replication Events plus data about the SharePointinformation that is packaged in a format specific to theReplication Transport being used.When anEventisbeingprocessed,OutboundEventProcessingcalls theSharePointobjectmodel toextract thechangedSharePointinformationfromtheSourceWebApplicationcontentdatabase.

The Package Transfer activity is responsible for the transfer of Queued Items and Replication Packages(Packages)fromtheSourceWebApplicationtotheTargetWebApplication.PackageTransferistheprocessthatsitsbetweenOutboundEventProcessing(ontheSourceWebApplication)andInboundEventProcessing(ontheTargetWebApplications).PackageTransferinvokesReplicatorEnginesoftwarecomponentsonboththeSourceandTargetWebApplications.

InboundEventProcessingisresponsibleforprocessingtheQueuedItemsandPackagesreceivedandacceptedbytheTargetWebApplicationandapplyingthemtothesitecollections,Websites,listsanddocumentlibrariesinthewebapplication’scontentdatabase.TheQueuedItemsandPackagesareappliedtotheTargetWebApplicationcontentbasebycallingtheSharePointObjectModel.

ReplicatorperformanceisbestunderstoodbyanalyzingtheReplicatorPipelineactivitiesdepictedinFigure9.MetalogixReplicatorPipeline.ThesamevariablesthataffectperformanceofaSharePointenvironmentalsoaffectReplicatorperformance.ReplicatorwillperformoptimallyiftheunderlyingSharePointenvironmenthasbeenconfiguredforoptimalperformanceforagivenuserworkload.Inaddition,thereareReplicator-specificconfigurationsettingsandoptimizationthatcanfurtherimproveReplicatorperformance.

TheperformanceprofileofSharePoint,andeachoftheReplicatorPipelineactivities,isbasedonavarietyoffactors including:userworkload,documentfile formats,documentsizedistributions,SharePointserverandfarmconfiguration,databaseserverconfiguration,localareanetwork(LAN)characteristics,wideareanetwork(WAN)characteristics,ReplicationdatacompressionandreplicationmonitorsettingsaswellasotherReplicatorconfigurationsettings.

Userworkloadisdefinedbytwofactors:usercontentchangerateandusercontentchangesizedistribution.Qualitatively, the user content change rate is simply the rate at which users of a particular SharePoint environment makecontentchanges:creatingnewlistitems,editingandsavingexisting

items, creating or uploading new documents, editing and saving existing documents and, less frequently,

REPLICATOR PERFORMANCE

18

Page 19: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

creatingnewsitecollections,websites,listsanddocumentlibrariesormakingmetadataorsecuritychanges.

User content change size distribution refers to the size of the content changes and their distribution. Forexample,inadocumentcollaborationsolution,theaveragesizeofaWorddocumentmightbeafewmegabytesandtheaveragesizeofaPowerPointmightbemeasuredin10’sofmegabytes.TheamountofdataneededtorepresentanewSharePointcontactortaskitemwouldbemeasuredin10sor100sofkilobytes.Thedistributionofsizesvaryfromsmall1Kbytedocumentstomediafilesmeasuredin10sofmegabytes.

In Replicator version 4, each Map Family contains a Replication Schedule. A Replication Schedule controls how frequentlyReplicationEventsforaparticularReplicationConnectionareprocessedandpackagedintoQueuedItemsandReplicationPackages.EventProcessingandPackagingcanbescheduledtorun:

Immediately(real-timereplication)

Afteratimedinterval(afteraspecifiednumberofseconds,minutes,hoursordays)

Scheduleddaily,weekly,monthly

Manually initiated

Withtheexceptionofimmediatereplication,OutboundEventProcessingperformanceisprimarilydeterminedbytheReplicationSchedule,usercontentsizedistributionandSharePointsystemperformance.Forimmediatereplication,OutboundEventProcessingperformanceisalsoaffectedbytheusercontentchangerate.Fromasystemperformanceperspective,allOutboundEventProcessingoccurslocallyintheSharePointfarmthatishostingaparticularSourceWebApplication.OnthelocalSharePointfarm,serverRAM,diskorganization,LANnetworkconfigurationanddatabaseserverconfigurationarethegreatestdeterminantsofSharePointandReplicator system performance.

When Replicator is configured for immediate replication, Outbound Event Processing (Event capture andprocessing)isperformedinnearrealtime.

OUTBOUND EVENT PROCESSING

19

Page 20: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

PackageTransferreferstothepartoftheReplicatorPipelinethatisresponsiblefortransferringQueuedItems(and their correspondingPackages) fromaSourceWebApplication to a TargetWebApplication. PackageTransferperformanceisdeterminedbyanumberofdifferentfactors:

Userworkload

WANnetworkperformance

SourceWebApplicationsystemperformance

TargetWebApplicationsystemperformance

Replicator data compression

AQueuedItemandcorrespondingReplicationPackagerepresentsabatchofchangesthatusershavemadetotheSourceWebApplicationcontent.QueuedItemsarerelativelysmallinsizeandarepassedfromtheSourceWebApplicationtotheTargetWebApplicationbycallingtheTargetWebApplicationReplicatorWebService.PackagesareeitherdownloadedoruploadedtotheTargetWebApplicationusingtheBITSprotocolrunningoverHTTP/HTTPS.ThemajorityofthePackageTransfertimeisconsumedbythetransferofthePackage.ThisisbecausethePackagecontainsthatchangedSharePointcontentthatneedstobereplicatedfromtheSourceWebApplicationtotheTargetWebApplicationandthechangedcontent,inrelativeterms,maybequitelargedependingonwhetherthechangedcontentissimplelistitemora10MBPowerPointdocument.

Package Transfer performance be improved using Package compression settings that can be set usingReplicatorCentralAdministrationonaReplicationConnectionbyConnectionbasis.The followingPackagecompressionsettingsaresupportedinReplicator:

None

ZIP(default)

RDC(basedonMicrosoftRemoteDifferentialCompressiontechnology)

OncethecharacteristicsoftheQueuedItemandPackagestreamandthePackagecompressionsettingsareunderstood,thenextmajordeterminantofPackageTransferperformancearethecharacteristicsoftheWAN:

Availablebandwidth

Round-trip latency

Packetloss

Reliability

For example, satellite links typically have lower bandwidth, high latency and lower reliability compared toterrestriallinks;andexceptionallypoorerperformancecomparedtoafiberringinametropolitanareanetwork(MAN) or LAN.Metalogix’s advanced use of theMicrosoft BITS protocol in the design of theReplicatorPackageTransferprocessallowsReplicatortofunctionextremelywelloverhighperformanceconnectionsaswellasoversloworunreliablesatellite,ship-to-shoreandbattlefieldnetworks.

PACKAGE TRANSFER

20

Page 21: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

IfReplicatorisinstalledandconfiguredonmultipleSharePointfarmsconnectedbyahigh-performancenetwork,thePackageTransferprocesswillperformonanearrealtimebasisbetweenanypairofSourceandTargetWebApplications.

InboundEventProcessingbeginsonceaQueuedItemanditscorrespondingPackagehavebeentransferredtotheTargetWebApplicationbythePackageTransferprocess.InboundEventProcessingexecutesentirelywithintheSharePointfarmfortheTargetWebApplication.

InboundEventProcessingperformanceisdeterminedbytherateatwhichtheReplicatorServicecanprocessthelocalQueuedItemsandtheircorrespondingPackages,decompressthePackagesandapplythecontentchangestotheTargetWebApplicationbycallingtheSharePointObjectModel.TheInboundEventProcessingperformanceisaffectedbythefollowingfactors:

Userworkload

SharePointserverandfarmconfiguration

Databaseserverconfiguration,

Localareanetwork(LAN)characteristics

Subject to theaboveconstraints,Replicator InboundEventProcessing isable tooperateatnear real timeperformance.

INBOUND EVENT PROCESSING

21

Page 22: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

Replicator Central Administration provides detailed replication monitoring, replication status and replication eventprocessinghistoryreportsformonitoringtheReplicatorPipeline.ThefollowingtwofiguresareexamplesofthestatisticsavailablefromtheReplicationStatusreportforasampleintranetSourceWebApplicationthatisbeingreplicatedtoanextranetTargetWebApplication.

NOTE:Thefollowingfiguresarefromasmall-scaletestperformedonvirtualsingleserverSharePointfarmthatishostingboththeSourceandTargetWebApplications.Thestatisticsshowninthesefiguresareforillustrationpurposes only.

Intheabovefigure,theOutboundEventProcessingstatisticstellsus:

13Packages(andQueuedItems)wereprocessedrepresenting13batchesofReplicationEvents

TotalnumberofReplicationEvents=212

Totalsizeofthe13Packages(212Events)=13.8MB

Figure10.SampleOutboundEventProcessingStatistics

PERFORMANCE MONITORING

 

22

Page 23: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R  

Figure11.SampleInboundEventProcessingStatistics

Intheabovefigure,theInboundEventProcessingstatisticstellsus:

14Packages(andQueuedItems)wereprocessedrepresenting14batchesofReplicationEvents

TotalnumberofReplicationEvents=213

Totalsizeofthe14Packages(213Events)=11.82MB

NOTE:InboundandOutboundEventProcessingstatisticsareneverexpectedtomatch,duetoseveralfactors,includingsystemgeneratedQueuedItemsandPackages,retries,filteredPackagesandrejectedPackages.

TomonitorlargeDistributedSharePointenvironmentsconsistingofseveralfarms,MetalogixConnectfor

SharePoint or Metalogix Connect for Replicator is the monitoring tool of choice for these environments.

23

Page 24: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

ThefollowingtablemapsthekeyMetalogixReplicatorperformancefeaturestoeachphaseoftheReplicatorPipeline.

NOTE:TheperformancefeaturesthataffectOutboundEventProcessingalsotendtoenhancetheperformanceofthedownstreamReplicatorPipelineactivities:PackageTransferandInboundEventProcessing.Thereisonlyonefeature,PackageDatabaseCaching,whichonlyaffectsInboundEventProcessing.

Table2.MetalogixReplicatorPipelinePerformanceFeatures

REPLICATOR PIPELINE PERFORMANCE FEATURES

24

Feature AllPipelineActivities

OutboundProcessing

PackageTransfer

InboundProcessing

Replicator Multi-Engine Deployment YES

Multi-Threaded Replication Queue Architecture YES

ConfigurablePackageEventCount YES

ConfigurablePackageEventProcessingDuration

YES

IndependentOutboundEventProcessingandIn-boundEventProcessingControls

YES

Selective Structure Replication YES

Rule-basedItem-levelContentReplication YES

SelectableReplicationEvents YES

CustomizableReplicationInterval YES

SharedReplicationPackages YES

ReplicationPackageCompression YES YES

HardwareNetworkCompressionSupport YES YES

SoftwarePackageCompression YES YES

RemoteDifferentialCompression(RDC) YES YES

PackageDatabaseCaching YES

Page 25: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

TheMetalogixReplicatorarchitectureisbasedonanunlimitedmulti-master,point-to-pointreplicationnetworkmodelthatdoesnotrequireglobalsharingofanyconfigurationinformation.Arbitrarynetworktopologiesaresupportedincluding:

Serial

Parallel

Hub-and-spoke

Cascadinghub

Federatedhub

Cross domain

Mesh, and

Offline.

MetalogixConnect isadesktopapplication thatprovidesvisualization,monitoringandmanagementof thewide area SharePoint environment. It can be used to visualize the status of each farm and its associatedconnections.Connectisavailableasanadd-onfeaturefortheBasicandtheStandardEdition(andisincludedintheEnterpriseEdition).

MetalogixConnectforReplicatorprovidesthehighlyscalablereplicationnetworkmanagementtoolsneededtomonitorandmaintainthelargest(andsmallest)distributedSharePointenvironments.

Source:http://msdn.microsoft.com/en-us/library/aa372948(VS.85).aspx

RemoteDifferentialCompression(RDC)allowsdatatobesynchronizedwitharemotesourceusingcompressiontechniquestominimizetheamountofdatasentacrossthenetwork.

RDCisdifferentfrompatching-orienteddifferencingmechanisms,suchasBinaryDeltaCompression(BDC),thataredesignedtooperateonlyonknownversionsofasinglefile.BDCrequirestheservertohavecopiesofallversionsofthefile,anddifferencesbetweeneachpairofversionsarepre-computedsothattheycanbedistributedefficientlyfromaservertomultipleclients.

REPLICATOR SCALABILITY

UNLIMITED MULTI-MASTER, POINT-TO-POINT REPLICATION NETWORK MODEL

METALOGIX CONNECT FOR REPLICATOR

APPENDIX A – ABOUT WINDOWS REMOTE DIFFERENTIAL COMPRESSION

25

Page 26: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

RDCmakesnoassumptionsaboutfilesimilarityorversioning.Becausedifferencesbetweenfilesarecomputedonthefly,RDCisideallysuitedforsynchronizingfilesthataredifferentorhavebeenupdatedindependently.

RDC does not assume that the file data to be synchronized resides in physical files. Therefore, the RDCapplicationisresponsibleforperformingfileI/OonbehalfoftheRDClibrary.

Becauseitistransportindependent,RDCcanbeusedwithRPC,HTTP,orotherdesiredtransportmechanisms.TheRDCapplicationbearstheresponsibilityforchoosingtheappropriatetransportandperforminganyclientor server authentication that is required to support the transport's security model.

RDCissuitableforapplicationsthatmovedataacrossawideareanetwork(WAN)wherethedatatransmissioncostsoutweightheCPUcostofsignaturecomputation.RDCcanalsobeusedonfasternetworksiftheamountofdatatobetransferredisrelativelylargeandthechangestothedataaretypicallysmall.

RDCdividesafile'sdataintochunksbycomputingthelocalmaximaofafingerprintingfunctionthatis

computedateverybytepositioninthefile.Afingerprintingfunctionisahashfunctionthatcanbecomputedincrementally.Forexample,ifyoucomputethefunctionFoverarangeofbytesfromthefile,Bi...Bj,itshouldthenbepossibletocomputeF(Bi+1...Bj+1)incrementallybyaddingthebyteBj+1andsubtractingthebyteBi.Therangeofbytesfromthefile,Bi...Bj,iscalledthehashwindow.Thelengthofthiswindow,inbytes,iscalledthe hash window size.

TheRDClibrary'sFilterMaxsignaturegenerator"slides"thehashwindowacrosstheentirefilebyaddingthebyteattheleadingedgeandsubtractingthebyteatthetrailingedgeofthewindow.Meanwhile,thegeneratorcontinuallyexaminesthesequenceoffingerprintfunctionvaluesoveragivenrangeofbytes,calledthehorizonsize.Ifafingerprintfunctionvalueisalocalmaximumwithintherange,itsbytepositionischosenasa"cutpoint,"orchunkboundary.

After thefilehasbeendivided intochunks, thesignaturegeneratorcomputesastronghashvalue (anMD4hash),calledasignature,foreachchunk.Thesignaturescanbeusedtocomparethecontentsoftwoarbitrarilydifferentversionsofafile.

Becausethesizeofthesignaturefilegrowslinearlywiththesizeoftheoriginalfile,comparingverylargefilescanbeexpensive.ThiscostisreduceddramaticallybyapplyingtheRDCalgorithmrecursivelytothesignaturefiles.Forexample,iftheoriginalfilesizeis9GB,thesignaturefilesizewouldtypicallybeabout

81MB.IftheRDCalgorithmisappliedtothesignaturefile,theresultingsecond-levelsignaturefilesizewouldbeabout5.7MB.

REMOTE DIFFERENTIAL COMPRESSION ALGORITHM OVERVIEW

26

Page 27: Metalogix Replicator for SharePoint - WordPress.com · 4 supports SharePoint Foundation 2010, SharePoint Server 2010, OfficeSharePoint Server 2007 and Windows SharePoint Services

m e t a l o g i x . c o m W H I T E PA P E R

When developing an application that uses RDC, it is important to understand the following concepts and terminology.

InatypicalRDCscenario,aserverandaclienthavedifferentversionsofafile.(Thetermsclientandserverreferonlytothecomputers'rolesinthisscenario,nottheiroperatingsystems.)Theclient'scopyofthefileiscalledtheseedfile.Theserver'scopyiscalledthesourcefile.TheobjectiveoftheRDCapplicationistodownloadthefileupdatestotheclient,whichusesthemtoconstructatargetfilethatcombinestheupdatesfromthesourcefilewiththeunchangedcontentsfromtheseedfile.

TheRDCclientandservereachusetheRDClibrary'sFilterMaxsignaturegeneratortodividetheircopyofthefileintochunksandcomputeastronghash,calledasignature,foreachchunkoffiledata.Thus,theclienthasalistofsignaturesfortheseedfile,andtheserverhasalistofsignaturesforthesourcefile.Thesesignaturelistscanbecomputedonthefly,ortheycanbepre-computed.

Theclient initiatestheRDCprotocolbyrequestingthesourcesignature list fromtheserver.Thentheclientcompares each source signature against the signatures in its own seed signature list. If a source signature matchesaseedsignature,theclientalreadyhasthefiledataforthatsignature.Ifasourcesignaturedoesnotappearintheclient'slistofseedsignatures,theclientmustrequestthespecifiedchunk(offiledata)fromtheserver.

Theresultofcomparingthetwosignaturelistsisaneedslist,whichdescribeswhichchunksoffiledata,fromwhere(seedorsourcefile),areneededtoconstructthetargetfileontheclientcomputer.Eachentry intheneedslistiscalledaneedsblock.

Theclientiteratesthrougheachneedsblockandcopiesthespecifiedchunkofthesourceorseedfiledatatothetargetfile.Seedfiledataiscopiedlocally.Sourcefiledataisdownloadedfromtheserver.Themoresimilartheseedandsourcefilesare,thelessnetworkbandwidthisrequiredtocreatethetargetfile.

REMOTE DIFFERENTIAL COMPRESSION APPLICATION CONCEPTS

Metalogix is the trusted provider of innovative content lifecycle management solutions for Microsoft SharePoint, ExchangeandCloudplatforms.Wedeliverhigh-performancesolutionstoscaleandcost-effectivelymanage,move, store, archive and protect enterprise content. Metalogix provides global support to thousands ofcustomers and strategic partners and is a Microsoft Gold Partner, a managed partner in Microsoft’s High PotentialISVGroupandGSAprovider.MetalogixisaprivatelyheldcompanybackedbyInsightVenturePartnersand Bessemer Venture Partners.

ABOUT METALOGIX

METALOGIX5335WisconsinAveNW,Suite510,WashingtonDC20015

[email protected]|www.metalogix.com|1202.609.9100

27


Recommended