Kim Kimball, CCRE, Inc. [email protected] 1 AFS Near Real Time Mirrors with Unison Wouldn’t RW replication be nice

Kim Kimball, CCRE, Inc. dhk@Kim Kimball, CCRE, Inc. [email protected]

11

AFS Near Real Time AFS Near Real Time Mirrors with UnisonMirrors with Unison

Wouldn’t RW replication be Wouldn’t RW replication be nicenice


22

RoadmapRoadmap

Describe experiences with attempt to Describe experiences with attempt to provide “RW replication” using provide “RW replication” using synchronization mechanismssynchronization mechanisms

Not a Unison tutorialNot a Unison tutorial

RW Replication? I want it.RW Replication? I want it.


33

ConundrumConundrum

So the customer said …So the customer said …• ““My RW data must be available. For ever My RW data must be available. For ever

and for always.”and for always.”

So the customer settled for …So the customer settled for …• Mostly for everMostly for ever• Mostly alwaysMostly always• Mostly no loss on fail overMostly no loss on fail over


44

AdministratorAdministrator Hopes: customer will forgetHopes: customer will forget Tries: Tries:

• BK/RO, periodic updates, restore/convert to RWBK/RO, periodic updates, restore/convert to RW RW busy during cloning; scares customer RW busy during cloning; scares customer Restore/convert times lengthy for complex volumesRestore/convert times lengthy for complex volumes

• Misguided DIY effortMisguided DIY effort Recursion at mount pointsRecursion at mount points SlowSlow Too much development time to get it rightToo much development time to get it right

Wants:Wants:• Speed, reliability, simplicitySpeed, reliability, simplicity• No following/recursion at MPsNo following/recursion at MPs


55

Can provide?Can provide?

RWRW11 nearly identical to RW nearly identical to RW22 standby standby Fast failure detectionFast failure detection Fast fail overFast fail over Small data loss on fail overSmall data loss on fail over


66

Tool: UnisonTool: Unisonhttp://www.cis.upenn.edu/_bcpierce/unison

Dr. Benjamin Pierce, University of PennsylvaniaDr. Benjamin Pierce, University of Pennsylvania

“Unison shares a number of features with tools such as configuration management packages (CVS, PRCS, etc.), distributed filesystems (Coda, etc.), uni-directional mirroring utilities (rsync, etc.), and other synchronizers (Intellisync, Reconcile, etc). However, there are several points where it differs:


77

http://www.cis.upenn.edu/_bcpierce/unison

“Unison runs on both Windows (95, 98, NT, and 2k) and Unix (Solaris, Linux, etc.) systems. Moreover,

Unison works across platforms, allowing you to synchronize a Windows laptop with a Unix server, for example.

• Unlike a distributed filesystem, Unison is a user-level program: there is no need to hack (or own!) the kernel, or to have superuser privileges on either host.

• Unlike simple mirroring or backup utilities, Unison can deal with updates to both replicas of a distributed directory structure. Updates that do not conflict are propagated automatically. Conflicting updates are detected and displayed.

• Unison works between any pair of machines connected to the internet, communicating over either a direct socket link or tunneling over an rsh or an encrypted ssh connection. It is careful with network bandwidth, and runs well over slow links such as PPP connections. Transfers of small updates to large files are optimized using a compression protocol similar to rsync.

• Unison has a clear and precise specification, described below.

• Unison is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times, even in case of abnormal termination or communication failures.

• Unison is free; full source code is available under the GNU Public License. “


88

Mount pointsMount points

Unison is not AFS mount point awareUnison is not AFS mount point aware Given opportunity, Unison recursesGiven opportunity, Unison recurses Unison will cheerfully follow mount Unison will cheerfully follow mount

points into volumes you have no points into volumes you have no intention of synchronizingintention of synchronizing

Or into la la landOr into la la land


99

Unison: Mount pointsUnison: Mount points

““-ignore pathspec causes Unison to completely -ignore pathspec causes Unison to completely ignore paths that match pathspec (as well as ignore paths that match pathspec (as well as their children) “their children) “

Discover mount points in volume(s) Discover mount points in volume(s) to be synch’dto be synch’d

-ignore -ignore <path>/<mountpoint><path>/<mountpoint>


1010

Finding mount pointsFinding mount points $ $ bos exec angel -cmd "/usr/afs/bin/salvager /vicepc 536901891 -showmountsbos exec angel -cmd "/usr/afs/bin/salvager /vicepc 536901891 -showmounts““

[kim@angel ~]$ bos getlog angel SalvageLog | grep 536901891 | grep mountpoint[kim@angel ~]$ bos getlog angel SalvageLog | grep 536901891 | grep mountpoint 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./gco to 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./gco to

'#grand.central.org:root.cell.''#grand.central.org:root.cell.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./music.k to 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./music.k to

'%music.k.rw.''%music.k.rw.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./sw to '#sw.'05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./sw to '#sw.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./laroia.net to 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./laroia.net to

'#laroia.net:root.cell.''#laroia.net:root.cell.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./music.k.rw to 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./music.k.rw to

'%music.k.''%music.k.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./music.k.ro to 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./music.k.ro to

'#music.k.readonly.''#music.k.readonly.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./SHOWMOUNTTEST/V to 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./SHOWMOUNTTEST/V to

'#testvolV.''#testvolV.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint

./private/JPL/FileLogConcordance/Report to '#ConcordanceReport.vol.'./private/JPL/FileLogConcordance/Report to '#ConcordanceReport.vol.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./dir1/sw.s to '#sw.s.'05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./dir1/sw.s to '#sw.s.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./dir1/dir2/sw.r to 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./dir1/dir2/sw.r to

'#sw.r.''#sw.r.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./dir1/dir2/dir3/sw.q to 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint ./dir1/dir2/dir3/sw.q to

'#sw.q.''#sw.q.' 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint 05/22/2008 18:00:02 In volume 536901891 (user.k.kim) found mountpoint

./dir1/dir2/dir3/dir4/sw.p1 to '#sw.p1.'./dir1/dir2/dir3/dir4/sw.p1 to '#sw.p1.'


1111

Using outputUsing output

SalvageLogSalvageLog reports mount points reports mount points relative to root node of volumerelative to root node of volume

--ignoreignore <path>/<mount point> <path>/<mount point>


1212

Unison: SpeedUnison: Speed

First run – not so fastFirst run – not so fast• Builds signature fileBuilds signature file• Uses signatures for subsequent speedUses signatures for subsequent speed

Subsequent runs – Subsequent runs – • No deltas – very fastNo deltas – very fast• Used whole-file synch, still fastUsed whole-file synch, still fast• Can merge deltas using external appsCan merge deltas using external apps


1313

DirectionalityDirectionality

-force -force is used to force unidirectionality is used to force unidirectionality Seems safe – force synch from master to Seems safe – force synch from master to

NRTM -- butNRTM -- but• RWRW2 2 put into playput into play

• New files, New files, ff11 f f22 f f33 in RW in RW22

• Unidirectional sync from RWUnidirectional sync from RW11

• ff11 f f22 f f33 cheerfully deleted from RWcheerfully deleted from RW22

Bidirectional synch is saferBidirectional synch is safer Conflicts require manual intervention …Conflicts require manual intervention …


1414

ACLs, tokensACLs, tokens

The principal identified by the latter The principal identified by the latter must have must have rlidwk rlidwk permission on the permission on the formerformer

(Didn’t test (Didn’t test kk requirement.) requirement.)

Standard token lifetime disclaimerStandard token lifetime disclaimer


1515

ExperiencesExperiences Ignored “token lifetime disclaimer”Ignored “token lifetime disclaimer” Unison logs updates made; times; etcUnison logs updates made; times; etc Synch window varies with number of Synch window varies with number of

deltas; structure of volume; number of deltas; structure of volume; number of files/directoriesfiles/directories

Fail over:Fail over:• change volume referenced in mount pointchange volume referenced in mount point• call backs handle MP changecall backs handle MP change

MP must be reinterpreted – users must ‘exit’ and ‘re-MP must be reinterpreted – users must ‘exit’ and ‘re-enter’ volume if already ‘in’ volume and both vols are enter’ volume if already ‘in’ volume and both vols are on lineon line

• Used for moves in denigrated environmentUsed for moves in denigrated environment


1616

LimitationsLimitations Manual conflict resolution, unless directionality forcedManual conflict resolution, unless directionality forced External failure detection requiredExternal failure detection required Manual/scripted failover requiredManual/scripted failover required ‘‘Twiddle’ mount points to force failoverTwiddle’ mount points to force failover Failover may require user actionFailover may require user action Fast but unpredictable synch windowFast but unpredictable synch window Not integral to file system; requiresNot integral to file system; requires

• ACL-specific tokensACL-specific tokens• Mount points or other external configMount points or other external config• Significant scripting/codingSignificant scripting/coding• MaintenanceMaintenance

““Operator error” is likely, esp. over timeOperator error” is likely, esp. over time Not suitable for frequent use or large number of volumesNot suitable for frequent use or large number of volumes


1717

RW ReplicationRW Replication

Integral to AFSIntegral to AFS• Less error proneLess error prone

No external scripting/configurationNo external scripting/configuration Failover integral to client, no external ‘failure Failover integral to client, no external ‘failure

detection’ & manual failoverdetection’ & manual failover Volume-level synch – no mount point issuesVolume-level synch – no mount point issues Single implementation benefits allSingle implementation benefits all Transparent to end userTransparent to end user

Time saver – if available, would have used Time saver – if available, would have used and eliminated significant development and eliminated significant development and test timeand test time


1818

In the meantimeIn the meantime

This is much better than nothingThis is much better than nothing Works well enough for small number Works well enough for small number

of volumes, large number with of volumes, large number with enough effortenough effort


1919

Unison optionsUnison options --addprefstoaddprefsto xxxxxx file to add new prefs to file to add new prefs to --addversionnoaddversionno add version number to name of unison executable on server add version number to name of unison executable on server --autoauto automatically accept default actions* automatically accept default actions* --backupbackup xxxxxx add a pattern to the backup list add a pattern to the backup list --backupcurrentbackupcurrent xxxxxx add a pattern to the backupcurrent list -backupcurrentnot xxx add a pattern add a pattern to the backupcurrent list -backupcurrentnot xxx add a pattern

to the backupcurrentnot list to the backupcurrentnot list --backupdirbackupdir xxxxxx Directory for storing centralized backups Directory for storing centralized backups --backuplocationbackuplocation xxxxxx where backups are stored ('local' or 'central') where backups are stored ('local' or 'central') --backupnotbackupnot xxxxxx add a pattern to the backupnot list add a pattern to the backupnot list --backupprefixbackupprefix xxxxxx prefix for the names of backup files prefix for the names of backup files --backupsbackups keep backup copies of all files (see also 'backup') keep backup copies of all files (see also 'backup') --backupsuffixbackupsuffix xxxxxx a suffix to be added to names of backup files a suffix to be added to names of backup files --batchbatch batch mode: ask no questions at all batch mode: ask no questions at all --confirmbigdeletesconfirmbigdeletes request confirmation for whole-replica deletes request confirmation for whole-replica deletes --confirmmergeconfirmmerge ask for confirmation before commiting results of a merge ask for confirmation before commiting results of a merge --contactquietlycontactquietly Suppress the 'contacting server' message during startup Suppress the 'contacting server' message during startup --debugdebug xxxxxx debug module xxx ('all' -> everything, 'verbose' -> more) * debug module xxx ('all' -> everything, 'verbose' -> more) * --docdoc xxxxxx show documentation ('-doc topics' lists topics) show documentation ('-doc topics' lists topics) --fastcheckfastcheck xxxxxx do fast update detection (`true', `false', or `default') * do fast update detection (`true', `false', or `default') * --followfollow xxxxxx add a pattern to the follow list add a pattern to the follow list --forceforce xxxxxx force changes from this replica to the other force changes from this replica to the other --forcepartialforcepartial xxxxxx add a pattern to the forcepartial list add a pattern to the forcepartial list --groupgroup synchronize group synchronize group --heightheight nn height (in lines) of main window in graphical interface height (in lines) of main window in graphical interface


2020

--hosthost xxxxxx bind the socket to this host name in server socket mode bind the socket to this host name in server socket mode --ignoreignore xxxxxx add a pattern to the ignore list * add a pattern to the ignore list * --ignorecaseignorecase xxxxxx ignore upper/lowercase in filenames (`true', `false', or `default') ignore upper/lowercase in filenames (`true', `false', or `default') --ignorelocksignorelocks ignore locks left over from previous run (dangerous!) ignore locks left over from previous run (dangerous!) --ignorenotignorenot xxxxxx add a pattern to the ignorenot list add a pattern to the ignorenot list --immutableimmutable xxxxxx add a pattern to the immutable list add a pattern to the immutable list --immutablenotimmutablenot xxxxxx add a pattern to the immutablenot list add a pattern to the immutablenot list --keykey xxxxxx define a keyboard shortcut for this profile (in some UIs) define a keyboard shortcut for this profile (in some UIs) --killserverkillserver kill server when done (even when using sockets) kill server when done (even when using sockets) --labellabel xxxxxx provide a descriptive string label for this profile provide a descriptive string label for this profile --loglog record actions in file specified by logfile preference * record actions in file specified by logfile preference * --logfilelogfile xxxxxx Log file name * Log file name * --maxbackupsmaxbackups nn number of backed up versions of a file number of backed up versions of a file --maxthreadsmaxthreads nn maximum number of simultaneous file transfers maximum number of simultaneous file transfers --mergemerge xxxxxx add a pattern to the merge list add a pattern to the merge list --mountpointmountpoint xxxxxx abort if this path does not exist abort if this path does not exist --numericidsnumericids don't map uid/gid values by user/group names don't map uid/gid values by user/group names --ownerowner synchronize owner * synchronize owner * --pathpath xxxxxx path to synchronize * path to synchronize * --permsperms nn part of the permissions which is synchronized part of the permissions which is synchronized --preferprefer xxxxxx choose this replica's version for conflicting changes choose this replica's version for conflicting changes --preferpartialpreferpartial xxxxxx add a pattern to the preferpartial list add a pattern to the preferpartial list --pretendwinpretendwin Use creation times for detecting updates Use creation times for detecting updates --repeatrepeat xxxxxx synchronize repeatedly (text interface only) synchronize repeatedly (text interface only)


2121

--retryretry nn re-try failed synchronizations N times (text interface only) re-try failed synchronizations N times (text interface only) --rootroot xxxxxx root of a replica root of a replica --rootaliasrootalias xxxxxx Register alias for canonical root names Register alias for canonical root names --rsrcrsrc xxxxxx synchronize resource forks and HFS meta-data (`true', `false', or `default') synchronize resource forks and HFS meta-data (`true', `false', or `default') --rsyncrsync activate the rsync transfer mode activate the rsync transfer mode --selftestselftest run internal tests and exit run internal tests and exit --servercmdservercmd xxxxxx name of unison executable on remote server name of unison executable on remote server --showarchiveshowarchive showshow name of archive and 'true names' (for rootalias) of roots name of archive and 'true names' (for rootalias) of roots --silentsilent print nothing (except error messages) print nothing (except error messages) --socketsocket xxxxxx act as a server on a socket act as a server on a socket --sortbysizesortbysize list changed files by size, not name list changed files by size, not name --sortfirstsortfirst xxxxxx add a pattern to the sortfirst list add a pattern to the sortfirst list --sortlastsortlast xxxxxx add a pattern to the sortlast list add a pattern to the sortlast list --sortnewfirstsortnewfirst list new before changed files list new before changed files --sshargssshargs xxxxxx other arguments (if any) for remote shell command other arguments (if any) for remote shell command --sshcmdsshcmd xxxxxx path to the ssh executable path to the ssh executable --terseterse suppress status messages suppress status messages --testservertestserver exit immediately after the connection to the server exit immediately after the connection to the server --timestimes synchronize modification times synchronize modification times --uiui xxxxxx select user interface ('text' or 'graphic'); command-line only select user interface ('text' or 'graphic'); command-line only --versionversion print version and exit print version and exit --xferbycopyingxferbycopying optimize transfers using local copies, if possible optimize transfers using local copies, if possible


2222

??

Documents

Kim Kimball, CCRE, Inc. [email protected] 1 AFS Near Real Time Mirrors with Unison Wouldn’t RW replication be nice