1

A highly-available move operation for replicatedtrees and distributed filesystems

Martin Kleppmann, Dominic P. Mulligan, Victor B. F. Gomes, and Alastair R. Beresford

Abstract—Replicated tree data structures are a fundamental building block of distributed filesystems, such as Google Drive andDropbox, and collaborative applications with a JSON or XML data model. These systems need to support a move operation that allowsa subtree to be moved to a new location within the tree. However, such a move operation is difficult to implement correctly if differentreplicas can concurrently perform arbitrary move operations, and we demonstrate bugs in Google Drive and Dropbox that arise withconcurrent moves. In this paper we present a CRDT algorithm that handles arbitrary concurrent modifications on trees, while ensuringthat the tree structure remains valid (in particular, no cycles are introduced), and guaranteeing that all replicas converge towards thesame consistent state. Our algorithm requires no synchronous coordination between replicas, making it highly available in the face ofnetwork partitions. We formally prove the correctness of our algorithm using the Isabelle/HOL proof assistant, and evaluate theperformance of our formally verified implementation in a geo-replicated setting.

Index Terms—Conflict-free Replicated Data Types (CRDTs), formal verification, distributed filesystems, distributed collaboration

F

1 INTRODUCTION

MANY applications use a tree-structured data model.For example, most file systems are trees: directories

are branch nodes, and files are leaf nodes. Moreover, XMLand JSON documents are trees, and they are used in manyapplications to represent e.g. rich text (a tree of paragraphs,lists, figures, sections, etc.), vector graphics, CAD drawings,and many other types of data. Typically, a graphical userinterface allows a user to edit this information interactively,resulting in updates to the underlying XML/JSON struc-ture: adding or deleting nodes in the tree, and moving nodesfrom one position in the tree to another.

In distributed filesystems and collaborative multi-usersoftware, this tree is replicated across multiple nodes. Ifusers attempt to update the tree concurrently on differentreplicas, concurrency control is required. However, standardtechniques such as two-phase locking require synchronouscoordination between replicas. If we want the system tocontinue processing read and write requests even in thepresence of arbitrary network partitions (in other words, ifwe want availability and partition-tolerance in the sense of theCAP theorem [1]), then we must avoid such coordination.This is an important practical consideration since networkpartitions are common in many networks [2], [3].

We can use optimistic replication [4], which means thatany replica can make changes to the data without waitingfor communication with any other replicas; operations aresent to other replicas at some later time when a networkconnection is available. This approach has the advantageof enabling disconnected operation on mobile devices, andhigh availability in geo-replicated settings. It also improves

• M. Kleppmann and A.R. Beresford are with the University of Cambridge.

• D.P. Mulligan is with Arm Research, Cambridge, UK.

• V.B.F. Gomes is with Google (this work was done while at the Universityof Cambridge).

performance for end users because the duration of a userrequest is independent of any network latency.

Dropbox and Google Drive are widely-deployed exam-ples of optimistically replicated filesystems: they run a dae-mon on the user’s machine that watches a designated direc-tory for changes. The user can read and arbitrarily modifythe files on their local disk, even while their computeris offline. However, when the filesystem is concurrentlyupdated on different computers, Google Drive and Dropboxexhibit bugs in their concurrency control, as we show in §2.

In this paper we introduce a novel algorithm for han-dling concurrent updates to a replicated tree, such as anXML document or filesystem. It allows replicas to manipu-late the tree by creating nodes, deleting nodes, or movingsubtrees to a new location within the tree. We rule outbugs like those in Google Drive by formally proving ouralgorithm correct using the Isabelle/HOL proof assistant.

Our algorithm supports optimistic replication, allowingreplicas to temporarily diverge as they are updated, andensuring that they always converge towards a consistentstate. It is an example of a Conflict-free Replicated Data Typeor CRDT [5], and it guarantees a consistency model calledstrong eventual consistency [5], [6]. Our contributions are:

• We define a Conflict-free Replicated Data Type fortrees that allow move operations without any coordi-nation between replicas such as locking or consen-sus. As discussed in §2.3, this has previously beenthought to be impossible to achieve [7], [8].

• We formalise the algorithm using Isabelle/HOL [9],a proof assistant based on higher-order logic, andobtain a computer-checked proof of correctness. Inparticular, we prove that arbitrary concurrent modi-fications to the tree can be merged such that all repli-cas converge to a consistent state, while preservingthe tree structure. Our proof technique can also beapplied to other distributed systems, making it of

2

independent interest.• To demonstrate the practical viability of our ap-

proach, we refine the algorithm to an executableimplementation within Isabelle/HOL and prove theequivalence of the two. We extract a formally verifiedScala implementation from Isabelle and evaluate itsperformance with replicas across three continents.

• We perform experiments with Dropbox and GoogleDrive, and show that they exhibit problems thatwould be prevented by our algorithm.

2 WHY A MOVE OPERATION IS HARD

Applications that rely on a tree data model often need tomove a node from one location to another location withinthe tree, such that all of its children move along with it:

• In a filesystem, any file or directory can be movedto a different parent directory. Moreover, renaming afile or directory is equivalent to moving it to a newname without changing its parent directory.

• In a rich text editor, a paragraph can be turned intoa bullet point. In the XML tree this corresponds tocreating new list and bullet point nodes, and thenmoving the paragraph node inside the bullet point.

• In graphics software, grouping two objects corre-sponds to creating a new group node, and thenmoving the two objects into the new group node.

As these operations are so common, it is not obvious why amove operation should be difficult in a replicated setting. Inthis section we demonstrate some problems that arise withreplicated trees, before proceeding to our solution in §3.

2.1 Concurrent moves of the same nodeThe first difficulty arises when the same node is concur-rently moved into different locations on different replicas.This scenario is illustrated in Figure 1, where replica 1moves node A to be a child of B, while concurrentlyreplica 2 moves A to be a child of C . After the replicascommunicate, what should the merged state of the tree be?

If a move operation is implemented by deleting themoved subtree from its old location, and then re-creatingit at the new location, the merged state will be as shown inFigure 1a: the concurrent moves will duplicate the movedsubtree, since each move independently recreates the sub-tree in each destination location. We believe that this du-plication is undesirable, since subsequent edits to nodes inthe duplicated subtree will apply to only one of the copies.Two users who believe they are collaborating on the samefile may in fact be editing two different copies, which willthen become inconsistent with each other. In the rich texteditor and graphics software examples, such duplication isalso undesirable.

Another possible resolution is for the destination loca-tions of both moves to refer to the same node, as shown inFigure 1b. However, the result is a DAG, not a tree. POSIXfilesystems do not allow this outcome, since they do notallow hardlinks to directories.

In our opinion, the only reasonable outcomes are thoseshown in Figure 1c and 1d: the moved subtree appears

either in replica 1’s destination location or in replica 2’sdestination location, but not in both. Which one of thesetwo is picked is arbitrary, due to the symmetry betweenthe two replicas. The “winning” location could be pickedbased on a timestamp in the operations, similarly to the “lastwriter wins” conflict resolution method of Thomas’s writerule [10]. The timestamp in this context need not come froma physical clock; it could also be logical, such as a Lamporttimestamp [11].

We tested this scenario with file sync products Dropboxand Google Drive by concurrently moving the same direc-tory to two different destination directories.1 Dropbox ex-hibited the undesirable duplication behaviour of Figure 1a,while the outcome on Google Drive was as in Figure 1c/d.

2.2 Moving a node to be a descendant of itselfOn a filesystem, the destination directory of a move op-eration must not be a subdirectory of the directory beingmoved. For example, if b is a subdirectory of a, then theUnix shell command mv a a/b/ will fail with an error.This restriction is required because allowing this operationwould introduce a cycle into the directory graph, and sothe filesystem would no longer be a tree. This restriction isrequired in any other tree structure that supports a moveoperation.

In an unreplicated tree it is easy to prevent cycles beingintroduced: if the node being moved is an ancestor of thedestination node, the operation is rejected. However, in areplicated setting, different replicas may perform operationsthat are individually safe, but whose combination leads toa cycle. One such example is illustrated in Figure 2. Here,replica 1 moves B to be a child of A, while concurrentlyreplica 2 moves A to be a child of B. As each replicapropagates its operation to the other replica, a careless im-plementation might end up in the state shown in Figure 2a:A and B have formed a cycle, detached from the tree.

Another possible resolution is shown in Figure 2b: thenodes involved in the concurrent moves (and their children)could be duplicated, so that both “A as a child of B” and“B as a child of A” can exist in the tree. However, suchduplication is undesirable for the same reasons as in §2.1.

In our opinion, the best way of handling the conflictingoperations of Figure 2 is to choose either Figure 2c or 2d:that is, either the result of applying replica 1’s operation andignoring replica 2’s operation, or vice versa. Like in §2.1, thewinning operation can be picked based on a timestamp.

As before, we tested this scenario with Google Driveand Dropbox. In Google Drive, one replica was able tosuccessfully sync with the server, while the other replicadisplayed the “unknown error” message shown in Figure 3.The replica in an error state refused to sync the conflictingdirectory, and its filesystem state remained permanentlyinconsistent with the other replica. This error state persisteduntil the directories on the erroring replica were manually

1. Experiment setup: we installed the official Mac OS clients forDropbox and Google Drive on two computers, logged into the sameDropbox/Google accounts, and configured them to sync a directoryon the local filesystem. To test concurrent operations, we disconnectedboth computers from the Internet, performed a move operation on thelocal filesystem of each computer, then reconnected and waited forthem to sync.

3

Replica 1:

Replica 2:

root

A

a1 a2 a3

B C

root

A

a1 a2 a3

B C

root

B

A

a1 a2 a3

C

root

B C

A

a1 a2 a3

?

?

Move A to bea child of B

Move A to bea child of C

netw

ork

com

mun

icat

ion

(a) (b)

(c) (d)

root

B

A

a1 a2 a3

C

A′

a′1 a′2 a′3

root

B C

A

a1 a2 a3

root

B

A

a1 a2 a3

C

root

B C

A

a1 a2 a3

Fig. 1. Replica 1 moves A to be a child of B, while concurrently replica 2 moves the same node A to be a child of C. Boxes (a) to (d) show possibleoutcomes after the replicas have communicated and merged their states.

Replica 1:

Replica 2:

root

A

C

B

root

A

C

B

root

A

B C

root

B

A

C

?

?

Move B to bea child of A

Move A to bea child of B

netw

ork

com

mun

icat

ion

(a) root

A

B C

(b)root

A

B C

B′

A′

C ′

(c) root

A

B C

(d)root

B

A

C

Fig. 2. Initially, nodes A and B are siblings. Replica 1 moves B to be a child of A, while concurrently replica 2 moves A to be a child of B. Boxes(a) to (d) show possible outcomes after the replicas have communicated and merged their states.

Fig. 3. Error message produced by Google Drive Backup and Sync on Mac OS as a result of performing the operations shown in Figure 2, indicatinga bug in the underlying replication algorithm.

4

moved to match the state of the other replica. We havereported this bug to Google. On the other hand, Dropboxexhibited the duplication behaviour shown in Figure 2b.

2.3 Is a highly-available move operation impossible?

Najafzadeh et al. [7], [8] previously implemented a repli-cated filesystem with a move operation, and analysed thecase of concurrent move operations introducing a cycle.Using the CISE proof tool [12], [13] the authors confirmthat it is not sufficient for the replica that generates a moveoperation to check whether the operation introduces a cycle:like in Figure 2, two concurrent operations may be safeindividually, but introduce a cycle when combined.

Najafzadeh et al. propose two solutions to this problem:either to duplicate tree nodes, as in Figure 2b, or to executea synchronous locking protocol that prevents two moveoperations from concurrently modifying the same part ofthe tree. The downside of a locking protocol is that themove operation is no longer highly available in the presenceof network partitions, since it must wait for synchronouscommunication with other replicas or a lock server.

While these solutions are valid, the authors go on toclaim that “no file system can support an unsynchronisedmove without anomalies, such as loss or duplication” [8].We refute that claim in this paper: our algorithm doesnot perform any locking, coordination or synchronisationamong replicas, but it nevertheless ensures that the treeinvariants are always satisfied (in particular, it never in-troduces cycles), and it never duplicates or loses any treenodes. To our knowledge, our algorithm is the first to pro-vide all of these properties simultaneously. We give a precisespecification of our algorithm’s consistency properties in §4.

3 THE REPLICATED MOVE OPERATION

We now introduce our algorithm for a replicated tree thatsupports a move operation. We model each replica as a statemachine that transitions from one state to the next by apply-ing an operation. The algorithm is executed independentlyon each replica with no shared memory between replicas.

When the user wants to make a change to the tree, theygenerate an operation and apply it to their local replica.Every operation is also asynchronously sent over the net-work to all other replicas, and applied to every remotereplica using the same algorithm as for local operations.The network may deliver operations in any order; thus, thecommunication can be performed peer-to-peer, and it doesnot require any central server or consensus protocol. Weonly assume that eventually every operation is applied onevery replica, provided that network partitions are of a finiteduration.

The key consistency property of our algorithm is con-vergence: that is, whenever any two replicas have appliedthe same set of operations, then they must be in the samestate—even if the operations were applied in a differentorder on different replicas. We prove this in §4 by showingthat applying operations is commutative.

Figure 4 gives the full source code for our algorithmin the Isabelle/HOL language [9]. We choose this languagebecause it combines the conciseness of pseudocode with the

precision of mathematical notation. It supports formal rea-soning, allowing us to prove the correctness of the algorithm(§4), and it can be exported to Scala, Haskell, or OCaml.

In this section we walk through the code step by step,explaining the Isabelle/HOL syntax as we encounter it.Additional background documentation is available [14].

3.1 Operations and trees

We allow the tree to be updated in three ways: by creatinga new child of any parent node, by deleting a node, or bymoving a node to be a child of a new parent. However,all three types of update can be represented by a moveoperation. To create a node, we generate a fresh ID for thatnode, and issue an operation to move this new ID to bea child of its desired parent; the node is then implicitlycreated. We also designate as “trash” some node ID thatdoes not exist in the tree; then we can delete a node bymoving it to be a child of the trash.

Thus, we define one kind of operation: Move t p m c(Figure 4, lines 1–5). A move operation is a 4-tuple consistingof a timestamp t of type ’t, a parent node ID p of type ’n,a metadata field m of type ’m, and a child node ID c oftype ’n. Here, ’t, ’n and ’m are type variables that can bereplaced with arbitrary types; we only require that nodeidentifiers ’n are globally unique (e.g. UUIDs); timestamps ’tneed to be globally unique and totally ordered (e.g. Lamporttimestamps [11]).

The meaning of an operation Move t p m c is that at time t,the node with ID c is moved to be a child of the parent nodewith ID p. The operation does not specify the old locationof c; the algorithm simply removes c from wherever it iscurrently located in the tree, and moves it to p. If c does notcurrently exist in the tree, it is created as a child of p.

The metadata field m in a move operation allows ad-ditional information to be associated with the parent-childrelationship of p and c. For example, in a filesystem, the par-ent and child are the inodes of a directory and a file withinit, respectively, and the metadata contains the filename ofthe child. Thus, a file with inode c can be renamed byperforming a Move t p m c, where the new parent directoryp is the inode of the existing parent (unchanged), but themetadata m contains the new filename.

When users want to make changes to the tree on theirlocal replica, they generate new Move t p m c operationsfor these changes, and apply these operations using thealgorithm described in the rest of this section.

We can represent the tree as a set of (parent, meta, child)triples, denoted in Isabelle/HOL as (’n × ’m × ’n) set. Whenwe have (p, m, c) ∈ tree, that means c is a child of p in the tree,with associated metadata m. Given a tree, we can constructa new tree’ in which the child c is moved to a new parent p,with associated metadata m, as follows:

tree’ = {(p’, m’, c’) ∈ tree. c’ 6= c} ∪ {(p, m, c)}

That is, we remove any existing parent-child relationshipfor c from the set tree, and then add {(p, m, c)} to representthe new parent-child relationship. This expression appearson lines 30 and 35 of Figure 4, as we shall explain shortly.

5

datatype ( ′t, ′n, ′m) operation1

= Move (move time: ′t)2

(move parent: ′n)3

(move meta: ′m)4

(move child: ′n)5

6

datatype ( ′t, ′n, ′m) log op7

= LogMove (log time: ′t)8

(old parent: ( ′n × ′m) option)9

(new parent: ′n)10

(log meta: ′m)11

(log child: ′n)12

13

type synonym ( ′t, ′n, ′m) state = ( ′t, ′n, ′m) log op list × ( ′n × ′m × ′n) set14

15

definition get parent :: ( ′n × ′m × ′n) set⇒ ′n⇒ ( ′n × ′m) option where16

get parent tree child ≡17

if ∃ !parent. ∃ !meta. (parent, meta, child) ∈ tree then18

Some (THE (parent, meta). (parent, meta, child) ∈ tree)19

else None20

21

inductive ancestor :: ( ′n × ′m × ′n) set⇒ ′n⇒ ′n⇒ bool where22

[[(parent, meta, child) ∈ tree]] =⇒ ancestor tree parent child |23

[[(parent, meta, child) ∈ tree; ancestor tree anc parent]] =⇒ ancestor tree anc child24

25

fun do op :: ( ′t, ′n, ′m) operation × ( ′n × ′m × ′n) set⇒ ( ′t, ′n, ′m) log op × ( ′n × ′m × ′n) set where26

do op (Move t newp m c, tree) =27

(LogMove t (get parent tree c) newp m c,28

if ancestor tree c newp ∨ c = newp then tree29

else {(p ′, m ′, c ′) ∈ tree. c ′ 6= c} ∪ {(newp, m, c)})30

31

fun undo op :: ( ′t, ′n, ′m) log op × ( ′n × ′m × ′n) set⇒ ( ′n × ′m × ′n) set where32

undo op (LogMove t None newp m c, tree) = {(p ′, m ′, c ′) ∈ tree. c ′ 6= c} |33

undo op (LogMove t (Some (oldp, oldm)) newp m c, tree) =34

{(p ′, m ′, c ′) ∈ tree. c ′ 6= c} ∪ {(oldp, oldm, c)}35

36

fun redo op :: ( ′t, ′n, ′m) log op⇒ ( ′t, ′n, ′m) state⇒ ( ′t, ′n, ′m) state where37

redo op (LogMove t p m c) (ops, tree) =38

(let (op2, tree2) = do op (Move t p m c, tree)39

in (op2 # ops, tree2))40

41

fun apply op :: ( ′t::{linorder}, ′n, ′m) operation⇒ ( ′t, ′n, ′m) state⇒ ( ′t, ′n, ′m) state where42

apply op op1 ([], tree1) =43

(let (op2, tree2) = do op (op1, tree1)44

in ([op2], tree2)) |45

apply op op1 (logop # ops, tree1) =46

(if move time op1 < log time logop47

then redo op logop (apply op op1 (ops, undo op (logop, tree1)))48

else let (op2, tree2) = do op (op1, tree1) in (op2 # logop # ops, tree2))49

50

definition apply ops :: ( ′t::{linorder}, ′n, ′m) operation list⇒ ( ′t, ′n, ′m) state where51

apply ops ops ≡ foldl (λstate oper. apply op oper state) ([], {}) ops52

53

definition unique parent :: ( ′n × ′m × ′n) set⇒ bool where54

unique parent tree ≡55

(∀ p1 p2 m1 m2 c. (p1, m1, c) ∈ tree ∧ (p2, m2, c) ∈ tree −→ p1 = p2 ∧ m1 = m2)56

57

definition acyclic :: ( ′n × ′m × ′n) set⇒ bool where58

acyclic tree ≡ (@ n. ancestor tree n n)59

Fig. 4. The move operation algorithm, implemented in the Isabelle/HOL language.

6

3.2 Replica state and operation logIn order to correctly apply move operations, a replica needsto maintain not only the current state of the tree, but alsoan operation log. The log is a list of LogMove records indescending timestamp order. LogMove t oldp p m c (lines 7–12) is similar to Move t p m c; the difference is that LogMovehas an additional field oldp of type (’n × ’m) option. Thisoption type means the field can either take the value None(similar to null), or a pair of a node ID and a metadata field.

When a replica applies a Move operation to its tree, italso records a corresponding LogMove operation in its log.The t, p, m and c fields are taken directly from the Moverecord, while the oldp field is filled in based on the state ofthe tree before the move. If c did not exist in the tree, oldp isset to None. Otherwise, oldp records the previous parent andmetadata of c: if there exist p’ and m’ such that (p’, m’, c) ∈tree, then oldp is set to Some (p’, m’). The get parent functionimplements this (lines 16–20).

In the first line of get parent, the expression between ::and where is the type signature of the function, in this case:

(’n × ’m × ’n) set⇒ ’n⇒ (’n × ’m) optionThis signature denotes a function that takes two arguments:a tree (’n× ’m× ’n) set and a node ID ’n. It then returns a (’n× ’m) option. The operator ∃ !x means “there exists a uniquevalue x such that. . . ”, while THE x means “choose the uniquevalue x such that. . . ”.

In line 14 we define the datatype for the state of a replica:a pair (log, tree) where log is a list of LogMove records, andtree is a set of (parent, meta, child) triples as before.

3.3 Preventing cyclesRecall from §2.2 that in order to prevent a cycle beingintroduced, the node being moved must not be an ancestorof the destination node. To implement this we first definethe ancestor relation in lines 22–24. It is the transitive closureof a tree’s parent-child relation: if (p, m, c) ∈ tree then p is anancestor of c (line 23); moreover, if a is an ancestor of p and(p, m, c) ∈ tree, then a is also an ancestor of c (line 24). Theinductive keyword indicates that this recursive definition isiterated until the least fixed point is reached.

The do op function (lines 26–30) now performs the actualwork of applying a move operation. This function takes asargument a pair consisting of a Move operation and thecurrent tree, and it returns a pair consisting of a LogMoveoperation (which will be added to the log) and an updatedtree. In line 28, the LogMove record is constructed as de-scribed in §3.2, obtaining the prior parent and metadata of cusing the get parent function.

Line 29 performs the check that ensures no cycles areintroduced: if ancestor tree c newp, i.e. if the node c is beingmoved, and c is an ancestor of the new parent newp, then thetree is returned unmodified—in other words, the operationis ignored. Similarly, the operation is also ignored if c = newp.Otherwise (line 30), the tree is updated by removing c fromits existing parent, if any, and adding the new parent-childrelationship (newp, m, c) to the tree.

3.4 Applying operations in any orderThe do op function is sufficient for applying operations ifall replicas apply operations in the same order. However, in

an optimistic replication setting, each replica may apply theoperations in a different order, and we need to ensure thatthe replica state nevertheless converges towards a consistentstate. This goal is accomplished by the undo op, redo op, andapply op functions (lines 32–49).

When a replica needs to apply an operation with time-stamp t, it first undoes the effect of any operations with atimestamp greater than t, then performs the new operation,and finally re-applies the undone operations. As a result, thestate of the tree is as if the operations had all been appliedin order of increasing timestamp, even though in fact theymight have been applied in any order.

The apply op function (lines 42–49) takes two arguments:a Move operation to apply and the current replica state; andit returns the new replica state. The constraint ’t::{linorder}in the type signature indicates that timestamps ’t are in-stances of linorder type class, and they can therefore becompared with the < operator defining a linear (or total)order. This comparison occurs on line 47.

Recall that the replica state includes the operation log(line 14), and we use this log to perform the undo-do-redo cycle. Lines 43–45 handle the case where the log isempty: in this case, we simply perform the operation usingdo op, and return the new tree along with a log containinga single LogMove record. If the log is nonempty (line 46),we take logop to be the first element of the log, and opsto be the rest. (The hash character in logop # ops is thelist cons operator that adds one element to the head of alist.) If the timestamp of logop is greater than the timestampof the new operation (line 47) we first undo logop withundo op, then recursively apply the new operation to theremaining log, and finally reapply logop with redo op (line48). Otherwise we perform the operation using do op, andadd the corresponding LogMove record as the head of thelog (line 49).

This logic ensures that the log is maintained in descend-ing timestamp order, with the greatest timestamp at thehead. undo op (lines 32–35) inverts the effect of a previousmove operation by restoring the prior parent and metadatathat were recorded in the LogMove’s additional field. redo op(lines 37–40) uses do op to perform an operation again andrecomputes the LogMove record (which might have changeddue to the effect of the new operation).

3.5 Handling conflictsDue to the undo-do-redo cycle, the state of the tree is as ifall operations had been applied using do op in increasingtimestamp order, regardless of the order in which theywere actually applied. This provides a clear and consistentapproach to the handling of conflicts:

• If two operations concurrently move the same node,the operation with the lower timestamp moves thenode first, and then the operation with the greatertimestamp moves it again, so the final parent isdetermined by the latter. Since move operations donot specify the old location of a node, but only thenew location, this sequential execution of concurrentoperations is well-defined.

• If two operations would introduce a cycle whencombined, as in §2.2, then the operation with the

7

greater timestamp is ignored. This is the case becausedo op checks for cycles based on the tree created byall operations with a lower timestamp. The lowerof two conflicting operations will take effect, sincethat operation by itself is safe. When the higher-timestamped operation is applied, do op detects thatit would introduce a cycle, and therefore ignores theoperation.

This resolution of a conflict between two operations canbe generalised to any number of conflicting operations byapplying the rules repeatedly pairwise.

Note that the safety of an operation (whether or notit would introduce a cycle) may change as subsequentoperations with lower timestamps are applied. For example,an operation may initially be regarded as safe, and then bereclassified as unsafe after applying a conflicting operationwith a lower timestamp. The opposite is also possible: anoperation previously regarded as unsafe may become safethrough the application of an operation that removes therisk of introducing a cycle. For this reason, the operation logmust include all operations, even those that were ignored.

One final type of conflict that we have not discussedso far is multiple child nodes with the same parent andthe same metadata. For example, in a filesystem, two userscould concurrently create files with the same name in thesame directory. Our algorithm does not prevent such aconflict, but simply retains both child nodes. In practice, thecollision would be resolved by making the filenames dis-tinct, e.g. by appending a replica identifier to the filenames.

3.6 Algorithm extensionsHardlinks and symlinks. Unix filesystems support hard-links, which allow the same file inode to be referencedfrom multiple locations in a tree. Our tree data structurecan easily be extended to support this: rather than placingfile data directly in the leaf nodes of the tree, the leaf nodemust reference the file inode. Thus, references to the samefile inode can appear in multiple leaf nodes of the tree.Symlinks are also easy to support, since they are just leafnodes containing a path (not a reference to an inode).Log truncation. The algorithm as specified in Figure 4retains operations in the log indefinitely, so the memoryuse grows without bound. However, in practice it is easyto truncate the log, because apply op only examines the logprefix of operations whose timestamp is greater than that ofthe operation being applied. Thus, once it is known that allfuture operations will have a timestamp greater than t, thenoperations with timestamp t or less can be discarded fromthe log. In this case, we say that t is causally stable [15].

A similar approach can be used to garbage-collect anytree nodes in the trash (§3.1). Initially, trashed nodes mustbe retained because a concurrent move operation may movethem back out of the trash. However, once the operationthat moved a node to the trash is causally stable, we knowthat no future operations will refer to this node, and so thetrashed node and its descendants can be discarded.

We can determine causal stability in a system wherethe set of replicas is known, where each replica generatesoperations with monotonically increasing timestamps, andwhere the communication link between any pair of replicas

is FIFO (messages are received in the order in which theyare sent, as implemented e.g. by TCP). In this case, we cankeep track of the most recent timestamp we have seen fromeach replica (including our own), and the minimum of thesetimestamps is the causally stable threshold.Ordering of sibling nodes. Another useful extension of thetree algorithm is to allow children of the same parent nodeto have an explicit ordering. For example, in XML, the set ofchildren of an element is ordered. This can be implementedby maintaining an additional list CRDT for each branchnode, e.g. using RGA [16] or Logoot [17]. These algorithmsassign a unique ID to each element of the list, and this IDcan be included in the metadata field of move operations inorder to determine the order of sibling nodes.

This approach to determining ordering also easily sup-ports reordering of child nodes within a parent: to movea node to a different position in a list, we use the listCRDT to generate a new ID at the desired position in thesequence [18]. Then we perform a move operation in whichthe parent node is unchanged, but the metadata is changedto this new ID.File merging. In a distributed filesystem, replication andconflict resolution is required not only for the directorystructure, but also for the contents of individual files. Thiscan be accomplished by using CRDTs for file contents aswell. We discuss distributed filesystems further in §6.2.

4 PROOF OF CORRECTNESS

We now discuss the correctness properties of the algorithmfrom §3. All theorems stated here have been formally provedand mechanically checked using Isabelle. For space reasonsthis paper gives only the statements that were proved, butelides a discussion of the reasoning steps. The Isabelle filescontaining the full details are open source [19], and areincluded as supplementary material with this paper.

To reason about the state of a replica we first define thefunction apply ops on lines 51–52 of Figure 4. It takes a listof operations ops and returns the state of a replica after ithas applied all the operations in ops. The apply ops functionworks by starting in the initial state ([], {}) consisting ofthe empty operation log [] and the tree represented by theempty set {}, and then applying the operations one by oneto the state using the apply op function (introduced in §3.4).The foldl function from the Isabelle/HOL standard libraryperforms the iteration over the list of operations.

4.1 Tree invariantsA tree is an acyclic graph in which every node has exactlyone parent, except for the root, which has no parent. In fact,we slightly generalise this property and allow more thanone root to exist, so the graph represents a forest, allowingan application to move nodes between different trees, ifdesired. For example, the “trash” node used for deletion(§3.1) can be separate from the main tree. To prove thatour algorithm maintains a forest structure, no matter whichoperations are applied, we demonstrate several invariants.Each node’s parent is unique. The first invariant we proveis that each tree node has either no parent (if it is the rootof a tree) or exactly one parent (if it is a non-root node).

8

We state this theorem in Isabelle/HOL as follows, whereapply ops unique parent is the name of this theorem:

theorem apply ops unique parent:assumes apply ops ops = (log, tree)shows unique parent tree

That is, we consider any list of operations ops and define(log, tree) to be the replica state after ops have been applied.We then prove that “unique parent tree” holds, where theunique parent predicate is defined on lines 54–56 of Figure 4:whenever the tree contains a triple whose third element isthe child node c, then the first and second elements of thetriple (the parent node and the metadata) are uniquely de-fined. As we make no assumptions about ops, this theoremholds for any replica state that can be reached by applyingany number of operations.

The graph contains no cycles. This second invariant isexpressed as follows in Isabelle/HOL:

theorem apply ops acyclic:assumes apply ops ops = (log, tree)shows acyclic tree

The acyclic predicate is defined on lines 58–59 of Figure 4,using the ancestor relation (the transitive closure of thegraph’s edges): a graph contains no cycles if no node is anancestor of itself.

Other correctness properties There are further criteria wemight use to determine if our algorithm is correct. Forexample, we might demonstrate that a single-replica systemoperates with the usual sequential semantics of a tree. In asystem with multiple disjoint trees, we could prove that thetrees don’t get tangled together. We conjecture that thoseproperties hold for our algorithm, but leave the proof out ofscope for this paper.

4.2 Convergence

As discussed in §3.4, we require that when replicas applythe same set of operations, they converge towards the samestate, regardless of the order in which the operations areapplied. We formalise this in Isabelle/HOL as follows:

theorem apply ops commutes:assumes set ops1 = set ops2

and distinct (map move time ops1)and distinct (map move time ops2)

shows apply ops ops1 = apply ops ops2

The predicate distinct takes a list as argument, and re-turns true if all elements of the list are distinct (i.e. no valueoccurs more than once in the list). Therefore, the assumptiondistinct (map move time ops1) states that in the list ops1, thereare no two operations with the same timestamp.

The function set takes a list and turns it into an un-ordered set with the same elements. Thus, the assumptionset ops1 = set ops2 means that the lists ops1 and ops2 containthe same elements, but perhaps in a different order—inother words, ops1 is a permutation of ops2. Under theseassumptions, apply ops commutes proves that applying thelist of operations ops1 results in the same replica state asapplying the list of operations ops2.

Strong eventual consistency. Gomes et al. [6] define aframework in Isabelle/HOL for proving the strong eventualconsistency properties of CRDTs. Using our convergenceproof above we integrate our tree datatype with this frame-work, and thus demonstrate that our move operation ontrees does indeed guarantee strong eventual consistency.The details appear in our Isabelle theory files [19].

4.3 Making the HOL definitions executable

Isabelle/HOL can generate executable Haskell, OCaml,Scala, and Standard ML code from HOL definitions usinga sophisticated code generation mechanism [20]. However,not all definitions can be realised in executable form: forexample, the use of choice principles (like in get parent) andinductively defined relations (e.g. ancestor) cause problems.Moreover, HOL’s set type allows infinite sets. Whilst theseconstructs are convenient for theorem proving, they do nottranslate well to executable code.

We therefore produce variants of the definitions in Fig-ure 4 that are designed for execution rather than theoremproving. Rather than representing the tree as a set of (parent,meta, child) triples, these definitions use a hash-map inwhich the keys are child nodes, and the values are (meta,parent) pairs, written in Isabelle/HOL as (’n::{hashable}, ’m× ’n) hm. The hashable type class means that keys must havea hashing function. In effect, this hash-map is an index overthe set of triples, using the fact that the parent and metadatafor a given child are unique (§4.1). The hash-map implemen-tation is from the Isabelle Collections Framework [21].

A hash-map t of type (’n, ’m × ’n) hm simulates a set T oftype (’n × ’m × ’n) set when their entries are the same:

definition simulates where simulates t T ≡(∀ p m c. hm.lookup c t = Some (m, p)←→ (p, m, c) ∈ T)

where hm.lookup c t looks up the key c in the hash-map t,returning Some x if c maps to the value x, and returningNone if c does not appear in the hash-map. We can nowprove the equivalence of the set-based and the hash-map-based implementations:

lemma executable apply ops simulates:assumes executable apply ops ops = (log1, t)

and apply ops ops = (log2, T)shows log1 = log2 ∧ simulates t T

That is, if executable apply ops and apply ops are appliedto the same list of operations, they produce identical logs,and also produce trees that contain the same set of key-value bindings—i.e. the trees are extensionally equivalent,despite having very different in-memory representations.We can also prove corollaries of apply ops commutes andapply ops acyclic for the hash-map-based implementation.

5 EVALUATION

5.1 Performance of move operation

We used Isabelle/HOL’s code generation mechanism to ex-tract executable Scala code from our HOL definitions, as dis-cussed in §4.3, and wrapped this implementation in a simplenetwork service. We deployed three replicas of this serviceon Amazon EC2 c5.large instances in different regions

9

Fig. 5. Median execution time for applying an operation to the replicastate, on an Amazon EC2 c5.large instance, at varying operationthroughput rates. Error bars indicate the minimum and 95th percentileexecution times.

worldwide: in Northern California (region us-west-1),Ireland (eu-west-1), and Singapore (ap-southeast-1).The network delays between these regions are substantial: inour experiments, a request-response round-trip from Irelandto California takes a median of 145 ms, while a request fromSingapore to California takes 176 ms.

We use a synthetic workload in which each replica startswith an empty tree, and then generates move operations ata fixed rate; for each move operation the generating replicachooses parent and child nodes uniformly at random froma set of 1,000 tree nodes. A tree node is created by the firstoperation that refers to it. We use Lamport timestamps [11]as operation timestamps, and 64-bit integers as tree nodeidentifiers. When a replica generates an operation, it imme-diately applies that operation to its local state, and asyn-chronously sends the operation to the other two replicasvia TCP connections. When a replica receives an operationfrom a remote replica, it also applies that operation to itsown replica state. Thus, of the operations applied by eachreplica, approximately one third are locally generated, andtwo thirds are received from the other two replicas.

For a given operation rate we ran the system for 10minutes to reach a steady state. We repeated the experimentwith different operation rates, and measured the executiontime of our algorithm for applying each operation. Figure 5shows the median execution times for local and remote op-erations respectively. The execution time for local operationsis near-constant (median times between 50 µs and 70 µs),while the execution time for remote operations increasesapproximately linearly from 0.14 ms to 1.5 ms over the rangeof 30 to 600 operations/sec. At around 600 operations/seceach single-threaded replica is fully utilising one CPU core,and increasing the rate at which operations are generateddoes not increase throughput any further.

Local operations take constant time is because a locallygenerated operation always has a timestamp greater thanany existing operation at the generating replica at that

Fig. 6. Median time to apply a local CRDT operation compared tothe median time to perform a move operation using state machinereplication, with a leader located in another region. Note the log scaleon both axes.

time (by definition of Lamport timestamps). Applying thatoperation requires only a do op, and no undo op or redo op.

On the other hand, for remote operations, a number ofundos and redos must be performed, depending on the de-gree of concurrency. As the time interval between successiveoperations decreases relative to the network delay, there aremore operations “in flight” at the same time, and more callsto undo op and redo op are required to order operations bytimestamp. The time taken to apply a remote operation isproportional to the number of undos and redos required,and hence proportional to the operation rate.

Note that when a user is interacting with the system,they only have to wait for local operations to execute; anyremote operations can be applied in the background withoutaffecting user interaction. Since our algorithm need notperform any undos or redos for local operations, these userinteractions are consistently fast.

5.2 Comparison to state machine replicationA simple alternative to our CRDT algorithm is to use statemachine replication [22]: that is, we use a leader replica orconsensus algorithm to impose a total order on all opera-tions, and then execute operations in that same order on allreplicas. For a move operation on trees, the state machinereplication algorithm is much simpler than the CRDT: it stillneeds to check for cycles, but it never needs to undo or redoany operations because they are never applied out-of-order.

To compare the performance of our algorithm to the statemachine approach we ran another set of experiments on thesame three replicas in California, Ireland, and Singapore.In this experiment, the Californian replica was designatedleader; it totally ordered all operations it received, and sentthem to all other replicas in the same order. The Irish andSingaporean replicas generated operations, sent them to theleader, and applied them to their local tree in the orderthey were received from the leader. In order to ensure afair comparison to the CRDT algorithm, these experimentsused the same Isabelle-generated Scala code.

The results from this experiment are shown in Figure 6.The leader-based approach was able to sustain a throughput

10

of 14,000 move operations per second, approximately 23times the CRDT’s throughput of 600 ops/sec, due to thefact that it does not need to spend CPU cycles undoing andredoing operations. On the other hand, the latency of eachoperation is around 3,000 times greater in the state machineapproach, due to the network delay to the leader.

Therefore, we have a clear trade-off: in applications thatneed to prioritise throughput, a state machine replicationapproach is preferable, while in applications that need tominimise response times to user requests, our CRDT algo-rithm is preferable. Our algorithm also has the advantagethat local user operations require no waiting for networkcommunication, so users can still modify the tree while dis-connected from the Internet. In the leader-based approach,clients cannot make updates while offline.

We note that the throughput of 600 ops/sec is for a singletree, e.g. a filesystem belonging to one user. Operationson independent trees, belonging to different users, can beprocessed in parallel. We believe that for many applications,600 ops/sec/user is plenty, and the ability to work offline isa valuable benefit.

5.3 Evaluation of formal proofThe formalisation of our algorithm, and the proofs of itsproperties as described in §4, have been formally checkedby Isabelle/HOL. Our proofs contain no unproven assump-tions (i.e. no occurrences of the sorry keyword). Checkingall of the proofs takes 3.5 minutes on a 2018 MacBook Pro.

Besides the 59 lines of definitions given in Figure 4, ourIsabelle/HOL formalisation consists of a further 2,166 linesof code. Of this, we use 203 lines to prove that every nodehas a unique parent, 443 lines to prove that the tree containsno cycles, 450 lines to prove that move operations commuteand replicas converge, 327 lines to prove the strong eventualconsistency property, and 743 lines to define the executablevariant of our algorithm and prove its equivalence to thedefinitions of Figure 4.

6 RELATED WORK

Many replicated data systems use optimistic replication [4],which allows the state of replicas to temporarily diverge, inorder to achieve better performance and availability in thepresence of faults than strongly consistent systems [1], [23].As a consequence, these systems require a mechanism formerging or reconciling conflicting updates that were madeconcurrently on different replicas. For example, versioncontrol systems such as Git [24] leave conflicts for the userto resolve manually. Databases such as Dynamo [25] andBayou [26] rely on the application programmer to provideexplicit conflict resolution logic; however, such logic is dif-ficult to get right [6], [27], [28]. Hence, we want to automat-ically ensure that all replicas converge towards a consistentstate, without requiring custom application logic—a consis-tency model known as strong eventual consistency [5], [6].

6.1 Conflict-free Replicated Data TypesOur algorithm is an example of an operation-based Conflict-free Replicated Data Type or CRDT [5], [29]. All CRDTsshare the property that concurrent changes on different

replicas can be merged in any order; any two replicas thathave seen the same set of updates are guaranteed to be in thesame state, regardless of the order in which they processedthese updates. Several CRDTs for trees have been proposed:

• Martin et al. [30] define a CRDT for XML data, andKleppmann and Beresford [31] define a CRDT forJSON. However, these algorithms only deal withinsertion and deletion of tree nodes, and do notsupport moves. A move operation can be emulatedby deleting and re-inserting the moved node, butthis approach suffers from the duplication problemdemonstrated in §2.1.

• As discussed in §2.3, Najafzadeh et al. [7], [8] proposetwo implementations for a replicated filesystem: aCRDT in which conflicting moves are handled byduplicating tree nodes (as in Figures 1b and 2b),and a centralised implementation in which moveoperations must obtain a lock before executing (not aCRDT since it relies on synchronous coordination).

• Ahmed-Nacer et al. [32] outline approaches to han-dling conflicts on trees, but provide no algorithms.

• Tao et al. [33] propose handling conflicting moveoperations by allowing the same object to appear inmore than one location; thus, their datatype is strictlya DAG, not a tree. Some conflicts are handled byduplicating tree nodes. Tao et al. also perform exper-iments with Dropbox, Google Drive, and OneDrive,similar to our experiments discussed in §2.

Besides CRDTs, another family of algorithms for con-current modification of data structures is Operational Trans-formation (OT) [34]. Several authors have defined concurrenttree structures using OT [35], [36], [37], but they only handleinsertion and deletion of nodes, and do not support moves.

Molli et al. [38] define an OT tree structure with a moveoperation. However, it requires that all communication be-tween replicas is performed via total order broadcast, whichrequires a leader replica or consensus algorithm, like in §5.2.Our algorithm has better availability characteristics in thepresence of network partitions because it allows messagesto be delivered in any order, e.g. via peer-to-peer protocols.

Collaborative graphics software Figma uses an approachinspired by CRDTs, but prevents cycles in their object treeby relying on a central server; its replication protocol allowsobjects to temporarily disappear while syncing [39].

6.2 Distributed filesystems

Many distributed filesystems, such as NFS, rely on syn-chronous interaction with a server. This avoids the need forconflict resolution, but rules out users working offline.

Coda is a client-server filesystem that allows clients tolocally cache copies of files stored in a server-side datarepository [40]. Clients can edit data in the cache whileoffline, during which time a kernel module keeps track of allupdates. When the client comes back online it attempts toresynchronise changes with the server. To resolve conflictsdue to concurrent updates, Coda uses application-specificresolvers [41], similarly to Bayou’s approach [26]. Concur-rent renaming and move operations have been considered,but the authors note that they do “not address transparent

11

resolution of cross-directory renames [i.e. move operations]in [their] current implementation” [42]. Furthermore, whilethe authors consider a number of conflicts associated withdirectory move operations, they do not highlight the poten-tial for the creation of cycles.

Ficus [43] is an in-kernel SunOS-based replicated peer-to-peer filesystem. Ficus supports updates to replicas duringperiods of network partition and claims “conflicting updatesto directories are detected and automatically repaired” [44].Unfortunately we were unable to find a precise definition ofthe algorithm used in any of the available publications.

Rumor [45], [46] is the successor to Ficus. While pre-vious work uses the kernel filesystem interface, Rumor is auserspace process that is invoked periodically by the user orby a daemon; when run, it compares the state of the replicas.The original version of Rumor was unable to scale beyond20 replicas, but an extension called Roam [47] allowed betterscaling. In an attempt to test Rumor’s conflict handling weobtained the source code from archive.org [48]; however, wewere unable to get it running after modest effort.

Unison is a file synchronisation tool with a formal spec-ification that allows two replicas to synchronise the state ofa directory [49]. It permits offline updates to both replicas.Like Rumor, Unison is a userspace process that comparesreplica states. Whenever it is run, Unison records a summaryof the filesystem state on each replica, and it uses thissummary to determine the changes made since the lastsynchronisation. When presented with the move operationsdescribed in Figure 1, Unison duplicates the files, resultingin the outcome shown in Figure 1a. Unison is unable toautomatically synchronise the move operations shown inFigure 2 and instead asks the user to choose one of fourpossible resolutions: those shown in Figure 2b, 2c, 2d, or todelete both directories.

Hughes et al. [50] test Dropbox and Google Drive againsta formal specification, but they do not consider moving files,and thus they do not find the issue described in §2.2.

Bjørner [51] discusses the development of the Dis-tributed File System Replication (DFS-R) component of Win-dows Server, during which a model checker found an issuewith concurrent moves similar to Figure 2a. Bjørner outlinesseveral possible solutions, but notes that model-checkingtheir algorithm was not feasible due to state space explosion.Our use of proof by induction, rather than model-checking,allows us to verify the correctness of our algorithm inunbounded executions.

After we performed the experiments described in Sec-tion 2, the Dropbox engineering team published a blogpost [52] acknowledging the problems of cycles and dupli-cation due to concurrent moves.

6.3 Totally ordered operation log

Our approach of ordering operations by a timestamp, andundoing/redoing them as necessary so that they take effectin ascending timestamp order, is conceptually very simple.Similar ideas appear in many other systems, including theBayou database [26], Jefferson’s Time Warp mechanism [53],Shapiro et al.’s capricious total order [54], and Burckhardt’sstandard conflict resolution [28, §4.3.3]. The concept of undo-redo is also well known from write-ahead logging [55].

However, to our knowledge, this approach has not pre-viously been applied to the problem of replicated trees, norare we aware of previous mechanised proofs (using Isabelleor other tools) that formalise this approach. The simplicityof the approach facilitates reasoning about it.

7 CONCLUSIONS

We have defined a novel algorithm that handles arbitraryconcurrent modifications of a tree data structure—adding,moving, and removing nodes—in a peer-to-peer replicationsetting. It is applicable to distributed filesystems, databases,and applications that use a tree-structured data model. Ourapproach ensures that all replicas converge to the same con-sistent state without needing any manual conflict resolution,and without requiring application developers to implementconflict handling logic. Updates made to a local replica takeeffect immediately, while operations from remote replicascan be propagated and applied in the background. Thisapproach means that user interaction is consistently fast,even in the face of unbounded communication delays orduring disconnected operation of mobile devices.

The principle behind our algorithm is easy to under-stand: undoing and redoing operations so that they areeffectively executed in timestamp order. Nevertheless, itsolves a real problem that has not been solved correctlyin widely deployed software such as Google Drive andDropbox, as we demonstrated in §2. To rule out such bugs,we formally verified the correctness of our algorithm usingthe Isabelle/HOL proof assistant; our theorems show thatreplicas can apply operations in any order, and that theresult is always a valid tree (nodes have at most one parent,and the graph does not contain any cycles). Moreover, theseresults apply to unbounded executions and an arbitrarynumber of replicas—an advantage of using a proof assistantrather than other formal approaches such as model check-ing.

One might wonder whether our algorithm’s consistencymodel is strong enough for practical use. On this point, wenote that this model is what Google Drive and Dropbox usetoday [50] (apart from the aforementioned bugs).

We also extracted a formally verified Scala implementa-tion from our Isabelle/HOL definitions, and evaluated itsperformance across three replicas in California, Ireland andSingapore. Our implementation applies local updates in ap-proximately 50 µs, three orders of magnitude faster than ispossible with a leader replica or consensus protocol operat-ing over these distances, and it remains available in the faceof network interruptions. However, compared to leader-based replication our algorithm has lower throughput of 600updates per second. A hand-optimised implementation ofour algorithm could achieve higher performance, since codegenerated from HOL definitions contains inefficiencies. Thegenerated code could serve as a reference implementationagainst which other implementations could be tested.

As highlighted in the introduction, our work is espe-cially interesting due to the ubiquity of tree data modelsacross many different types of applications and databases.In future work we hope to build novel applications, such asa distributed filesystem, using our algorithm.

12

We are also exploring whether the undo-do-redo ap-proach can be used in other concurrent data structures;for example, there is an open problem in collaborative textediting [18] that might be solved by this approach. We ex-pect that our Isabelle/HOL formalisation will be a valuableresource for future work in this area, which can use ourdefinitions and proofs as starting point. The formalisation isincluded as supplementary data of this paper.

ACKNOWLEDGEMENTS

The authors wish to acknowledge the support of The BoeingCompany and the EPSRC “REMS: Rigorous Engineeringfor Mainstream Systems” programme grant (EP/K008528).Martin Kleppmann is supported by a Leverhulme TrustEarly Career Fellowship and by the Isaac Newton Trust. Ourevaluation was conducted using AWS credits from the AWSEducate program. Thank you to Marc Shapiro for feedbackon a draft of this paper.

REFERENCES

[1] S. Gilbert and N. A. Lynch, “Brewer’s conjecture and the feasibilityof consistent, available, partition-tolerant web services,” ACMSIGACT News, vol. 33, no. 2, pp. 51–59, 2002.

[2] A. Alquraan, H. Takruri, M. Alfatafta, and S. Al-Kiswany, “Ananalysis of network-partitioning failures in cloud systems,” in 13thUSENIX Symposium on Operating Systems Design and Implementa-tion (OSDI), Oct. 2018, pp. 51–68.

[3] P. Bailis and K. Kingsbury, “The network is reliable,” ACM Queue,vol. 12, no. 7, Jul. 2014.

[4] Y. Saito and M. Shapiro, “Optimistic replication,” ACM ComputingSurveys, vol. 37, no. 1, pp. 42–81, Mar. 2005.

[5] M. Shapiro, N. Preguica, C. Baquero, and M. Zawirski, “Conflict-free replicated data types,” in 13th International Symposium onStabilization, Safety, and Security of Distributed Systems (SSS), Oct.2011, pp. 386–400.

[6] V. B. F. Gomes, M. Kleppmann, D. P. Mulligan, and A. R. Beresford,“Verifying strong eventual consistency in distributed systems,”Proceedings of the ACM on Programming Languages (PACMPL),vol. 1, no. OOPSLA, Oct. 2017.

[7] M. Najafzadeh, “The analysis and co-design of weakly-consistentapplications,” Ph.D. dissertation, Universite Pierre et Marie Curie,Aug. 2016. [Online]. Available: https://tel.archives-ouvertes.fr/tel-01351187v1

[8] M. Najafzadeh, M. Shapiro, and P. Eugster, “Co-design and verifi-cation of an available file system,” in 19th International Conferenceon Verification, Model Checking, and Abstract Interpretation (VMCAI),Jan. 2018, pp. 358–381.

[9] M. Wenzel, L. C. Paulson, and T. Nipkow, “The Isabelle frame-work,” in 21st International Conference on Theorem Proving in HigherOrder Logics (TPHOLs), Aug. 2008, pp. 33–38.

[10] P. R. Johnson and R. H. Thomas. (1975, Jan.) RFC 677:The maintenance of duplicate databases. [Online]. Available:https://tools.ietf.org/html/rfc677

[11] L. Lamport, “Time, clocks, and the ordering of events in a dis-tributed system,” Communications of the ACM, vol. 21, no. 7, pp.558–565, Jul. 1978.

[12] A. Gotsman, H. Yang, C. Ferreira, M. Najafzadeh, and M. Shapiro,“‘Cause I’m strong enough: reasoning about consistency choicesin distributed systems,” in 43rd ACM Symposium on Principles ofProgramming Languages (POPL), Jan. 2016, pp. 371–384.

[13] M. Najafzadeh, A. Gotsman, H. Yang, C. Ferreira, and M. Shapiro,“The CISE tool: Proving weakly-consistent applications correct,”in 2nd Workshop on the Principles and Practice of Consistency forDistributed Data (PaPoC), Apr. 2016.

[14] T. Nipkow and G. Klein, Concrete Semantics - With Isabelle/HOL.Springer, 2014. [Online]. Available: http://concrete-semantics.org/

[15] C. Baquero, P. S. Almeida, and A. Shoker, “Making operation-based CRDTs operation-based,” in 14th IFIP International Confer-ence on Distributed Applications and Interoperable Systems (DAIS),Jun. 2014, pp. 126–140.

[16] H.-G. Roh, M. Jeon, J.-S. Kim, and J. Lee, “Replicated abstract datatypes: Building blocks for collaborative applications,” Journal ofParallel and Distributed Computing, vol. 71, no. 3, pp. 354–368, 2011.

[17] S. Weiss, P. Urso, and P. Molli, “Logoot-Undo: Distributed col-laborative editing system on P2P networks,” IEEE Transactions onParallel and Distributed Systems, vol. 21, no. 8, pp. 1162–1174, Jan.2010.

[18] M. Kleppmann, “Moving elements in list CRDTs,” in 7th Workshopon Principles and Practice of Consistency for Distributed Data (PaPoC),Apr. 2020.

[19] M. Kleppmann, D. P. Mulligan, V. B. F. Gomes, and A. R.Beresford, “Source code accompanying ‘A highly-available moveoperation for replicated trees and distributed filesystems’,” 2020.[Online]. Available: https://github.com/trvedata/move-op

[20] F. Haftmann and T. Nipkow, “Code generation via higher-orderrewrite systems,” in 10th International Symposium on Functional andLogic Programming (FLOPS), 2010, pp. 103–117.

[21] P. Lammich and A. Lochbihler, “The Isabelle Collections Frame-work,” in 1st International Conference on Interactive Theorem Proving(ITP), 2010, pp. 339–354.

[22] F. B. Schneider, “Implementing fault-tolerant services using thestate machine approach: A tutorial,” ACM Computing Surveys,vol. 22, no. 4, pp. 299–319, Dec. 1990.

[23] P. Bailis, A. Fekete, M. J. Franklin, A. Ghodsi, J. M. Hellerstein,and I. Stoica, “Coordination avoidance in database systems,”Proceedings of the VLDB Endowment, vol. 8, no. 3, pp. 185–196, Nov.2014.

[24] S. Chacon and B. Straub, Pro Git, 2nd ed. Berkeley, CA: Apress,2014. [Online]. Available: https://git-scm.com/book/en/v2

[25] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Laksh-man, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels,“Dynamo: Amazon’s highly available key-value store,” in 21stACM Symposium on Operating Systems Principles (SOSP), Oct. 2007,pp. 205–220.

[26] D. B. Terry, M. M. Theimer, K. Petersen, A. J. Demers, M. J.Spreitzer, and C. H. Hauser, “Managing update conflicts in Bayou,a weakly connected replicated storage system,” in 15th ACMSymposium on Operating Systems Principles (SOSP), Dec. 1995, pp.172–182.

[27] P. Bailis and A. Ghodsi, “Eventual consistency today: Limitations,extensions, and beyond,” ACM Queue, vol. 11, no. 3, Mar. 2013.

[28] S. Burckhardt, “Principles of eventual consistency,” Foundationsand Trends in Programming Languages, vol. 1, no. 1-2, pp. 1–150,Oct. 2014.

[29] S. Burckhardt, A. Gotsman, H. Yang, and M. Zawirski, “Repli-cated data types: Specification, verification, optimality,” in 41stACM SIGPLAN-SIGACT Symposium on Principles of ProgrammingLanguages (POPL), Jan. 2014, pp. 271–284.

[30] S. Martin, P. Urso, and S. Weiss, “Scalable XML collaborativeediting with undo,” in On the Move to Meaningful Internet Systems,Oct. 2010, pp. 507–514.

[31] M. Kleppmann and A. R. Beresford, “A conflict-free replicatedJSON datatype,” IEEE Transactions on Parallel and Distributed Sys-tems, vol. 28, no. 10, pp. 2733–2746, Apr. 2017.

[32] M. Ahmed-Nacer, S. Martin, and P. Urso, “File system onCRDT,” INRIA, Tech. Rep. RR-8027, Jul. 2012. [Online]. Available:https://hal.inria.fr/hal-00720681/

[33] V. Tao, M. Shapiro, and V. Rancurel, “Merging semantics forconflict updates in geo-distributed file systems,” in 8th ACMInternational Systems and Storage Conference (SYSTOR), May 2015.

[34] C. Sun and C. Ellis, “Operational transformation in real-timegroup editors: Issues, algorithms, and achievements,” in ACMConference on Computer Supported Cooperative Work (CSCW), Nov.1998, pp. 59–68.

[35] T. Jungnickel and T. Herb, “Simultaneous editing of JSON objectsvia operational transformation,” in 31st Annual ACM Symposiumon Applied Computing (SAC), Apr. 2016, pp. 812–815.

[36] C.-L. Ignat and M. C. Norrie, “Customizable collaborative editorrelying on treeOPT algorithm,” in 8th European Conference onComputer-Supported Cooperative Work (ECSCW), Sep. 2003, pp. 315–334.

[37] A. H. Davis, C. Sun, and J. Lu, “Generalizing operational trans-formation to the Standard General Markup Language,” in ACMConference on Computer Supported Cooperative Work (CSCW), Nov.2002, pp. 58–67.

[38] P. Molli, G. Oster, H. Skaf-Molli, and A. Imine, “Using the transfor-mational approach to build a safe and generic data synchronizer,”

13

in 2003 International ACM SIGGROUP Conference on SupportingGroup Work, Nov. 2003, pp. 212–220.

[39] E. Wallace. (2019, Oct.) How Figma’s multi-player technology works. Archived at https://perma.cc/79TM-6FEE. [Online]. Available: https://www.figma.com/blog/how-figmas-multiplayer-technology-works/

[40] J. J. Kistler and M. Satyanarayanan, “Disconnected operation in thecoda file system,” ACM Transactions on Computer Systems (TOCS),vol. 10, no. 1, pp. 3–25, 1992.

[41] P. Kumar and M. Satyanarayanan, “Flexible and safe resolution offile conflicts,” in USENIX Winter Technical Conference, Jan. 1995.

[42] ——, “Log-based directory resolution in the coda file system,” in2nd International Conference on Parallel and Distributed InformationSystems (PDIS), 1993, pp. 202–213.

[43] P. Reiher, J. Heidemann, D. Ratner, G. Skinner, and G. J. Popek,“Resolving file conflicts in the Ficus file system,” in SummerUSENIX Conference, Jun. 1994, pp. 183–195.

[44] R. G. Guy, J. S. Heidemann, W.-K. Mak, T. W. Page Jr, G. J. Popek,D. Rothmeier et al., “Implementation of the Ficus replicated filesystem,” in Summer USENIX Conference, 1990, pp. 63–72.

[45] R. Guy, P. Reiher, D. Ratner, M. Gunter, W. Ma, and G. Popek,“Rumor: Mobile data access through optimistic peer-to-peer repli-cation,” in Advances in Database Technologies, LNCS 1552. SpringerBerlin Heidelberg, Nov. 1999, pp. 254–265.

[46] P. Reiher, “Rumor 1.0 user’s manual,” 1998. [Online].Available: https://web.archive.org/web/20170705140958/ftp://ftp.cs.ucla.edu/pub/rumor/rumor users manual.ps

[47] D. Ratner, P. Reiher, and G. J. Popek, “Roam: a scalable replicationsystem for mobile computing,” in 10th International Workshop onDatabase and Expert Systems Applications (DEXA). IEEE, Sep. 1999,pp. 96–104.

[48] Rumor development team at UCLA, “Ru-mor 1.0.2 source code,” 1998. [Online]. Avail-able: https://web.archive.org/web/20170705140950/ftp://ftp.cs.ucla.edu/pub/rumor/rumor.src.release-1.0.2.tar.gz

[49] B. C. Pierce and J. Vouillon, “What’s in Unison? A formalspecification and reference implementation of a file synchronizer,”Dept. of Computer and Information Science, University ofPennsylvania, Tech. Rep. MS-CIS-03-36, 2004. [Online]. Available:http://www.cis.upenn.edu/∼bcpierce/papers/unisonspec.pdf

[50] J. Hughes, B. C. Pierce, T. Arts, and U. Norell, “Mysteries ofDropbox: Property-based testing of a distributed synchronizationservice,” in IEEE International Conference on Software Testing, Verifi-cation and Validation (ICST), Apr. 2016, pp. 135–145.

[51] N. Bjørner, “Models and software model checking of a distributedfile replication system,” in Formal Methods and Hybrid Real-TimeSystems, LNCS 4700. Springer Berlin Heidelberg, 2007, pp. 1–23.

[52] S. Jayakar, “Rewriting the heart of our sync engine,”Mar. 2020, archived at https://perma.cc/HU2D-H9X4.[Online]. Available: https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine

[53] D. R. Jefferson, “Virtual time,” ACM Transactions on ProgrammingLanguages and Systems (TOPLAS), vol. 7, no. 3, pp. 404–425, Jul.1985.

[54] M. Shapiro, M. S. Ardekani, and G. Petri, “Consistency in 3D,”in 27th International Conference on Concurrency Theory (CONCUR),Aug. 2016, pp. 3:1–3:14.

[55] C. Mohan, D. Haderle, B. Lindsay, H. Pirahesh, and P. Schwarz,“ARIES: A transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead log-ging,” ACM Transactions on Database Systems (TODS), vol. 17, no. 1,pp. 94–162, Mar. 1992.

Martin Kleppmann is a Senior Research As-sociate at the University of Cambridge. His re-search spans distributed systems, databases,security, and formal verification, with a partic-ular focus on decentralised collaboration soft-ware and CRDTs. His book Designing Data-Intensive Applications was published in 2017and has been translated into six languages. Pre-viously, he worked as a software engineer andentrepreneur at several internet companies, in-cluding Rapportive and LinkedIn.

Dominic P. Mulligan is a Staff Research En-gineer in the Security Group at Arm Research,UK. Prior to moving to Arm, he worked as apostdoctoral researcher at the Universities ofCambridge and Bologna, Italy, investigating theformal specification and verification of systemssoftware such as C compilers and linkers. Hisinterests include formal verification, distributedsystems, and privacy and security.

Victor B. F. Gomes received a diplomed’ingenieur from INSA Lyon and a PhD degreefrom the University of Sheffield. He was a re-search associate at the University of Cambridgefrom 2016 to 2019 and he is currently working atGoogle. His main research interests are seman-tics of programming languages, formal verifica-tion via theorem prover assistants and algebraicapproaches for program verification.

Alastair R. Beresford is Professor of ComputerSecurity in the Department of Computer Scienceat the University of Cambridge. His researchwork explores the security and privacy of large-scale distributed systems, with a particular focuson networked mobile devices such as smart-phones, tablets and laptops. He looks at thesecurity and privacy of the devices themselves,as well as the security and privacy problems in-duced by the interaction between mobile devicesand cloud-based Internet services.