WHAT’S IN UNISON?A FORMAL SPECIFICATION AND REFERENCE IMPLEMENTATION

OF A FILE SYNCHRONIZER

Presentation type: paper presentationClass: CS 589 – Domain Specific Languages

Presenter: Sergii ShmarkatiukDate: 5/20/2014

2

TARGET DOMAIN: FILE REPLICATION AND SYNCHRONIZATION

File synchronizatio

n

Replication

MirroringDeployment

Cloud storage

3

COMMON USE CASE: CLOUD STORAGE AND SYNCHRONIZATION BETWEEN DEVICES

4

COMMON USE CASE: DEPLOYMENT OF WEB-APPLICATION

development instance production instance

http://localhost/MyApp http://myapp.com

3. upload

2. test

ftp://@myapp.com/var/www/html/

1. edit files

4. test

/var/www/html/MyApp

5

COMMON USE CASE: DEPLOYMENT OF WEB-APPLICATION

development instance production instance

http://localhost/MyApp http://myapp.com

4. test

/var/www/html/MyApp ftp://@myapp.com/var/www/html/

1. edit files

5. test

2. edit files

3. sync

+

6

FILE SYNCHRONIZATION SOFTWARE

Rsync

Beyond Compare

Synchronize It!

Dropbox

Google Drive

Box.net

OneDrive

Manual synchronization

Cloud storage

7

SOFTWARE DEPLOYMENT TOOLS

Puppet

Chef

Capistrano

CFEngine

Ansible

8

UNISON

File synchronization toolCommand line interfaceImplemented in OCaml (DSL) and C

(tool)Available for all major platforms

(UNIX, Win, Mac)Unlike rsync, Unison is not included

into basic UNIX distributions

9

PAPER CONTRIBUTIONS

Paper presents mathematical model, DSL, mechanics and challenges of file synchronization

Authors proved some properties of file synchronization operations using Coq

Authors described the gap between idealistic representation and actual tool implementation

10

QUESTIONSKeying Xu, Chao Peng:What is the semantic domain of Unison? Is it a deep or shallow embedded DSL?

Unison is a deep embedded DSL

11

UNISON: SYNTAX AND SEMANTIC DOMAIN

Current states of the replicas 2 file trees Archive (last

synchronized state)

Synchronized replicas 2 file trees Archive (last

synchronized state)

Syntax Semantic domain

sem

A B

~

A B

=

12

QUESTIONSChao Peng:What is the basic object of Unison?

13

UNISON: BASIC DATA STRUCTURESFILESYSTEM

type name = stringtype contents = stringtype properties = string

type fs = Dir of properties * dContents| File of properties * contents| Symlink of contents| Botand dContents = (name * fs) list

OCaml

14

UNISON: BASIC DATA STRUCTURESUPDATE DETECTION

type prevState = DIR| FILE| SYMLINK| ABSENT type ’a leafUpdate = LeafSame| LeafUpdated of ’a * ’a option

OCaml

15

UNISON: BASIC DATA STRUCTURESUPDATE DETECTION

type updateItem = Same| Updated of updateContent * prevState| Errorand updateContent =UCDir of properties leafUpdate * updateChildren| UCFile of properties leafUpdate * contents leafUpdate| UCSymlink of contents leafUpdate| UCAbsentand updateChildren = (name * updateItem) list

OCaml

16

UNISON: BASIC DATA STRUCTURESRECONCILIATION

type direction = Conflict| LeftToRight| RightToLeft| Equal type transportInstr =Instr of updateItem * updateItem * direction| NoInstr| Problemtype transportInstrTree = Node of transportInstr * transportInstrListand transportInstrList = (name * transportInstrTree) list

OCaml

17

UNISON: BASIC OPERATIONS

• Comparison of two file trees• Description of detected difference

Update detection (buildUpdates)

• Building set of transport instructions

Reconciliation (reconcile)

• Performing transport instructions• Giving user the opportunity to verify

changes

Propagation (propagate)

18

UNISON: SEMANTIC FUNCTIONSFILESYSTEM OPERATIONS

• assoc• lookupPath• reverse• mem• fsKind• remove• deletions• buildUpdates• buildUpdateChildren

19

UNISON: SEMANTIC FUNCTIONSUPDATE DETECTION

• hasErrors• hasErrorsChildren• propagateErrors• noConflictInstr• reconcileNoConflict• reconcileNoConflictChildren• leafDirection• combineDirections• uassoc• reconcile• reconcileChildren• reconcileLeft

20

UNISON: SEMANTIC FUNCTIONSRECONCILIATION

• leafApply• updateArchive• updateArchiveChildren• emptySource• copyRec• copyChildren• copy• checkNoUpdates• replaceRec• replace

21

UNISON: SEMANTIC FUNCTIONSPROPAGATION

• performInstrLeaf• newProps• unchangedProps• performInstrDir• performInstr• propagateLocally• propagate• propagateInChildren

22

QUESTIONSKeying Xu:How do authors deal with modeling gap between the reference implementation and the specification?

Authors describe ‘modeling gap’ limitations in their paper

23

UNISON: THE “MODELING GAP”

Functional program (Ocaml) Returns new replicas

without changing content Written as if it “owns”

filesystems Regards filesystems as

simple, mathematical tree structures

Assumes that all operations can be implemented 111111111111

Treats archive as full-blown filesystem

Imperative program (C) Modifies real filesystems in-

place Runs with live filesystems

123123 Operates on real

implementations of filesystems (POSIX, NTFS, …)

Deals with operations that might be impossible to implement

Stores just a fingerprint of each file’s contents

Reference implementation (DSL)

Real implementation (software tool)

24

QUESTIONSBrent Carmer:What is the connection between Unison and your DSL?Panini Patapanchala:Relation with the version control and what aspects you can take from this paper.

• SCMF-DSL also uses concept of replication

• SCMF-DSL also operates with file trees

• SCMF-DSL also detects file changes to perform such automatic actions as version numbering

25

UNISON VS SCMF-DSL

Unison

SCMF-DSL

Deplo

ym

ent

Replication Version control

Version numberingConflicts

26

UNISON VS SCMF-DSL

Takes into account only latest synchronized state

Allows synchronization only between latest states

Treats both file trees as equal sources of changes (everything is writable)11111111111

There is no defined direction for replication 1111111111111111111

Operates with file trees Replicates contents of

filesystem Might generate incompatible

changes (conflicts)

Saves information about all synchronized states

Allows to roll back to previous states

Treats file trees as primary and secondary (can be writable or read-only)

Operates with certain direction for replication: from primary replica to secondary replica

Operates with version trees Replicates contents of version

control system Does not generate

incompatible changes

Unison SCMF-DSL

27

UNISON VS SCMF-DSL

A B

~

A B

=

sem

Unison: file trees

SCMF-DSL: version trees

sem1

x

2

3

57

8

9

11

x

4

6

x

10

1

x

2

3

57

8

9

11

x

4

6

x

10

12

28

UNISON VS SCMF-DSL

A B

~

A B

=

sem

Unison: file trees, 2 platforms (A, B)

SCMF-DSL: version trees, N platforms (P1, P2, … PN)

sem1

x

2

3

57

8

9

11

x

4

6

x

10

1

x

2

3

57

8

9

11

x

4

6

x

10

12

29

QUESTIONS

Brent Carmer:Does the user ever construct things using the types listed in the reference implementation?

NO

User uses real implementation (tool) instead of reference implementation

(DSL)

30

QUESTIONSAmin Alipour:How can they make sure that function synch is run atomically?Brent Carmer:How do they use Coq to verify their reference implementation?Panini Patapanchala:Maximal runs are unique can you justify the theorem with an example.

31

QUESTIONS

Authors use Coq to prove following properties of their DSL: • Laziness is safe (replication with itself is safe )• Mirroring is a special case (replication with

previously synchronized state o and replica a gives replica a)

• Maximal runs are unique (it is impossible to generate two different synchronizations on the same two replicas a and b)

• Success in the absence of conflicts (if replication does not generate conflicts first time, it won’t generate conflicts next time as well)

32

QUESTIONSAmin Alipour:The paper assumes that there are only two replica of filesystem's. Is that right? If so, how it can scale synchronization to more than two replicas?Rui Qin:How about more than two replicas, does it also work?Panini Patapanchala:I feel the paper explained the base cases for reconciliation and presently the more important problems are the one presented in future scope like multi-replica synchronization for more number of replicas.

NO

Unison works only with pairs of replicas

33

QUESTIONSAmin Alipour:What is the relation of conflict as described in the paper and merge in git?

Chao Peng:Is Unison easily extensible? can you conclude some low-level or high-level aspects of Unison?

34

QUESTIONSPanini Patapanchala:The buildupdate of the implementation is a bit like imperative implementation than functional. 

This is partly because of mixed nature of OCaml language -it incorporates functional, imperative and object-oriented paradigms