Offline-firstapplication data and synchronization18 February 2016

⌘

Maksym Barbul

2

Maksym Barbul

Senior Software Engineermaxbarbul [at]

gmail.com

Introduction

3

Problem statement

4

So, why is it a problem?

● user is blocked in offline● connection (blink, detection delay,

unstable)● data is always changingIt may bring user to loose data 5

What is different?

6

● Client UX (non-blocking, responsive, reliable)

● Client has smart caching● Client partially duplicates server’s logic● Controlled connectivity● Smart synchronization● Client-side security and audit

Offline-first is a Paradigmis a distinct set of concepts or thought patterns, including theories, research methods, postulates, and standards for what constitutes legitimate contributions to a field 7

§

Who needs these super-smart apps?

● Field workers● Store observation● Technical audit● Medical

observationB

US

INE

SS

EFFICIENCY8

Who needs these super-smart apps?

● Instant messengers● Social networks● Document editors● Media organizers● GTD toolsA

LL U

SE

RS

LOYALTY9

Offline Application

10

Components of an offline-first app

11

Troubleshooting

User Experience

Offline Data

Connectivity

Synchronization

Offline First

How is an offline-first app built?

12

Offline data

Synchronization

Connectivity

User Experience

Trouble-shooting

Offline Data

13

Data slices

14

Application

Static Meta-Data

BusinessData

System Runtime Data

Data slices: Application

15

Application

Static Meta-Data

BusinessData

System Runtime Data

Core (scripts)Modules (scripts)Styles & Views (scripts, CSS)Media resources

Data slices: Static Meta-Data

16

Application

Static Meta-Data

BusinessData

System Runtime Data

Configuration & SettingsLayout DeclarationPermissionsTranslations

Data slices: Business Data

17

Application

Static Meta-Data

BusinessData

System Runtime Data

DictionariesFetched Data (Read only)Entered Data (User Data)

Data slices: System Runtime Data

18

Application

Static Meta-Data

BusinessData

System Runtime Data

Logs (action, error, profiling)Session DataTransaction States (Sync)

Data Synchronization

19

Data Synchronization Types

20

Data Synchronization Types

21

Replication

Synchronization (replication) Strategies

22

● Pessimistic - no inconsistency● Optimistic - eventual consistency

Eventual Consistency

means that replicas eventually reach the same value if clients stop submitting updates

23

§

Optimistic replication algorithm

24

1.Operation submission (user submission)2.Propagation to other replicas3.Scheduling of operations to apply by

each replica4.Conflict resolution (syntactic, semantic)5.Commitment on a final schedule and

conflicts resolution result

Strong Eventual Consistency (SEC)

is a property of some eventually-consistent systems: replicas that have received and applied the same set of updates must immediately have equivalent state

25

§

Conflict-free replicated data type (CRDT)

26

is a type of specially-designed data structure used to achieve strong eventual consistency (SEC) and monotonicity (absence of rollbacks)

● operation-based

● state-based

Operation-based CRDTs (CmRDT)

27

commutative replicated data types

The operations are commutative, so can be received and applied in any order; however, they are not idempotent, and additional network protocol guarantees are required to ensure unique delivery

a * b = b * a

O(a) ≠ O(O(a))

State-based CRDTs (CvRDT)

28

convergent replicated data types

CvRDTs send their full local state to other replicas

State-based CRDTs (CvRDT): Interface

29

➔ query - reads the state of the replica, with no side effects

➔ update - writes to the replica state in accordance with certain restrictions (monotonically increase)

➔ merge - merges local state with the state of some remote replica (commutative, associative, and idempotent)

a * b = b * aO(a, b) = O(O(a,

b), b)

a*(b*c) = (a*b)*c

CvRDT: increment only

30

is an element offor each i

CvRDT: grow-only set (add)

31

add {e} to set A

S.A is a subset of T.A

unite sets

CvRDT: 2P-set (add, remove)

32

Operational transformation (OT)

33

is a technology for supporting a range of collaboration functionalities in advanced collaborative software systems. OT was originally invented for consistency maintenance and concurrency control in collaborative editing of plain text documentsApach

e Wave

Google Docs

Operational transformation (OT)

34

➔ O1 = Insert[0, "x"] // “xabc”(to insert character "x" at position "0")

➔ O2 = Delete[2, "c"] // “xabc” ???(to delete the character "c" at position "2")

“abc”

➔ * O2 = Delete[3, "c"] // “xab”

Operational transformation (OT)

35

Unfortunately, implementing OT sucks. There's a million algorithms with different tradeoffs, mostly trapped in academic papers. The algorithms are really hard and time consuming to implement correctly. ... Wave took 2 years to write and if we rewrote it today, it would take almost as long to write a second time.Joseph Gentle

who is an ex Google Wave engineer and an author of the Share.JS

Example: Chat

36

{String} title

{Object} participants

{Array} messages

CvRDT increment-only (LWW)

CvRDT set

CmRDT or CvRDT

Example: Booking

37

{Date} date_from,

date_to

{String} user_id

{String} room_id

{String} booking_status

CvRDT (LWW)

NONE (not editable)

CvRDT (LWW)

CvRDT

Example: Collaborative field editing

38

{String} textOperational

transformation

Don’t forget about AUDIT

39

●Why do we have this value?●Who did this change?●When?●What was the previous value?

References: read theory● Offline First manifesto http://offlinefirst.org/● Links to articles, events, samples https://github.com/offlinefirst● Designing Offline-First Web Apps http://alistapart.com/article/offline-first ● SAY HELLO TO OFFLINE FIRST http://hood.ie/blog/say-hello-to-offline-first.html ● Optimistic replication https://www.wikiwand.com/en/Optimistic_replication ● CRDT http://www.wikiwand.com/en/Conflict-free_replicated_data_type ● OT http://www.wikiwand.com/en/Operational_transformation ● appSync.org: open-source patterns & code for data synchronization in mobile apps

http://confluence.tapcrowd.com/pages/viewpage.action;jsessionid=3F4D2C44DBFC46644A7955F82A416DC2?pageId=2262404

● * Free Book “Distributed Systems” by Mikito Takada http://book.mixu.net/distsys/index.html● Roshi CRDT explanation

https://developers.soundcloud.com/blog/roshi-a-crdt-system-for-timestamped-events● A comprehensive study of CRDTs by Marc Shapiro

https://hal.inria.fr/file/index/docid/555588/filename/techreport.pdf

40

References: source code● appSync source https://bitbucket.org/nikonelissen/appsync/src ● CRDT Roshi by SoundCloud with great expanation https://github.com/soundcloud/roshi ● CRDT Riak by Basho (League of Legends) https://github.com/basho/ ● ShareJS https://github.com/share/ShareJS ● Hoodie https://github.com/hoodiehq/hoodie

41

References: borrowed resources● Sync types diagram http://confluence.tapcrowd.com/pages/viewpage.action?pageId=2262404 ● Free icons found here http://findicons.com/

42

Summary: how to build offline-first apps

43

● Know the Paradigm● Plan data structure and

synchronization approaches before development

● Be paranoid about user’s data● Develop offline-first applications

The End?

44

Thank you.eager to use your great Offline

apps

😉

Questions...

45

🙋