Distributed deterministic dataflowprogramming for Erlang

Manuel Bravo 1 Zhongmiao Li 1 Peter Van Roy 1

Christopher Meiklejohn 2

1

Université catholique de Louvain

2

Basho Technologies, Inc.

Erlang User ConferenceStockholm, Sweden, 2014

June 9, 2014

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 1 / 37

Overview

1 Introduction

2 Background

3 Semantics

4 Implementation

5 Examples

6 Caveats and future work

7 References

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 2 / 37

SyncFree

Funded by the EuropeanUnionFocusing on Conflict-freeReplicated Data Types(CRDTs)Basho, Rovio, TriforkINRIA, Universidade Nova deLisboa, Université Catholiquede Louvain, Koç Üniversitesi,Technische UniversitätKaiserslautern

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 3 / 37

SyncFree

Build a programming model for conflict-free replicated data types(CRDTs). [12]Deterministic, distributed, parallel programming in Erlang.Similar work to LVars [10] and Bloom. [5]Key focus on distributed computation, high scalability, andfault-tolerance.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 4 / 37

Conflict-free replicated data types

Comes in two main flavors: state-based and operations-based.State-based CRDTs:

Data structure which ensures convergence under concurrent operations.Based on bounded join-semilattices.Data structure which grows state monotonically.

Imagine a vector clock.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 5 / 37

Motivation

Erlang implements a message-passing execution model in whichconcurrent processes send each other asynchronous messages.This model is inherently non-deterministic, in that a process canreceive messages sent by any process which knows its processidentifier.Concurrent programs in non-deterministic languages, are notoriouslyhard to prove correct.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 6 / 37

Correctness

Treat every message received by a process as a ‘choice’.A series of these ‘choices’ define one execution of a program.Prove each execution is correct; or terminates.Further complicated by distributed Erlang and its semantics. [13]OTP is essentially "programming patterns" to reduce this burden.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 7 / 37

Contributions

An "alternative" approach to this non-determinism.Deterministic data flow programming model for Erlang, implementedas a library.Concurrent programs, which regardless of execution, produce the sameresult.Fault-tolerance and distribution of computations provided byriak_core. [3]

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 8 / 37

Deterministic dataflow programming

Historically:1974: First proposed as Kahn networks. [7]1977: Lazy version of this same model was proposed by Kahn andDavid MacQueen [9].

More recently:CTM/CP: Oz [14]Akka [1, 15]Ozma [6]

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 9 / 37

Single-assignment store

Relies on a single assignment store: � = {x1

, . . . , xn}Example: � = {x

1

= x

2

, x2

= ?, x3

= 5, x4

= [a, b, c], . . . , xn = 9}Where:

xi = ?: Variable xi is unbound.xi = xm: Variable xi is partially bound; therefore, it is assigned toanother dataflow variable (xm). This also implies that xm is unbound.xi = vi : Variable xi is bound to a term (vi ).

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 10 / 37

Metadata

xi = {value, waiting_processes, bound_variables}Where:

value : empty, or dataflow value.waiting_processes : processes waiting for xi to be bound.bound_variables : dataflow variables which are partially bound.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 11 / 37

Basic Primitives I

declare() creates a new dataflow variable.Before: � = {x1, . . . , xn}xn+1 = declare()

create a unique dataflow variable xn+1

store xn+1

into �

After: � = {x1, . . . , xn+1 = ?}bind(xi , vi ) binds the dataflow variable xi to the value vi .

Before: � = {x1, . . . , xi = ?, . . . , xn}bind(xi , vi )

8p 2 xi .waiting_proccesses,notify p

8x 2 xi .bound_variables, bind(x , vi )xi .value = vi

After: � = {x1, . . . , xi = vi , . . . , xn}

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 12 / 37

Basic Primitives II

read(xi ) returns the term bound to xi .Before: � = {x1, . . . , xi , . . . , xn}vi = read(xi )

if xi .value == (xm _?)

xi .waiting_processes [ {self ()}wait

vi = xi .value

After: � = {x1, . . . , xi = vi , . . . , xn}thread(function, args) runs function(args) in a different process.

Implemented using the Erlang spawn primitive.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 13 / 37

Streams I

Streams of dataflow variables: si = x

1

| . . . | xn�1

| xn, xn = ?Extend metadata to store pointer to next position:xi = {value, waiting_processes, bound_variables, next}produce(xn, vn) extends the stream by binding the tail xn to vn andcreating a new tail xn+1

.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 14 / 37

Stream Primitives I

produce(xn, vn) extends the stream by binding the tail xn to vn andcreating a new tail xn+1

.Before: � = {x1, . . . , xn = ?}xn+1 = produce(xn, vn)

bind(xn, vn)xn+1

= declare()xn.next = xn+1

After: � = {x1, . . . , xn = vn, xn+1 = ?}

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 15 / 37

Stream Primitives II

consume(xi ) reads the element of the stream represented by xi .Before: � = {x1, . . . , xi = vi _ xm _?, xi+1, . . . , xn}{vi , xi+1} = consume(xi )

vi = read(xi )xi+1

= xi .next

After: � = {x1, . . . , xi = vi , xi+1, . . . , xn}

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 16 / 37

Laziness

Provide non-strict evaluation primitive.Extend metadata:xi = {value, waiting_processes, bound_variables, next, lazy}wait_needed(x) suspends until the caller until x is needed.

Before: � = {x1, . . . , xi = ?, . . . , xn}wait_needed(xi )

if xi .waiting_processes == ;xi .lazy [ self ()wait until a read(xi ) is issued

After: � = {x1, . . . , xi , . . . , xn}Modify read operation to notify, if lazy.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 17 / 37

Non-determinism

Provide a primitive which supports non-deterministic execution.Introduces non-determinism because it allows a choice to be taken onwhether the variable is bound or not.is_det(x) determines whether a variable is bound yet.

Before: � = {x1, . . . , xi , . . . , xn}bool = is_det(xi )

bool = xi .value == vi

After: � = {x1, . . . , xi , . . . , xn}

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 18 / 37

Failure handling

Failures introduce non-determinism.One approach: wait forever until the variables are available.Does not ensure progress, for example:

Process p0 is supposed to bind a dataflow variable,however fails before completing its task.Processes p1 . . . pn are waiting on p0 to bind.Processes p1 . . . pn wait forever, resulting in non-termination.

Two classes of errors:Computing process failures.Dataflow variable failure.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 19 / 37

Computing process failures

Consider the following:Process p0 reads a dataflow variable, x1.Process p0 performs a computation based on the value of x1, and bindsthe result of computation to x2.

Two possible failure conditions can occur:If the output variable never binds, process p0 can be restarted and willallow the program to continue executing deterministically.If the output variable binds, restarting process p0 has no effect, giventhe single-assignment nature of variables.

Handled via Erlang primitives.Supervisor trees; restart the processes.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 20 / 37

Dataflow variable failures

Consider the following:Process p0 attempts to compute value for dataflow variable x1 and fails.Process p1 blocks on x1 to be bound by p0, which will not completesuccessfully.

Re-execution results in the same failure.Explore extending the model with a non-usable value.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 21 / 37

Deterministic dataflow API

{id, Id::term()} = declare():Creates a new unbound dataflow variable in the single-assignmentstore. It returns the id of the newly created variable.{id, NextId::term()} = bind(Id, Value):Binds the dataflow variable Id to Value. Value can either be an Erlangterm or any other dataflow variable.{id, NextId::term()} = bind(Id, Mod, Fun, Args):Binds the dataflow variable Id to the result of evaluatingMod:Fun(Args).Value::term() = read(Id):Returns the value bound to the dataflow variable Id. If the variablerepresented by Id is not bound, the caller blocks until it is bound.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 22 / 37

Streams

{id, NextId::term()} = produce(Id, Value):Binds the variable Id to Value.{id, NextId::term()} = produce(Id, Mod, Fun, Args):Binds the variable Id to the result of evaluating Mod:Fun(Args).{Value::term(), NextId::term()} = consume(Id):Returns the value bound to the dataflow variable Id and the id of thenext element in the stream. If the variable represented by Id is notbound, the caller blocks until it is bound.{id, NextId::term()} = extend(Id):Declares the variable that follows the variable Id in the stream. Itreturns the id of the next element of the stream.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 23 / 37

Laziness

ok = wait_needed(Id):Used for adding laziness to the execution. The caller blocks until thevariable represented by Id is needed when attempting to read the value.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 24 / 37

Non-determinism

Value::boolean() = is_det(Id):Returns true if the dataflow variable Id is bound, false otherwise.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 25 / 37

Partition strategies

Each variable has a home process, which coordinates notifying allprocesses which should be told of changes in binding.Each process knows information about all processes which should benotified.Partitioning of the single assignment store, where processes

communicate to the local process.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 26 / 37

Design considerations

mnesia

Problems during network partitions. [8]Allows independent progress with no way to reconcile changes.Replication not scalable enough or provide fine-grained enough control.

riak_core

Minimizes reshuffling of data through consistent hashing andhash-space partitioning.Facilities for causality tracking. [11]Anti-entropy and hinted handoff.Dynamic membership.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 27 / 37

Riak Core

DHT with fixed partitionsize/count.Partitions claimed onmembership change.Replication overring-adjacent partitions.(preference lists)Sloppy quorums (fallbackreplicas) for added durability.

Figure : Ring with 32 partitions and3 nodes

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 28 / 37

Implementation on riak_core

Partition the single-assignment store across the cluster.Writes are performed against a strict quorum of the replica set.As variables become bound:

Notify all waiting processes using a strict quorum.In the event of node failures, anti-entropy mechanism is used to updatereplicas which missed the update during handoff.

Under network partitions, we do not make progress.In the event of a failure, we can restart the computation at any point.

Redundant re-computation doesn’t cause problems.Dynamic membership.

Transfer the portion of the single-assignment store held locally to thetarget replica.Duplicate notifications are not problematic.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 29 / 37

Concurrent map example

Concurrent map example

concurrent_map(S1, M, F, S2) ->case derflow:consume(S1) of

{nil, _} ->derflow:bind(S2, nil);

{Value, Next} ->{id, NextOutput} = derflow:extend(S2),spawn(derflow, bind, [S2, M, F, Value]),concurrent_map(Next, F, NextOutput)

end.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 30 / 37

Caveats with non-determinism

Given the following processes: � = {x1

, x2

, x3

, x4

, x5

}Process p0 binds x1Process p1 reads x1 and binds x2.Process p2 reads x2, does some non-deterministic operation.

Using is_det on x6, which may or may not be bound based on

scheduling.

Process p3 reads x3 and binds x4.Process p4 reads x4 and binds x5.

Possible failures:If execution fails in p0 or p1, we can restart.If execution fails in p3 or p4, we can restart p3 and p4, and continueon without worrying about non-determinism.If execution fails in p2, what do we do?

Local vs. global side-effects?

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 31 / 37

Future work

Generalize variables to join semi-lattices.Currently a semi-lattice with two states: bound and unbound.Use the diverse set of CRDTs available in Erlang. [4]Provide eventually consistent computations, which deterministic valuesregardless of the execution model.

Provide an analysis tool to determine where you are introducingnon-determinism.

Similar to the Deadalus work. [2]Possible use for Dialyzer here?

Explore alternative syntax.Parse transformation.Some other type of grammar.

Make the library a bit more idiomatic.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 32 / 37

References I

Akka: Building powerful concurrent and distributed applications more easily,2014.

P. Alvaro, W. Marczak, N. Conway, J. M. Hellerstein, D. Maier, and R. C.Sears.Dedalus: Datalog in time and space.Technical Report UCB/EECS-2009-173, EECS Department, University ofCalifornia, Berkeley, Dec 2009.

Basho Technologies Inc.Riak core source code repository.http://github.com/basho/riak_core.

Basho Technologies Inc.Riak dt source code repository.http://github.com/basho/riak_dt.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 33 / 37

References II

N. Conway, W. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier.Logic and lattices for distributed programming.Technical Report UCB/EECS-2012-167, EECS Department, University ofCalifornia, Berkeley, Jun 2012.

S. Doeraene and P. Van Roy.A new concurrency model for scala based on a declarative dataflow core.In Proceedings of the 4th Workshop on Scala, SCALA ’13, pages 4:1–4:10,New York, NY, USA, 2013. ACM.

K. Gilles.The semantics of a simple language for parallel programming.In In Information Processing’74: Proceedings of the IFIP Congress,volume 74, pages 471–475, 1974.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 34 / 37

References III

Joel Reymont.[erlang-questions] is there an elephant in the room? mnesia networkpartition.http://erlang.org/pipermail/erlang-questions/2008-November/

039537.html.

G. Kahn and D. MacQueen.Coroutines and networks of parallel processes.In Proc. of the IFIP Congress, volume 77, pages 994–998, 1977.

L. Kuper and R. R. Newton.Lvars: Lattice-based data structures for deterministic parallelism.In Proceedings of the 2Nd ACM SIGPLAN Workshop on FunctionalHigh-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA,2013. ACM.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 35 / 37

References IV

N. M. Preguiça, C. Baquero, P. S. Almeida, V. Fonte, and R. Gonçalves.Dotted version vectors: Logical clocks for optimistic replication.CoRR, abs/1011.5808, 2010.

M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski.Conflict-free replicated data types.In X. Défago, F. Petit, and V. Villain, editors, Stabilization, Safety, andSecurity of Distributed Systems, volume 6976 of Lecture Notes in ComputerScience, pages 386–400. Springer Berlin Heidelberg, 2011.

H. Svensson and L.-A. Fredlund.Programming distributed erlang applications: Pitfalls and recipes.In Proceedings of the 2007 SIGPLAN Workshop on ERLANG Workshop,ERLANG ’07, pages 37–42, New York, NY, USA, 2007. ACM.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 36 / 37

References V

P. Van Roy and S. Haridi.Concepts, techniques, and models of computer programming.MIT press, 2004.

D. Wyatt.Akka concurrency: Building reliable software in a multi-core world.Artima, 2013.

Bravo et al (Louvain; Basho) Distributed deterministic dataflow EUC ’14 37 / 37