The Push/Pull Model of Transactions

Eric KoskinenIBM Research, New York

United States

Matthew ParkinsonMicrosoft Research, Cambridge

United Kingdom

PLDI 2015

Concurrent HashMap

Concurrent Queue

Concurrent List

Thread 1

Thread 3

Thread 2

Thread 4

put get enq deq add rm

Shared Memory

rd wr

Concurrent HashMap

Concurrent Queue

Concurrent List

Thread 1

Thread 3

Thread 2

Thread 4

put get enq deq add rm

Shared Memory

rd wr

Adding atomic to a language . . .

Version 4.7

Serializable. Opaque. …

Concurrent HashMap

Concurrent Queue

Concurrent List

Thread 1

Thread 3

Thread 2

Thread 4

put get enq deq add rm

Shared Memory

rd wr

Adding atomic to a language . . .

Concurrent HashMap

Concurrent Queue

Concurrent List

Thread 1

Thread 3

Thread 2

Thread 4

put get enq deq add rm

Shared Memory

rd wr

many issues arise in the implementation.

Serializable. Opaque. …? ? ?

Adding atomic to a language . . .

Pessimism!Thread

atomic { x := 3; y := 9; }

x changed?

y changed?

Thread

atomic { x := 3; y := 9; }all ok?

Optimism!

Serializability!

Interleaved execution equivalent to some

serial execution.

Opacity!

Interleaved execution equivalent to some

serial execution and

cannot observe intermediate state.

Pessimism!

Optimism!

Dependent Txns

ADT Ops (Boosting)

Thread

atomic { x := 3; y := 9; }

Thread

atomic { stk.push(4); ht.get('a'); }

Serializability!

Opacity!Pessimism!

Optimism!Memory Ops

ADT Ops (Boosting)

Serializability!

Opacity!Pessimism!

Optimism!Memory Ops

Hardware TMSoftware TM

Hybrid TM

Dependent Txns

ADT Ops (Boosting)

Serializability!

Opacity!Pessimism!

Optimism!Memory Ops

Hardware TMSoftware TM

Hybrid TM Open Nested

Dependent Txns

Irrev

ocab

ility

Consistency

Eager Writes

Lazy Reads

ADT Ops (Boosting)

Serializability!

Opacity!Pessimism!

Optimism!Memory Ops

Hardware TMSoftware TM

Hybrid TM Open Nested

Dependent Txns

Irrev

ocab

ility

Consistency

Eager Writes

Lazy Reads

Serializable. Opaque. …? ? ?

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

Serializable. …Opaque.

Serializable. …Opaque.

Linearizable Commute

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

Threadatomic { if(b) { opA(…); } else … }

Threadatomic { while(…){ if(x>MAX) opB(…); … }

Threadatomic { if(b) { opA(…); } else … }

Threadatomic { while(…){ if(x>MAX) opB(…); … }Linearizable

OperationLinearizable Operation

Thread Thread

{tx c1, σ1} {tx c2, σ2}

1. Abstract away the language semantics

Thread Thread

{tx c1, σ1} {tx c2, σ2}

1. Abstract away the language semantics

example language:

1. Abstract away the language semantics

Thread Thread

2. Abstract away the state (with logs)

{tx c1, σ1, L1} {tx c2, σ2, L2}G

⟨op1,_⟩⟨op2,_⟩

1. Abstract away the language semantics

Thread Thread

2. Abstract away the state (with logs)

{tx c1, σ1, L1} {tx c2, σ2, L2}G

Closed under log prefix

Parameterized by predicate allowed

⟨op1,_⟩⟨op2,_⟩

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}G

Closed under log prefix

Parameterized by predicate allowed

Log Equality

⟨op1,_⟩⟨op2,_⟩

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}G

Log Equality

⟨op1,_⟩⟨op2,_⟩

7 Simple Rules:

Apply Push Pull

Unpull Unpush Unapply

Commit

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}Two Logs

atomic { op1; op2; … }

atomic { op3; op4; … }

G

⟨ht.map(3,x),σ,σ1,c⟩⟨ht.map(7,2),_,gUC⟩

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }A

pp

ly

G

1

⟨ht.map(3,x),σ,σ1,c⟩⟨q.enq(‘a’),σ1,σ'1,c1⟩

⟨ht.map(7,2),_,gUC⟩

1

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }A

pp

ly

G

Criterion (i):Criterion (ii):Criterion (iii):

L1 allows ⟨m,σ1,σ'1,id⟩

fresh(id)

APPLY

1

⟨ht.map(3,x),σ,σ1,c⟩⟨q.enq(‘a’),σ1,σ'1,c1⟩

⟨ht.map(7,2),_,gUC⟩

1

{tx c1, σ1, L1}, G ➝

{tx c'1, σ'1, L1·[op,σ1,σ'1,c1]}, G

append

Possible next op

Valid semantics of log

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }U

na

pp

ly

G

UNAPPLY

⟨ht.map(7,2),_,gUC⟩⟨ht.map(3,x),σ,σ1,c⟩⟨q.enq(‘a’),σ1,σ'1,c1⟩

{tx c'1, σ'1, L1·[op,σ1,σ'1,c1]}, G ➝

{tx c1, σ1, L1}, G

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}Two Logs

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩

G

⟨ht.map(7,2),_,gUC⟩

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}Two Logs

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.map(3,x),_⟩Push

⟨q.enq(‘a’),_⟩⟨ht.map(3,x),_,gUC⟩

G

⟨ht.map(7,2),_,gUC⟩

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}Two Logs

atomic { op1; op2; … }

atomic { op3; op4; … }

Push

G

PUSH

Criterion (i):Criterion (ii):Criterion (iii):

op ◀ ⎣`L⎦unpushed

G allows op

Act as if op happens next

No conflict w/ other uncmted

Left-mover over logs⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩

⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

{tx c, σ, `L·[op]·L'}, G ➝

{tx c, σ, `L·[op]·L'}, G·[op,gUC]

⎣G⎦gUC ∖ ⎣L1⎦pushed ◀ op

Uncommitted

Application: Out-of-order PUSHing with redo-logsApplication: Optimism vs. Pessimism

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.map(3,x),_⟩Pull

G

PULL

L2 allows opCriterion (i):Criterion (ii):

op ◀ ⎣L2⎦pushed ∪ ⎣L2⎦unpushedCriterion (iii):

op ∉ L2Didn’t pull already

Local log allows op

Can act as if op happened earlier

⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩

⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

{tx c2, σ2, L2}, G1·[op,g]·G2 ➝

{tx c2, σ2, L2·[op]}, G1·[op,g]·G2

Application: Opacity [GK’08] and dependent transactions [RRHW’09]

⟨q.enq(‘a’),_,gUC⟩Push

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }A

pp

ly ⟨ht.get(5),_⟩

G

⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩

⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

Push⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩

Pull

G

⟨ht.get(5),_⟩⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩

G

⟨ht.get(5),_⟩⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

Pull⟨q.enq(‘a’),_⟩

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩

G

⟨ht.get(5),_⟩⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

⟨q.enq(‘a’),_⟩⟨ht.get(9),_⟩

Co

mm

it

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩

G

⟨ht.get(5),_⟩⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

⟨q.enq(‘a’),_⟩⟨ht.get(9),_⟩

Co

mm

it

COMMIT

cmt(G, L1, G')⎣L1⎦pulled ⊆ ⎣G⎦gC

Criterion (ii):Criterion (iii):Criterion (iv):

L1 ⊆ GCriterion (i): fin(c1)

Pushed all my stuff

Pulled ops are committed

Swap my flags from gUC to gC

{tx c1, σ1, L1}, G ➝

{tx c1, σ1, L1}, G’

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩⟨ht.get(5),_⟩

⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩

⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

⟨q.enq(‘a’),_⟩⟨

Un

ap

ply

G

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩⟨ht.get(5),_⟩

⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩

⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

⟨q.enq(‘a’),_⟩U

np

ull

G

UNPULL

(Local log is still allowed, even with element removed.)

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩

⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩

⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

⟨q.enq(‘a’),_⟩

G

Unpush

UNPUSH

(Everything I subsequently pushed could have been pushed earlier.)

Application: Inverses or “compensating actions”

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩

⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩

⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

⟨q.enq(‘a’),_⟩

G

Un

ap

ply

q.deq()

Apply Push Pull Unpull Unpush Unapply Commit

Thread Thread

{tx c1, σ1, L1} {tx c2, σ2, L2}

atomic { op1; op2; … }

atomic { op3; op4; … }

⟨ht.get(5),_,gUC⟩⟨ht.get(5),_⟩

⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩⟨q.enq(‘a’),_⟩

⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

⟨q.enq(‘a’),_⟩

G

Un

ap

ply

q.deq()

Theorem. The Push/Pull model is serializable.Simulation with uninterleaved machine

Closure under log rewind

Preservation invariant, universal quantification over dropping uncommitted global ops universal quantifying over local log rewind.

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

Theorem. Fragments of The Push/Pull model are opaque.

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

example: Don’t PULL uncommitted effects.

Other Applications.

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

• Pessimistic models of STM. • Pessimistic models of STM/Boosting. • Optimistic models of STM. • Optimistic models of STM/Boosting (TRANSACT’15) • Closed-nested transactions: don’t share until commit. • Dependent transactions: view uncommitted effects but

delay commit (or cascading aborts)

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

Mixing hardware TM with transactional boosting

Models that are yet-to-come.

Serializable. …Opaque.

Linearizable Commute

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

The PUSH/PULL Model

Push⟨ht.get(5),_⟩

Pull⟨ht.map(3,x),_⟩⟨ht.map(3,x),_⟩

⟨q.enq(‘a’),_⟩ ⟨q.enq(‘a’),_,gUC⟩⟨ht.map(3,x),_,gUC⟩⟨ht.map(7,2),_,gUC⟩

Eric KoskinenIBM Research, New York

United States

Matthew ParkinsonMicrosoft Research, Cambridge

United Kingdom

Thank you!