Linked Lists: Optimistic, Lock-Free, … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Modified by Pavol Černý,

Linked Lists: Optimistic, Lock-Free, …

Companion slides forThe Art of Multiprocessor Programming

by Maurice Herlihy & Nir Shavit

Modified by Pavol Černý, Programming Paradigms for

Concurrency, Fall 2010, IST Austria

Art of Multiprocessor Programming 2

Set Interface

• Unordered collection of items• No duplicates• Methods

– add(x) put x in set– remove(x) take x out of set– contains(x) tests if x in set


List-Based Sets

public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(T x);}


List Node

public class Node { public T item; public int key; public Node next;}


The List-Based Set

a b c

Sorted with Sentinel nodes(min & max possible keys)

-∞

+∞


Invariants

• Sentinel nodes– tail reachable from head

• Sorted• No duplicates


This Lecture

• Introduce three “patterns”– Bag of tricks …– Methods that work more than once …


This Lecture

• Introduce three “patterns”– Bag of tricks …– Methods that work more than once …

• For highly-concurrent objects– Concurrent access– More threads, more throughput


First:Optimistic Synchronization

• Search without locking …• If you find it, lock and check …

– OK: we are done– Oops: start over

• Evaluation– Usually cheaper than locking, but– Mistakes are expensive


Second:Lazy Synchronization

• Postpone hard work• Removing components is tricky

– Logical removal• Mark component to be deleted

– Physical removal• Do what needs to be done


Third:Lock-Free Synchronization

• Don’t use locks at all– Use compareAndSet() & relatives …

• Advantages– No Scheduler Assumptions/Support

• Disadvantages– Complex– Sometimes high overhead


Drawbacks of fine-grained locking

• Better than coarse-grained lock– Threads can traverse in parallel

• Still not ideal– Long chain of acquire/release– Inefficient


Optimistic Synchronization

• Find nodes without locking• Lock nodes• Check that everything is OK


Optimistic: Traverse without Locking

b d ea

add(c)

Aha!


Optimistic: Lock and Load

b d ea

add(c)


Optimistic: Lock and Load

b d ea

add(c)

c


What could go wrong?

b d ea

add(c)

Aha!



b d ea

add(c)



b d ea

remove(b)



b d ea

remove(b)



b d ea

add(c)



b d ea

add(c)

c



d ea

add(c) Uh-oh


Validate – Part 1

b d ea

add(c)

Yes, b still

reachable from head


What Else Could Go Wrong?

b d ea

add(c)

Aha!


What Else Coould Go Wrong?

b d ea

add(c)

add(b’)


What Else Coould Go Wrong?

b d ea

add(c)

add(b’)b’



b d ea

add(c)

b’



b d ea

add(c)

c


Validate Part 2(while holding locks)

b d ea

add(c)

Yes, b still

points to d


Optimistic: Linearization Point

b d ea

add(c)

c


Correctness

• If– Nodes b and c both locked– Node b still accessible– Node c still successor to b

• Then– Neither will be deleted– OK to delete and return true


Unsuccessful Remove

a b d e

remove(c)

Aha!


Validate (1)

a b d e

Yes, b still reachable from head

remove(c)


Validate (2)

a b d e

remove(c)

Yes, b still points to

d


OK Computer

a b d e

remove(c)

return false


Correctness

• If– Nodes b and d both locked– Node b still accessible– Node d still successor to b

• Then– Neither will be deleted– No thread can add c after b– OK to return false


Validationprivate boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}


private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Predecessor & current

nodes



Validation

Begin at the beginning



Validation

Search range of keys



Validation

Predecessor reachable


private boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Is current node next?



Validation

Otherwise move on



ValidationPredecessor not

reachable


Remove: searchingpublic boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …


public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

Remove: searching

Search key



Remove: searching

Retry on synchronization conflict



Remove: searching

Examine predecessor and current nodes



Remove: searching

Search by key



Remove: searching

Stop if we find item



Remove: searching

Move along


On Exit from Loop

• If item is present– curr holds item– pred just before curr

• If item is absent– curr has first higher key– pred just before curr

• Assuming no synchronization problems


Remove Methodtry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}


try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally {


}}}

Remove Method

Always unlock




}}}

Remove Method

Lock both nodes




}}}

Remove Method

Check for synchronization

conflicts




}}}

Remove Method

target found, remove node




}}}

Remove Method

target not found


Optimistic List

• Limited hot-spots– Targets of add(), remove(), contains()– No contention on traversals

• Moreover– Traversals are wait-free– Food for thought …


So Far, So Good

• Much less lock acquisition/release– Performance– Concurrency

• Problems– Need to traverse list twice– contains() method acquires locks


Evaluation

• Optimistic is effective if– cost of scanning twice without locks

is less than– cost of scanning once with locks

• Drawback– contains() acquires locks– 90% of calls in many apps


Lazy List

• Like optimistic, except– Scan once– contains(x) never locks …

• Key insight– Removing nodes causes trouble– Do it “lazily”


Lazy List

• remove()– Scans list (as before)– Locks predecessor & current (as

before)• Logical delete

– Marks current node as removed (new!)

• Physical delete– Redirects predecessor’s next (as

before)


Lazy Removal

aa b c d

c


Lazy Removal

aa b d

Present in list

c


Lazy Removal

aa b d

Logically deleted


Lazy Removal

aa b c d

Physically deleted


Lazy Removal

aa b d

Physically deleted


Lazy List

• All Methods– Scan through locked and marked

nodes– Removing a node doesn’t slow down

other method calls …• Must still lock pred and curr nodes.


Validation

• No need to rescan list!• Check that pred is not marked• Check that curr is not marked• Check that pred points to curr


Business as Usual

a b c


Business as Usual

a b c


Business as Usual

a b c


Business as Usual

a b c

remove(b)


Business as Usual

a b c

a not marked


Business as Usual

a b c

a still points to b


Business as Usual

a b c

Logical

delete


Business as Usual

a b c

physical

delete


Business as Usual

a b c


Validationprivate boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); }


private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); }

List Validate Method

Predecessor not Logically removed




Current not Logically removed




Predecessor stillPoints to current


Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {


}}}




}}}

Validate as before




}}}

Key found




}}}

Logical remove




}}}

physical remove


Containspublic boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}



Start at the head



Search key range



Traverse without locking(nodes may have been

removed)



Present and undeleted?


Summary: Wait-free Contains

a 0 0 0a b c 0e1d

Use Mark bit + list ordering 1. Not marked in the set2. Marked or missing not in the set


Lazy List

a 0 0 0a b c 0e1d

Lazy add() and remove() + Wait-free contains()


Evaluation

• Good:– contains() doesn’t lock– In fact, its wait-free! – Good because typically high %

contains()– Uncontended calls don’t re-traverse

• Bad– Contended add() and remove() calls

do re-traverse– Traffic jam if one thread delays


Traffic Jam

• Any concurrent data structure based on mutual exclusion has a weakness

• If one thread– Enters critical section– And “eats the big muffin”

• Cache miss, page fault, descheduled …– Everyone else using that lock is stuck!– Need to trust the scheduler….


Reminder: Lock-Free Data Structures

• No matter what …– Guarantees minimal progress in any

execution– i.e. Some thread will always complete

a method call– Even if others halt at malicious times– Implies that implementation can’t use

locks


Lock-free Lists

• Next logical step– Wait-free contains()– lock-free add() and remove()

• Use only compareAndSet()– What could go wrong?


Remove Using CAS

a 0 0 0a b c 0e1c

Logical Removal =Set Mark Bit

PhysicalRemovalCAS pointer

Use CAS to verify pointer is correct

Not enough!


Problem…

a 0 0 0a b c 0e1c


PhysicalRemovalCAS

0dProblem: d not added to list…Must Prevent manipulation of removed node’s pointer

Node added BeforePhysical Removal CAS


The Solution: Combine Bit and Pointer

a 0 0 0a b c 0e1c


PhysicalRemovalCAS

0d

Mark-Bit and Pointerare CASed together(AtomicMarkableReference)

Fail CAS: Node not added after logical Removal


Solution

• Use AtomicMarkableReference• Atomically

– Swing reference and– Update flag

• Remove in two steps– Set mark bit in next field– Redirect predecessor’s pointer


Marking a Node

• AtomicMarkableReference class– Java.util.concurrent.atomic package

address F

mark bit

Reference


Extracting Reference & Mark

Public Object get(boolean[] marked);


Extracting Reference & Mark

Public Object get(boolean[] marked);

Returns reference

Returns mark at array index

0!


Extracting Mark Only

public boolean isMarked();

Value of mark


Changing State

Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark);


Changing State


If this is the current reference

…

And this is the current mark …


Changing State


…then change to this new reference …

… and this new mark


Changing State

public boolean attemptMark( Object expectedRef, boolean updateMark);


Changing State


If this is the current reference …


Changing State


.. then change to this new mark.


Removing a Node

a b c d

remove c

CAS


Removing a Node

a b d

remove b

remove c

cCASCAS

failed


Removing a Node

a b d

remove b

remove c

c


Removing a Node

a d

remove b

remove c


Traversing the List

• Q: what do you do when you find a “logically” deleted node in your path?

• A: finish the job.– CAS the predecessor’s next field– Proceed (repeat as needed)


Lock-Free Traversal(only Add and Remove)

a b c dCAS

Uh-oh

pred currpred curr


The Window Class

class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }}


The Window Class

class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }}

A container for pred and current

values


Using the Find Method

Window window = find(head, key); Node pred = window.pred; curr = window.curr;




Find returns window




Extract pred and curr

Art of Multiprocessor Programming© Herlihy-Shavit 2007

126

The Find Method

Window window = find(item);

At some instant,

pred curr succ

itemor …

Art of Multiprocessor Programming© Herlihy-Shavit 2007

127

The Find Method

Window window = find(item);

At some instant,

predcurr= null

succ

item not in list


Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}



Keep trying


Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}} Find neighbors


Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}} She’s not there …



Try to mark node as deleted



If it doesn’t work, just retry, if it does, job

essentially done



Try to advance reference(if we don’t succeed, someone else did or will).

a


Addpublic boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}}


Addpublic boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}} Item already there.



create new node



Install new node, else retry loop


Wait-free Contains

public boolean contains(Tt item) { boolean marked; int key = item.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }


Wait-free Contains

public boolean contains(T item) { boolean marked; int key = item.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }

Only diff is that we get and check

marked


Lock-free Findpublic Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}


Lock-free Findpublic Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

If list changes

while traversed, start overLock-Free

because we start over

only if someone

else makes progress


public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Start looking from head



Lock-free Find

Move down the list



Lock-free Find

Get ref to successor and current deleted bit



Lock-free Find

Try to remove deleted nodes in path…code details

soon



Lock-free Find

If curr key that is greater or equal, return pred and

curr



Lock-free Find

Otherwise advance window and loop again


Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…


Lock-free Find


Try to snip out node


Lock-free Find


if predecessor’s next field changed must

retry whole traversal


Lock-free Find


Otherwise move on to check if next node

deleted


Performance

On 16 node shared memory machineBenchmark throughput of Java List-based Setalgs. Vary % of Contains() method Calls.


High Contains Ratio

Lock-free Lazy list

Course GrainedFine Lock-coupling


Low Contains Ratio

Lock-free

Lazy list



As Contains Ratio Increases

Lock-free

Lazy list


% Contains()


Summary

• Coarse-grained locking• Fine-grained locking• Optimistic synchronization• Lock-free synchronization


Homework

Projects

• Topic: • good: choose from among our suggestions• better: define your own project

• On your own or in groups of two

• Pick a project by: before Christmas• Progress report 1: January 15th (2 pages)• Presentation and final report: January 27th and

February 3rd (final report: 4 pages)

Project 1: Irregular data parallelism

Cavity

Example: Delaunay mesh refinement

Effects of updates are local, but not bounded statically (“irregular”).

Can we still exploit locality for parallelism?

Project 1: Irregular data parallelism

http://iss.ices.utexas.edu/lonestar/index.html

Locality of effects: Mesh retriangulation

Project 1: Irregular data parallelismLonestar benchmark suite:http://iss.ices.utexas.edu/lonestar/index.html

1. Barnes-Hut N-body Simulation

2. Delaunay Mesh refinement3. Focused communities4. Delaunay triangulation5. …

Project: 1. pick one of these applications, 2. find a good (possibly novel) way of parallelizing it, 3. implement it (by modifying the sequential implementation provided in Lonestar benchmarks), 4. confirm improvement in running time by experimentation.

Project 2: Deductive verification: proving a concurrent data structure correct

1. Pick an implementation of a concurrent data structure: a stack, a queue, a set, ..

2. Pick a theorem prover or a verification tool: for example: PVS or QED

3. Prove that the implementation is linearizable

3 5 7 9

P1: remove(7)

P2: remove(5)

Project 2: Deductive verification: proving a concurrent data structure correct

References:1. R. Colvin, L. Groves, V. Luchangco, M. Moir: Formal Verification of a Lazy

Concurrent List-Based Set Algorithm. CAV 20062. T. Elmas, S. Qadeer, A. Sezgin, O. Subasi, S. Tasiran: Simplifying

Linearizability Proofs with Reduction and Abstraction. TACAS 2010.

3 5 7 9

P1: remove(7)

P2: remove(5)

Project 3:Performance measurement/ performance model for concurrent programs

1. Pick a problem with at least three-four different solutions1. Lock implementations2. Data structures: queues, stacks, sets…

2. Examine the performance of the solutions in different settings:1. small number of threads vs large number of threads2. 2 cores, small amount of memory (laptop) vs 8 cores, large

memory/cache (server)3. different usage models4. input that generates little vs input that generates lots of contention

3a. Find a hybrid solution that works well in a particular settingor

3b. Find a performance model that explains the data

Project 4: Your own!


Summary• Mutual Exclusion Algorithms (Peterson,

Filter, Bakery), Lower Bounds (Covering proof)

• Correcteness conditions: 1. Linearizability, 2. Sequential consistency

• Proving Linearizability: 1. Rely-Guarantee, 2. Reductions (left movers, right movers)

• Case study: Concurrent Lists (coarse-grained, fine-grained, optimistic, lazy, lock-free implementations)

Documents

Linked Lists: Optimistic, Lock-Free, … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Modified by Pavol Černý,