HyflowCPP: A Distributed Transactional MemoryFramework for C++

Sudhanshu Mishra, Alexandru Turcu, Roberto Palmieri, Binoy RavindranECE Department, Virginia Tech, Blacksburg, VA, 24061

{mishras, talex, robertop, binoy}@vt.edu

Abstract—We present the first ever distributed transactionalmemory (DTM) framework for distributed concurrency control inC++, called HyflowCPP. HyflowCPP provides distributed atomicsections, and pluggable support for policies for concurrency con-trol, directory lookup, contention management, and networking.While there exists other DTM frameworks, they mostly targetVM-based languages (e.g., Java, Scala). Additionally, HyflowCPPprovides uniquely distinguishing TM features including strongatomicity, closed and open nesting, and checkpointing. Ourexperimental studies revealed that HyflowCPP achieves up to6x performance improvement over state-of-the-art DTM frame-works.

I. INTRODUCTION

Transactional systems based on Software TransactionalMemory (STM) are increasingly becoming a promising tech-nology for designing and implementing transactional applica-tions. STM is a framework for managing concurrent operationson a shared data set. With STM, programmers organize codethat read/write in memory elements (e.g., memory chunks orobject pointers) as memory transactions, and the frameworktransparently ensures transactional properties such as atomic-ity, consistency, and isolation [1]. The transactional abstrac-tion thus allows programmers to easily implement concurrentapplications by simply enclosing parts of the code accessingshared data in transactions, without being burdened with themechanisms needed to ensure transactional properties. More-over, relying on STM means that, programmers do not haveto cope with deadlocks, livelocks, lock convoying, etc., whichare the typical pitfalls when synchronization mechanisms aremanually implemented. As a result, code reliability increasesand development time is shortened.

The nature of STM in memory transactions results in trans-action execution times that are several orders of magnitudesmaller than that in conventional transactional systems [2].This allows STM systems to simultaneously service a massivenumber of concurrent requests. In fact, STM’s popularity hasgrown with the widespread adoption of multi-core architec-tures, simply because these architectures can naturally supportan increased level of application parallelism.

Distributed STM is a logical way to exploit STM’s ad-vantages in distributed settings i.e., systems with nodes inter-connected using message passing links. In particular, pitfallsof manual, lock-based distributed synchronization (e.g., dis-tributed deadlocks, distributed livelocks) are orders of magni-tude harder to debug than those in a multiprocessor setting.DTM’s (distributed) transactional abstraction is therefore acompelling distributed synchronization abstraction. Addition-ally, DTM protocols are increasingly becoming a way to target

scalability due to their superior performance in comparisonwith manually implemented distributed locks [3], [4], [5].

Most existing DTM frameworks are prototyped on top ofVM-based programming languages (e.g., Java, Scala): Cluster-STM [6], D2STM [7], DiSTM [8], GenRSTM [9], Hyflow-Java [3], and HyflowScala [10]. Some DTM implementationshave been developed for C and C++: DMV [11], Cluster-STM [6], and Sinfonia [12]. However, DMV and Cluster-STM are implemented as proofs-of-concept and Sinfonia isdesigned as a backup and restore service, rather than as DTMframeworks.

C++ is one of the most popular programming languagesfor high performance systems. In the last decade a numberof protocols for DTM have been proposed, and some ofthem have been developed. The languages usually adoptedfor their implementation are based on virtual machine (e.g.,Java, Scala). The reason is typically related to the currenttechnological trend. In fact, in order to easily integrate suchsystems with benchmarks, applications and other sub-systemsrecently implemented, the designers adhere to the employmentof these programming languages. Even though the support fora transparent integration with other systems is a critical featurefor several products, their performance still certainly representsthe core comparison point among different possible solutions.

Lower level programming languages, like C++, enable theprogrammer to exploit architectural advantages that could biassystem performance. This is especially true in transactionalapplications based on in memory operations like STM or DTMin which the overhead for layering the framework, ensuringinformation hiding and wrapping methods typically results inmore complex execution flow composed by several steps thatsignificantly impact the overall performance. This is the reasonmany high-performance production systems are developed inC++, instead of VM-based languages, usually, to overcomeperformance issues inherent in VM-based languages [13].

In addition, memory management in C++ is manual. Eventhough this results in a more complex implementation of trans-actional framework, it potentially eliminates garbage collectionoverheads of VM-based environments. Moreover, consider-ing the absence of I/O interactions in the STM transactionsprocessing, having the direct control of memory manage-ment allows specific optimization unfeasible with VM-basedlanguages. Therefore, a C++ framework for DTM is highlydesirable as that enables DTM support for a popular language,and provides high-performance distributed concurrency controlfor C++ applications with high programmability.

Motivated by these observations, we design and imple-

2013 IEEE 12th International Symposium on Network Computing and Applications

978-0-7695-5043-5/13 $26.00 © 2013 IEEE

DOI 10.1109/NCA.2013.30

219

ment HyflowCPP, the first ever DTM framework for C++.HyflowCPP has a modular architecture and provides pluggablesupport for various DTM policies (e.g., concurrency control,directory lookup, contention management, networking) and asimple atomic section-based DTM interface. The distributedconcurrency control currently implemented in the frameworkis TFA [14]. TFA uses Herlihy and Sun’s dataflow executionmodel [15] (in which objects migrate and transactions areimmobile). The adoption of TFA as a baseline protocol formanaging concurrent requests allows a fair comparison ofHyflowCPP against the same TFA implementation in two VM-based DTM frameworks such as HyflowJava [3] and HyflowS-cala [16], highlighting the actual benefit of the lower levelprogramming language. Furthermore, HyflowCPP embeds sup-port for using closed [17] and open [18] nested transactionsand checkpointing [19]. To the best of our knowledge, no pastDTM frameworks in C++ support all these features.

We evaluated HyflowCPP through a set of experimentalstudies that measured transactional throughput on an array ofmicro- and macro-benchmarks, and compared with past VM-based DTM systems such as GenRSTM, DecentSTM, Hyflow-Java, and HyflowScala. The key result from our experimentalstudies is that, HyflowCPP outperforms its nearest competitor,HyflowScala, by a factor of 6. Other competitors, includingGenRSTM, DecentSTM, and HyflowJava, perform even worsein comparison to HyflowScala.

Additionally, we conducted an evaluation contrasting thetwo typical approaches for implementing partial rollback oftransactions as a way to minimize the time to retry a transactionafter its abort, namely checkpointing and closed-nesting. Theexperiments reveal that, unlike in [20] where closed-nesting isalways better than checkpointing due to the cost of saving andrestoring memory images, in HyflowCPP, exploiting the effi-cient mechanism for saving execution states without incurringsignificant overheads [21], the checkpointing technique ensuresbetter performance. Generally, we show that HyflowCPP’scheckpointing outperforms flat nesting by as much as 100%and closed nesting by as much as 50%. Open nesting-basedtransactions outperform flat nesting by as much as 140%and closed nesting by as much as 90%. These trends areconsistent with past DTM studies on nesting [17], [18], but ourrelative improvements (i.e., checkpointing and open nestingwhen compared to closed nesting) are higher, which can beattributed to a more efficient design and the limited overheadsof various mechanisms in C++ [13].

The rest of the paper is organized as follows: Section IIdescribes TFA, transaction nesting and checkpointing, whichrepresent the background for this work. Section III details theprogramming model of HyflowCPP and Section IV describesHyflowCPP’s architecture. We report our experimental resultsin Section V, and conclude in Section VI.

II. BACKGROUND

In this section we provide a brief introduction to TFA, thebase protocol implemented in HyflowCPP. We then proceed todescribe the different transaction nesting models and transac-tion checkpointing.

A. TFA Protocol

Transactional Forwarding Algorithm (TFA) [14] is a lock-based DTM algorithm with lazy lock acquisition and bufferedwrites. All read objects are stored in a local read-set, so allreads can later be revalidated. Written objects are also storedin a local write-set.

TFA uses a variant of the Lamport clocks mechanism toestablish “happens before” relationships across nodes. Eachdistributed node has a local clock lc and atomically incrementsits local clock on the commit of every transaction that changesthe shared state (write transactions). All messages sent betweennodes piggyback the local clock value of the sender node.Upon receiving such a message, each node compares theremote clock value included in the message with its own localclock. If the remote value is greater, the local clock is updatedto this greater value.

Each transaction records the local clock value lc at the timeit starts (i.e., starting clock, sc). Then, when it communicateswith remote nodes (for the purpose of accessing new objects),it compares the clock value of the remote node (rc) to itsown start clock (sc). If rc > sc, the transaction undergoesa Transactional Forwarding procedure: it validates its read-set, and, should that be successful, updates its starting time tosc = rc. If the validation fails the transaction aborts. Validationis performed by comparing the object’s latest version with thecurrent transaction’s start clock.

B. Nested Transactions and Checkpointing

Transactions are nested when they appear within anothertransaction’s boundary. Transaction nesting makes code com-posability easy: multiple operations inside a transaction will beexecuted atomically, regardless of whether the said operationscontain transactions or not, and without breaking encapsula-tion. This is an important advantage of transactional memorywhen compared to traditional lock-based synchronization.

Three transactional nesting models were proposed in theliterature: Flat, Closed [22] and Open [23], [22].

Flat nesting is the simplest form of nesting, which simplyignores the existence of transactions in inner code. All op-erations are executed in the context of the parent transaction.Aborting any inner-transaction causes the parent transaction toabort. Clearly, flat nesting does not provide any performanceimprovement over non-nested transactions.

Closed nesting allows inner-transactions to abort individ-ually. Aborting an inner-transaction does not necessarily leadto also aborting the parent transaction (i.e., partial rollback ispossible). However, inner-transactions’ commits are not visibleoutside the parent transaction. An inner-transaction commits itschanges only into the private context of its parent transaction,without exposing any intermediate results to other transactions.Only when the parent transaction commits, the shared state ismodified.

Open nesting considers the operations performed by sub-transactions at a higher level of abstraction, in an attemptto avoid false conflicts occurring at the memory level. Itallows inner-transactions to commit or abort individually, andtheir commits are globally visible immediately. In case an

220

enclosing transaction aborts, due to any fundamental conflicts(i.e., not false) at the higher levels of abstraction, all the inner-transactions are roll-backed by using compensating actions,which are predefined for each abstract operation.

Checkpointing [24], [25] addresses this issue by allowingexecution to return to any previously saved state (checkpoint)within the current transaction, regardless of whether the sub-transaction encompassing that checkpoint is still active or not.This allows developing a very fine grained partial rollbackmechanism, which can identify the exact operation to rollbackexecution to, in order to resolve the current conflict. On abort, acheckpoint that can resolve the conflict is located and activated,effectively reverting transaction execution to the state it had atthe time the checkpoint was originally taken. The programcontrol flow is managed by saving and restoring the thread’sexecution state (i.e., CPU registers and activation stack) andemploys a mechanism called continuations [26].

III. PROGRAMMING INTERFACE

Since objects are dispersed over nodes in a distributedsetting and normal object references cannot be used, weprovide a base class called HyflowObject, which must beinherited by each object. HyflowObject provides a getId()method, which returns a unique key to access the objectfrom anywhere in the network. HyflowCPP serializes/de-serializes objects using the Boost serialization library [27].Each extended HyflowObject field is registered with the Boostserialization function, for serialization/de-serialization over thenetwork. HyflowCPP provides transactional interfaces usingmacros.

A. Transaction Support using Macros

HYFLOW ATOMIC START, HYFLOW ATOMIC END.These macros are provided to define atomic sections. Anyobject can be opened in the Read or Write mode. Once anobject is requested, HyflowCPP fetches the object from itscurrent location and copies it to the transaction read-set orwrite-set depending upon the access type. Objects are accessedusing the macro HYFLOW FETCH invoked by the frameworkand not accessible by the programmer.HYFLOW ON READ, HYFLOW ON WRITE. Thesemacros are used for reading or writing, respectively, the fetchedobjects. HYFLOW ON READ returns a constant referencepointer to HyflowObject. HYFLOW ON WRITE returns anon constant object reference pointer. The constant referencepointer can also be used in place of unique object key toretrieve an object in write mode.HYFLOW PUBLISH OBJECT. This macro is used for pub-lishing a locally created object in the distributed directory.HYFLOW PUBLISH DELETE This macro is used to pub-lish a cancellation of an object in the distributed directory.HYFLOW CHECKPOINT INIT. This macro is used forinitiating the checkpointing environment.HYFLOW CHECKPOINT HERE. This is used for creatinga checkpoint of transactional execution.HYFLOW STORE. The macro is used by the programmerwhen the checkpointing environment is activated for savingany primary data structure object on the stack or the heap.For heap objects, the programmer is responsible for creatingobject copies and managing memory; the macro only saves the

c l a s s Lis tNode : : Hyf lowObjec t {. . .void Lis tNode : : addNode ( i n t v a l u e ) {

HYFLOW ATOMIC START{s t d : : s t r i n g head=”HEAD” ;HYFLOW FETCH( head , f a l s e ) ;L i s tNode∗ headNRead =( L i s tNode ∗ )

HYFLOW ON READ( head ) ;s t d : : s t r i n g o ldNex t =headNRead−>g e t N e x t I d ( ) ;L i s tNode∗ newNode=new Lis tNode ( va lue ,

L i s tBenchmark : : g e t I d ( ) ) ;newNode−>s e t N e x t I d ( o ldNex t ) ;HYFLOW PUBLISH OBJECT( newNode ) ;L i s tNode∗ headNodeWri te =( L i s tNode ∗ )

HYFLOW ON WRITE( head ) ;headNodeWrite−>s e t N e x t I d ( newNode−>g e t I d ( ) ) ;

} HYFLOW ATOMIC END;}void Lis tNode : : d e l e t e N o d e ( i n t v a l u e ) {

HYFLOW ATOMIC START{s t d : : s t r i n g head ( ”HEAD” ) ;s t d : : s t r i n g p rev =head , n e x t ;HYFLOW FETCH( head , t rue ) ;t a r g e t N =( Li s tNode ∗ )HYFLOW ON READ( head ) ;n e x t = t a r g e t N−>g e t N e x t I d ( ) ;whi le ( n e x t . compare ( ”NULL” ) != 0){

HYFLOW FETCH( next , t rue ) ;t a r g e t N =( Li s tNode ∗ )HYFLOW ON READ( n e x t ) ;i n t nodeValue= t a r g e t N−>g e t V a l u e ( ) ;i f ( nodeValue == v a l u e ) {

Lis tNode∗ prevNode =( Li s tNode ∗ )HYFLOW ON WRITE( p rev ) ;

L i s tNode∗ c u r r e n t N o d e =( L i s tNode ∗ )HYFLOW ON WRITE( n e x t ) ;

prevNode−>s e t N e x t I d ( cu r r en tNode−>g e t N e x t I d ( ) ) ;

HYFLOW DELETE OBJECT( c u r r e n t N o d e ) ;break ;

}prev = n e x t ;n e x t = t a r g e t N−>g e t N e x t I d ( ) ;

}} HYFLOW ATOMIC END;

}}

Fig. 1. The implementation of add and remove operations in HyflowCPP.

void BankAccount : : t r a n s f e r ( Acc1 , Acc2 , Money ) {HYFLOW ATOMIC START{

HYFLOW CHECKPOINT INIT ;wi thdraw ( Acc1 , Money , c o n t e x t ) ;HYFLOW CHECKPOINT HERE;d e p o s i t ( Acc2 , Money , c o n t e x t ) ;

}HYFLOW ATOMIC END;}

Fig. 2. Checkpointing in a Bank transfer function

address value. The macro’s arguments include the variable ref-erence and value, and it automatically restores the saved valuewhen the transaction resumes from the selected checkpoint.

Figure 1 reports the implementation of add and removeoperations on a Linked List using the presented HyflowCPPmacros. Figure 2 shows how checkpointing can be imple-mented in a transaction (part of Bank benchmark). Note thatthe current context instance must be passed to the withdrawfunction. This requirement exists for all functions which arecalled within the atomic block and are required to be executed

221

atomically.

IV. SYSTEM ARCHITECTURE

Figure 3 shows HyflowCPP’s architecture of a transactionalnode. It is made of six modules. The transaction interfacemodule is summarized in Section III. Here, we will focus ona subset of the rest of the modules.

� �

Fig. 3. HyflowCPP’s node’s architecture

Transaction Validation Module.

The entire concurrency control logic is performed in the Trans-action Validation Module (TVM) by extending the base classHyflowContext. By default, HyflowContext is extended as DTL-Context, which is responsible for providing the implementationof the concurrency control algorithm (i.e., TFA). DTLContextis an interface offering all the calls needed for implementingits own concurrency control protocols. Extending this class, theprogrammer can easily integrate a new contention manager inthe HyFlowCpp. TVM validates memory locations and retriesthe transactional code if needed upon commit or abort. Themodule can also be configured for supporting checkpointing,closed and open nesting.

This module also provides the data structures needed formanaging all the transactional contexts (i.e., the meta-dataassociated to the on-going transactions). The most relevantstructures are: the lock-table, used for object-level or word-level locking (implemented using the concurrent hash-mapof TBB library [28]); the context-map, a concurrent hashtable providing access to transactional contexts indexed bytransaction ID. The latter structure enables the contentionmanager to manage meta-data and to make updates to differenttransactional contexts.

Object Access Module

The Object Access Module (OAM) provides facilities tomanage distributed shared objects and to interact with thedistributed objects directory for changing the object ownership,performing remote validation and updating the directory itself.Objects are located using the unique object ID. The moduleencapsulates a directory lookup protocol to access distributedobjects.

The OAM contains two efficient concurrent hash-maps: lo-cal cache and object directory. The local cache maintainsauthoritative copies of objects owned by the current node, andthe object directory maintains the meta data information (e.g.,object owner information in the case of tracker directory).Similar to the TVM, the programmer can rely on one orboth of these structures for implementing the object locationservice. By default, it implements the efficient tracker directoryprotocol, which moves an object across nodes and maintainsthe current owner information on a specific tracker node. Thisis because TFA requires an object directory for changing theobject ownership when transactions commit.

OAM closely interacts with the TVM. In fact, to providestrong atomicity, HyflowCPP directs all accesses to objectsthrough the context-map managed by TVM. Therefore, allobject requests to the OAM go through the TVM. Objectdeletion or publication requests made by the TVM are servedin the OAM. This module also handles object validationrequests.

Object Serialization Module

To free programmers from message serialization and de-serialization, HyflowCPP provides two classes: HyflowMes-sage and BaseMessage. BaseMessage acts as a parent classfor any message in HyflowCPP; it can be extended to createany new message type to perform a protocol-specific operation.HyflowMessage provides a standard interface for networking.All HyflowMessage objects are converted in a binary blob,before forwarding to the network library, to communicateover the network. In this way, network implementation isindependent of the transaction validation module and the objectaccess module. Serialization and de-serialization of messagesis done using Boost [27].

Network Module

This module (NM) provides pluggable support for a networklibrary. An efficient network interaction support is fundamen-tal for distributed protocol managing transactions executeddirectly in main memory. This is because the transactionexecution time is dominated by network time. Relying on anhigh performance communication layer significantly reducesthe transaction response time.

HyflowCPP provides two networking libraries: MsgCon-nect [29] and ZeroMQ [30]. In the current release, in order todesign a fast and scalable networking solution, we use industrystandard ZeroMQ library. It is a socket level library, providingvery efficient solutions for in-process communication betweenthreads, which makes it suitable for any multi-threaded net-working requirement.

Any two transactional nodes A and B communicate witheach other using the forwarder and catcher networking threads.These threads are responsible for connecting with differentnodes using the ZeroMQ router socket. Forwarder threads re-ceive message requests from transactional threads and forwardthem to catcher threads of the target nodes, which assignthem to available worker threads. Worker threads process themessage and send back reply to catcher threads if required.Catcher threads return the messages back to the forwarderthreads, which route them to the original requester transac-tional threads.

222

Our studies revealed that ZeroMQ provides 4 to 5 timesbetter performance over MsgConnect. In addition, using multi-part messaging of ZeroMQ, we are able to reduce the amountof polling between threads to a bare minimum at two sockets.

Message Processing Module

This module (MPM) provides message handling capabilityto the NM. When a programmer using the framework forimplementing own protocol creates a new message type, amessage handler is also created for that message and registeredusing the message handler interface. Later, when the NMreceives a request message, MPM creates a proper responseusing the message handler, and replies back as required.

The module also enables asynchronous messaging usinga class HyflowMessageFuture, which allows a transactionalthread to send a message asynchronously, proceed with othertasks, and later wait for the message response.

V. IMPLEMENTATION AND EXPERIMENTAL EVALUATION

A. Implementation

HyflowCPP has been implemented from scratch adoptingtechnologies and design choices for making the frameworkgeneral and optimized for in memory transactions. The im-plementation of flat nesting execution model (i.e., plain trans-actions without nesting) follows by the TFA protocol rules.Additional mechanisms are needed for implementing nestingsupports.

Regarding the closed-nesting supports, for each transac-tional context, we maintain a reference to its parent transactioncontext. In the commit phase, we directly merge the contextobjects with the parent context objects for inner-transactions.The commit procedure of outermost transactions is the sameas that in flat nesting.

To support the open open-nesting model, we use abstractlocks as higher-level locks for inner-transactions. Further, weallow programmers to define onAbort and onCommit functions,which are to be performed in case of abort or commit ofthe parent transaction. Abstract locks are acquired in inner-transactions and released in the parent transaction, on com-mit or abort. Each inner-transaction commits as a pure flat-transaction, releasing its transactional context, included read-set and write-set.

The implementation of checkpointing model is the one thattakes more advantages from the manual memory managementallowed by C++ language. We partition the read-set, write-set, publish-set and the delete-set objects on the basis ofthe first access checkpoint. This allows us to identify theconflicting objects and calculate the dependency to determinethe valid checkpoint. To save the transaction execution stackstate, we use the setContext and getContext functions. Heapobjects are maintained in the context read-set, write-set, andpublish-set. HyflowCPP provides a helper class CheckPoint-Provider to create, iterate and maintain transaction check-points. HYFLOW STORE macro is provided to store anystack or heap object values just before creating a checkpoint(see Section III). The values are automatically restored uponcontinuing from a checkpoint after partial abort. On commitfailure, instead of restarting the transaction, we resume from an

available valid checkpoint. Also, all acquired locks are releasedbefore resuming.

B. Experimental Evaluation

In this section we describe the evaluation results ofHyflowCPP compared with the other state of the art JVM-based DTM systems. Being the first ever DTM frameworkin C++, we cannot compare it with other C++ based DTMs,therefore we evaluated HyflowCPP by comparing with otherJVM-based DTM frameworks.

We firstly contrasted our HyflowCPP with two state of theart DTM frameworks: GenRSTM [9] and DecentSTM [31].Subsequently, for the rest of the evaluation, we directly com-pared HyflowCPP with its JVM-based counterparts: Hyflow-Java [3] and HyflowScala [10]. We excluded the initial com-petitors from the comparison because an extensive evaluationhas already been presented in [32] where HyflowJava has beenshown to outperform both GenRSTM and DecentSTM.

Our experiments were conducted using up to 48 nodeslocated in a private cluster and interconnected by a Gigabitconnection, running two application threads per node. We usethe Ubuntu Linux 10.04 server OS.

Our evaluation workloads included both micro-benchmarksand macro-benchmarks. Regarding the former, we used thedistributed version of: Linked List, Skip List, Binary SearchTree and Hash Table. Regarding the latter we used Bank,Loan and a distributed version of the STAMP’s Vacation [33].For the micro-benchmarks, each transaction consists of sev-eral operations on the shared data-structure. For the macro-benchmarks: Bank simulates a banking system; Loan simulatesa mortgage lending setting; Vacation simulates an itineraryplanning and reservation system.

We divided the evaluation in three subsections correspond-ing to the three transaction execution models supported byHyflowCPP: flat nesting, closed-nesting/checkpointing, open-nesting. Each data-point in the plots is the average of 3repeated experiments. Due to space constraints we cannotpresent all the plots. The complete evaluation study can befound in [34].

Flat Nesting

Experiments with flat transactions allow us to understand theraw performance of different DTM frameworks. In Figure 4the comparison between HyflowCPP and other JVM-basedDTM frameworks is presented. The plots clearly show the

0

2000

4000

6000

8000

10000

12000

14000

16000

0 10 20 30 40 50

Thr

ough

put (

tran

sact

ions

/s)

Number of Nodes

HyflowCPPHyflowJava

HyflowScalaDecentSTMGenRSTM

(a) Bank 80% read

0

2000

4000

6000

8000

10000

12000

14000

16000

0 10 20 30 40 50

Number of Nodes

HyflowCPPHyflowJava

HyflowScalaDecentSTMGenRSTM

(b) Bank 20% read

Fig. 4. Comparison between HyflowCPP and GenRSTM, DecentSTM,HyflowJava, HyflowScala (20% and 80% read workload)

performance gap between HyflowCPP and other competitors.

223

As mentioned at the begin of this section, in the rest of theevaluation HyflowJava and HyflowScala have been used ascompetitors.

Micro-benchmarks. For Linked List, Skip List, and BST, 50objects are deployed; for Hash Table, 2000 distributed bucketswere used. Objects and buckets are uniformly distributed overnodes. Each benchmark has been configured producing twodifferent workloads, one read intensive, characterized by 80%of read-only transactions, and the second with 20% of read-only transaction, therefore mostly write intensive.

0

50

100

150

200

250

300

350

400

450

500

550

0 10 20 30 40 50

Thr

ough

put (

tran

sact

ions

/s)

Number of Nodes

HyflowCPPHyflowJava

HyflowScala

(a) Linked List 80% read

0

100

200

300

400

500

600

700

0 10 20 30 40 50

Number of Nodes

HyflowCPPHyflowJava

HyflowScala

(b) Linked List 20% read

0

200

400

600

800

1000

1200

1400

1600

1800

0 10 20 30 40 50

Thr

ough

put (

tran

sact

ions

/s)

Number of Nodes

HyflowCPPHyflowJava

HyflowScala

(c) Skip List 80% read

0

500

1000

1500

2000

2500

0 10 20 30 40 50

Number of Nodes

HyflowCPPHyflowJava

HyflowScala

(d) Skip List 20% read

0

5000

10000

15000

20000

25000

30000

0 10 20 30 40 50

Thr

ough

put (

tran

sact

ions

/s)

Number of Nodes

HyflowCPPHyflowJava

HyflowScala

(e) Hash Table 80% read

0

5000

10000

15000

20000

25000

30000

0 10 20 30 40 50

Number of Nodes

HyflowCPPHyflowJava

HyflowScala

(f) Hash Table 20% read

0

200

400

600

800

1000

1200

1400

1600

0 10 20 30 40 50

Thr

ough

put (

tran

sact

ions

/s)

Number of Nodes

HyflowCPPHyflowJava

HyflowScala

(g) BST 80% read

0

500

1000

1500

2000

2500

0 10 20 30 40 50

Number of Nodes

HyflowCPPHyflowJava

HyflowScala

(h) BST 20% read

Fig. 5. Performance of micro-benchmarks with (20% and 80% read workload)

Figure 5 shows the performance of HyflowCPP againstHyflowJava/Scala. Linked List and Skip List are affected by anumber of conflicts due to the reduced number of objects avail-able in the system. This scenario mainly stresses the mech-anisms needed for implementing the distributed concurrencycontrol. Due to the smaller read-write set size, Hash Table’sthroughput is highest among all the benchmarks. Linked List’sthroughput is lower than Hash Table’s due to the high abortrate caused by the large read-set. Conversely, the Skip List’sperformance are better than Linked List due to smaller read set

size. BST exhibits high level of concurrency; its performanceis up to 2× better in comparison to Linked List.

The plots reveal that HyflowCPP performs up to 3 to 5×better than its nearest competitor HyflowScala. HyflowJava isworse than HyflowScala. We recall that all the DTMs testedimplement TFA. Here the HyflowCPP’s high throughput canbe mainly attributed to the fast network layer, obtained fromthe higher quality implementation with optimized load balanc-ing, synchronization and packet handling, which yields lowerqueuing delays, response times, and smaller TCP connectionre-initiations. In addition, we observed average CPU timeutilization for HyflowScala to be around 20%-30%, whereasHyflowCPP utilized 50%-60%, which supports our argumentregarding the minimized CPU idle time due to the optimizednetwork management layer.

Macro-benchmarks. The analysis with macro-benchmarks areuseful for understanding DTM performance in a more “realworld” settings. Their transactions are composed by severaloperations leading to high messaging and transaction executiontime. Due to their characteristics, such benchmarks are appro-priate to analyze possible bottlenecks in terms of messaging,synchronization or memory usages in the framework. Unfor-tunately, HyflowScala does not provide the implementation ofVacation and Loan, therefore HyflowCPP has been contrastedwith the HyflowJava for Vacation and Loan. We configuredthe benchmarks with a total of 10,000 accounts/objects.

0

200

400

600

800

1000

1200

1400

1600

1800

0 10 20 30 40 50

Thr

ough

put (

tran

sact

ions

/s)

Number of Nodes

HyflowCPPHyflowJava

(a) Vacation (80% reads)

0

500

1000

1500

2000

2500

0 10 20 30 40 50

Number of Nodes

HyflowCPPHyflowJava

(b) Vacation (20% reads)

0

500

1000

1500

2000

2500

3000

0 10 20 30 40 50

Thr

ough

put (

tran

sact

ions

/s)

Number of Nodes

HyflowCPPHyflowJavaDecentSTM

(c) Loan (80% reads)

0

500

1000

1500

2000

2500

0 10 20 30 40 50

Number of Nodes

HyflowCPPHyflowJavaDecentSTM

(d) Loan (20% reads)

Fig. 6. Transactional throughput of macro-benchmarks

Figures 6 and 4 show results for 20% and 80% read ratio.HyflowCPP, in general, performs up to 3 to 5× better thancompetitors (5 to 10× better for Loan). It is relevant to noticethe scalability trend of HyflowCPP, which can be approximatedas linear. Conversely, the others performance after 20 or 30nodes (depending on the benchmark) stall, showing a verylimited scalability compared to our proposal.

The plots clear reveal the positive impact of the optimiza-tion allowed by the manual memory and network managementof C++ instead of JVM-based implementation.

Checkpointing and Closed Nesting

224

Closed nesting and checkpointing are two techniques forimplementing partial rollback after an abort is issued. A per-formance comparison between these two models also appearedin [20].This paper assesses that supporting saving and restoringof large memory chunks is more costly than running closed-nested transactions, electing closed-nesting as reference modelfor implementing partial abort. The goal of this study is toshow how HyflowCPP can subvert the latter decision rein-forcing checkpointing as the better way to restore transactionexecution from a safe point. Indeed, no JVM-based competitorsare included.

To understand checkpointing and closed nesting’s bene-fits over flat nesting, we conducted experiments on micro-benchmarks and Bank, because other macro-benchmarks donot define rules for splitting transactions in sub-transactions.For the checkpointing experiments, we inserted a checkpointjust before accessing an object (or a subset of objects accordingto the selected checkpointing granularity). For the closednesting experiments, we performed each operation as an inner-transaction. In order to do a fair comparison, we configuredthe benchmarks to perform 20% of read-only transactions.In this way we increase the conflict probability, thereforethe likelihood to incur an abort. In the plots we varied thenode count (2− 16 for micro-benchmarks and 2− 28 forBank benchmark) and the inner-transactions/checkpoints count(2, 5, 10) to understand the impact of transactional lengthand partial rollback points on the performance. Increasingthe number or inner-transactions/checkpoints allows a moreaccurate conflict resolution and an efficient transaction restart.

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 4 8 12 16

Thr

ough

put r

elat

ive

to fl

at n

estin

g

Number of Nodes

Flat-NestingCheckpointing-2Close-Nesting-2Checkpointing-5Close-Nesting-5

Checkpointing-10Close-Nesting-10

(a) List

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 4 8 12 16

Number of Nodes

Flat-NestingCheckpointing-2Close-Nesting-2Checkpointing-5Close-Nesting-5

Checkpointing-10Close-Nesting-10

(b) Skip List

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 4 8 12 16

Thr

ough

put r

elat

ive

to fl

at n

estin

g

Number of Nodes

Flat-NestingCheckpointing-2Close-Nesting-2Checkpointing-5Close-Nesting-5

Checkpointing-10Close-Nesting-10

(c) Hash Table

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 10 20 30 40 50

Number of Nodes

Flat-NestingCheckpointing-1Close-Nesting-1Checkpointing-2Close-Nesting-2Checkpointing-5Close-Nesting-5

Checkpointing-10Close-Nesting-10

(d) Bank

Fig. 7. HyflowCPP’s performance of closed-nesting and checkpointing

Figure 7 shows checkpointing and closed nesting’sthroughput relative to flat nesting. Both outperform flat nest-ing by up to 100%. Also, as the inner-transactions countincreases, checkpointing and closed nesting’s performanceimprove, due to increase in partial rollback points. In allthe experiments, checkpointing guarantees better performancethan closed-nesting. The efficient mechanism for saving andrestoring the transaction’s context allows HyflowCPP to catchthe exact operation that caused the abort, minimizing thetime spent by the transaction to re-execute code that is stillconsistent, and quickly restore the valid memory image. This

result has been obtained relying on C++ features not availablein other JVM-based languages in which, closed-nesting isunquestionably preferred to checkpointing due to the resultingworse performance of the latter.

Open Nesting

To understand open nesting’s benefits, we conducted experi-ments on Hash Table, Skip List and BST. Open-nested trans-actions were created similar to closed-nested using multipleread/write operations performed as inner-transactions, but withan additional abstract lock mechanism.

In the evaluation we contrasted the performance of open-nesting against closed-nesting, reporting their throughput im-provement with respect to flat nesting. For each benchmarktwo plots are reported. In both we fixed the number ofshared objects in the system and we varied the number ofprocessing nodes (2− 48). Then, in the first plot we set thepercentage of read-only transactions to 20%, generating anhigh conflict scenario, and we varied the number of inner-transactions (2, 3, 4, 8). In the second plot we fixed the numberof nested transactions to 3, and we varied the workload movingthe read ratios (20%, 50%, 80%).

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 10 20 30 40 50

Thr

ough

put r

elat

ive

to fl

at n

estin

g

Number of Nodes

Flat-NestingOpen-Nesting-2

Closed-Nesting-2Open-Nesting-3

Closed-Nesting-3Open-Nesting-4

Closed-Nesting-4Open-Nesting-8

Closed-Nesting-8

(a) Hash Table 20% read

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 10 20 30 40 50

Number of Nodes

Flat-NestingOpen-Nesting-Reads20

Closed-Nesting-Reads20Open-Nesting-Reads50

Closed-Nesting-Reads50Open-Nesting-Reads80

Closed-Nesting-Reads80

(b) Hash Table with 3 inner-txs

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 10 20 30 40 50

Thr

ough

put r

elat

ive

to fl

at n

estin

g

Number of Nodes

Flat-NestingOpen-Nesting-2

Closed-Nesting-2Open-Nesting-3

Closed-Nesting-3Open-Nesting-4

Closed-Nesting-4Open-Nesting-8

Closed-Nesting-8

(c) Skip List 20% read

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 10 20 30 40 50

Number of Nodes

Flat-NestingOpen-Nesting-Reads20

Closed-Nesting-Reads20Open-Nesting-Reads50

Closed-Nesting-Reads50Open-Nesting-Reads80

Closed-Nesting-Reads80

(d) Skip List with 3 inner-txs

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 10 20 30 40 50

Thr

ough

put r

elat

ive

to fl

at n

estin

g

Number of Nodes

Flat-NestingOpen-Nesting-2

Closed-Nesting-2Open-Nesting-3

Closed-Nesting-3Open-Nesting-4

Closed-Nesting-4Open-Nesting-8

Closed-Nesting-8

(e) BST 20% read

0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7

3 3.3 3.6 3.9

0 10 20 30 40 50

Number of Nodes

Flat-NestingOpen-Nesting-Reads20

Closed-Nesting-Reads20Open-Nesting-Reads50

Closed-Nesting-Reads50Open-Nesting-Reads80

Closed-Nesting-Reads80

(f) BST 3 with inner-txs

Fig. 8. HyflowCPP performance using open-nesting transaction model.

Figure 8(a) shows open and closed nesting’s throughputrelative to flat nesting configuring the shared hash table with300 buckets. Initially, at small node count, contention is low,and flat and open nesting have similar performance. How-ever, as the node count and inner-transaction count begin toincrease, enough partial aborts occur, degrading flat nesting’sperformance, while open nesting’s performance remains steadydue to reduced false conflicts. At very high contention, with

225

8 inner-transactions, open nesting’s performance drops dueto increased abstract lock contention. Figure 8(b) shows theperformance with 3 inner-transactions per parent-transaction.Open nesting’s throughput increases with increase in readratio due to decreased contention. We recall that the cost forcommitting an open-nested transaction is comparable to thecost for committing a parent-transaction, therefore decreasingthe data contention allows the framework to successfullycommit more inner-transactions, and maximize the usefulnessof adopting the open nesting model.

In Figures 8(c) and 8(d) we present the results of SkipList benchmark configured with 100 share objects. We canobserve that as we increase the number of inner-transactions,open nesting performance decreases. It can be explained basedon read-set object caching in a scenario characterized byhigh contention. For flat nesting and closed nesting objectsaccessed by previous inner-transaction are cached in read-set. For higher inner-transaction count, almost all objects arelocally cached after the first few inner-transactions executed.Meanwhile, open-nested transactions are required to fetch theobjects accessed over network, without exploiting the read-set of previous committed open-nested transactions, whichincreases the execution time significantly. It is also worth men-tioning that open nesting has additional messaging overheadfor maintaining distributed abstract-locks.

BST results confirm the same trend of Skip List and arepresented in Figure 8(e) and 8(f).

VI. CONCLUSIONS

HyflowCPP provides a generic programming interface-based DTM abstraction for C++, with pluggable support forvarious DTM policies, and support for closed nesting, opennesting, and checkpointing. Our experimental studies revealedthat, HyflowCPP’s throughput scales robustly with increase inread/write ratio, and nodes, and that it outperforms its nearestJAVA-based competitors by up to 6x.

HyflowCPP is freely available as open-source software athttp://www.hyflow.org/hyflow/wiki/HyflowCPP.

ACKNOWLEDGEMENTS

We thank anonymous reviewers for their useful comments andsuggestions.This work is supported in part by US National ScienceFoundation under grants CNS 0915895, CNS 1116190, CNS1130180, and CNS 1217385.

REFERENCES

[1] T. Harris, J. Larus, and R. Rajwar, “Transactional memory,” SynthesisLectures on Computer Architecture, vol. 5, 2010.

[2] R. Palmieri, F. Quaglia, P. Romano, and N. Carvalho, “Evaluatingdatabase-oriented replication schemes in software transactional memorysystems,” in Proc. of DPDNS, 2010.

[3] M. M. Saad and B. Ravindran, “Hyflow: a high performance distributedsoftware transactional memory framework,” in HPDC, 2011.

[4] S. Peluso, P. Romano, and F. Quaglia, “Score: A scalable one-copyserializable partial replication protocol,” in Middleware, 2012.

[5] S. Peluso, P. Ruivo, P. Romano, F. Quaglia, and L. Rodrigues, “Whenscalability meets consistency: Genuine multiversion update-serializablepartial data replication,” in ICDCS, 2012.

[6] R. L. Bocchino, V. S. Adve, and B. L. Chamberlain, “Softwaretransactional memory for large scale clusters,” in PPoPP, 2008.

[7] M. Couceiro, P. Romano, N. Carvalho, and L. Rodrigues, “D2stm: De-pendable distributed software transactional memory,” in PRDC, 2009.

[8] C. Kotselidis, M. Ansari, K. Jarvis, M. Lujan, C. Kirkham, andI. Watson, “DiSTM: A software transactional memory framework forclusters,” in ICPP, 2008.

[9] N. Carvalho, P. Romano, and L. Rodrigues, “A generic framework forreplicated software transactional memories,” in NCA, 2011.

[10] A. Turcu and B. Ravindran, “Hyflow2: A high performance distributedtransactional memory framework in Scala,” Virginia Tech, Tech. Rep.,2012, http://hyflow. org/hyflow/chrome/site/pub/hyflow2-tech. pdf.

[11] K. Manassiev, M. Mihailescu, and C. Amza, “Exploiting distributedversion concurrency in a transactional memory cluster,” in PPoPP,2006.

[12] M. K. Aguilera, A. Merchant, M. Shah, A. Veitch, and C. Karamanolis,“Sinfonia: a new paradigm for building scalable distributed systems,”in SOSP, 2007.

[13] R. Hundt, “Loop recognition in C++/Java/Go/Scala,” Scala Days, 2011.

[14] M. M. Saad and B. Ravindran, “Transactional forwarding: Supportinghighly-concurrent stm in asynchronous distributed systems,” in SBAC-PAD, 2012.

[15] M. Herlihy and Y. Sun, “Distributed transactional memory for metric-space networks,” ser. DISC, 2005.

[16] A. Turcu and B. Ravindran, “Hyflow2: A high perfor-mance distributed transactional memory framework in scala,”Virginia Tech, Tech. Rep., April 2012. [Online]. Available:http://hyflow.org/hyflow/chrome/site/pub /hyflow2-tech.pdf

[17] A. Turcu, B. Ravindran, and M. M. Saad, “On closed nesting indistributed transactional memory,” in TRANSACT, 2012.

[18] A. Turcu and B. Ravindran, “On open nesting in distributed transac-tional memory,” in SYSTOR, 2012.

[19] E. Koskinen and M. Herlihy, “Checkpoints and continuations insteadof nested transactions,” in SPAA, 2008.

[20] A. Dhoke, B. Ravindran, and B. Zhang, “On closed nesting andcheckpointing in fault-tolerant distributed transactional memory,” inIPDPS, 2013.

[21] GNU C Library, “Complete context control,” http ://www.gnu.org/software/libc/manual/htmlnode/System −V − contexts.html.

[22] J. E. B. Moss and A. L. Hosking, “Nested tm: Model and architecturesketches,” Sci Comp Prog, vol. 63, 2006.

[23] J. E. B. Moss, “Open nested transactions: Semantics and support(poster),” in Workshop on Mem Perf Issues, 2006.

[24] R. Koo and S. Toueg, “Checkpointing and rollback-recovery for dis-tributed systems,” Software Engineering, IEEE Transactions on, 1987.

[25] E. Koskinen and M. Herlihy, “Checkpoints and continuations insteadof nested transactions,” in SPAA, 2008.

[26] C. Flanagan, A. Sabry, B. Duba, and M. Felleisen, “The essence ofcompiling with continuations,” in ACM SIGPLAN Notices. ACM, 1993.

[27] B. Karlsson, Beyond the C++ Standard Library: An Introduction toBoost. Addison-Wesley Professional, 2005.

[28] T. Willhalm and N. Popovici, “Putting Intel R© threading building blocksto work,” in Workshop on Multicore software engineering, 2008.

[29] MsgConnect, “Eldos corporation,” 2012,http://www.eldos.com/msgconnect/.

[30] P. Hintjens, “ØMQ-The Guide,” 2011, http://zguide.zeromq.org.

[31] A. Bieniusa and T. Fuhrmann, “Consistency in hindsight: A fullydecentralized STM algorithm,” in IPDPS, 2010.

[32] M. M. Saad and B. Ravindran, “Transactional forwarding algorithm,”Virginia Tech, Tech. Rep., January 2012.

[33] C. Cao Minh, J. Chung et al., “STAMP: Stanford transactional appli-cations for multi-processing,” in IISWC, September 2008.

[34] S. Mishra, “HyflowCPP: A distributed transactional memory frameworkfor C++,” Master’s thesis, Virginia Tech, Blacksburg, VA, USA, 2013,http://hyflow.org/hyflow/chrome/site/pub/Mishra-Thesis.pdf.

226