An Automatic Actors to Threads MappingTechnique for JVM-based Actor Frameworks

Ganesha UpadhyayaIowa State University, USAganeshau@iastate.edu

Hridesh RajanIowa State University, USA

hridesh@iastate.edu

AbstractActor frameworks running on Java Virtual Machine (JVM)platform face two main challenges in utilizing multi-core ar-chitectures, i) efficiently mapping actors to JVM threads, andii) scheduling JVM threads on multi-core. JVM-based actorframeworks allow fine tuning of actors to threads mapping,however scheduling of threads on multi-core is left to theOS scheduler. Hence, efficiently mapping actors to threadsis critical for achieving good performance and scalability. Inthe existing JVM-based actor frameworks, programmers se-lect default actors to threads mappings and iteratively finetune the mappings until the desired performance is achie-ved. This process is tedious and time consuming when build-ing large scale distributed applications. We propose a tech-nique that automatically maps actors to JVM threads. Ourtechnique is based on a set of heuristics with the goal of bal-ancing actors computations across JVM threads and reduc-ing communication overheads. We explain our technique inthe context of the Panini programming language, which pro-vides capsules as an actor-like abstraction for concurrency,but also explore its applicability to other approaches.

Categories and Subject Descriptors D.1.3 [ProgrammingTechniques]: Concurrent Programming ; D.2.7 [Distribu-tion, Maintenance, and Enhancement]: Portability ; D.3.3[Programming Languages]: Language Constructs and Fea-tures; D.3.4 [Processors]: Code generation, Optimization

General Terms Experimentation, Measurement, Perfor-mance, Scalability

Keywords Actors, Panini programming language, Capsule-oriented Programming, Implicit Concurrency, Multi-core,Java, JVM, Thread Mapping, CPU Utilization

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.AGERE! 2014, October 20, 2014, Portland, OR, USA..Copyright c© 2014 ACM 978-1-4503-2189-1/14/10. . . $15.00.http://dx.doi.org/10.1145/2687357.2687367

1. IntroductionMulticore and many-core systems have become the indus-try standards. The number of cores on these systems israpidly increasing. Sequential programs are ill-suited to runon multicore systems as they do not utilize the availablecores. Many concurrent and parallel programming modelshave emerged to utilize cores effectively. One such model isthe Actor model [8] which has self contained concurrentlyrunnable actors that communicate via message passing. Ac-tor model exposes parallelism by design. The parallelism inactor systems stems from being able to execute multiple ac-tors in parallel. The popularity of actor model has influencedprogrammers to build large distributed applications that per-form data, task and pipeline parallelism at fine and coarsegranularity.

However, for utilizing the underlying multicore, actorsneed to be mapped to cores carefully. Mapping is a twostep process: 1) actors threads mapping and 2) threads cores mapping (or scheduling). Often, the actor languageruntime handles both steps by creating required threads andscheduling them on different cores using an efficient actor tocore mapping technique. However, in JVM-based actor lan-guages, actors to threads mapping is performed by program-mers and JVM leaves scheduling of threads on multicoreto the OS scheduler. Programmers select default actors tothreads mappings and iteratively fine tune the mappings un-til the desired performance is achieved. This process is easyfor simple or embarrassingly parallel applications, howeverfor applications that have sub-linear performance, improvingthe application performance is tedious and time consuming[21].

It can be argued that an optimal solution for actorsto threads mapping using brute-force technique cannot befound in polynomial time (similar to the task mapping prob-lem [21]). For this purpose, we propose a novel heuristic-based approach that automatically maps actors to threads atcompile time. Our heuristics use actor characteristics suchas lifetime of the actor, amount of computation, nature ofcomputation, and nature of interaction with other actors inthe system, etc. The central goal of the heuristics is to mapactors to threads in a way that balances actor computational

29

workloads and reduces communication overheads. The out-come of applying the proposed heuristics in a systematicmanner results in one of the four different execution poli-cies (Thread, Task, Sequential, Monitor) for actors. The ex-ecution policy defines how the messages are processed. Inthread execution policy messages are processed by a ded-icated JVM thread. In task execution policy messages areprocessed by a shared thread and in sequential/monitor casethe actor that sends message itself processes the message atthe receiver actor. By means of assigning different execu-tion policies to actors at compile-time we achieve actors tothreads mappings.

We propose that programmers use our heuristic-basedmapping technique for initially mapping actors to threads inplace of default mapping technique. We believe that by usingour heuristic-based mapping technique programmers timeand efforts to arrive at initial optimal mapping are saved.

We have realized our technique in the Panini program-ming language [1, 20]. Panini programming language pro-vides an abstraction for implicit concurrency called cap-sules, which are a flavor of actors for sequentially trainedprogrammers1. Even though, our technique is well-suited forPanini’s capsules where programmers can concentrate theirefforts in application development rather than understandingand improving concurrent execution, it is applicable to anyJVM-based actor frameworks. Our evaluation over a widerange of actor programs shows that our heuristic-based map-ping achieves on average 50% improvement in program run-time over default-thread and default-task mappings.

2. Related WorkActor frameworks such as Akka [15], Kilim [25], Scala Ac-tors [16], Jetlang [5], ActorFoundry [10], SALSA [12] andActors Guild [2] allow programmers to map their actors toJVM threads and fine tune their application using schedulersand dispatchers. In Akka, dispatchers are responsible foractor scheduling. Akka provides four kinds of dispatchers:default, pinned, balancing, and calling thread. The defaultdispatcher is an event-based single queue implementationbacked by a thread-pool of configurable size. In pinned dis-patcher each actor has it own OS thread. The balancing dis-patcher employs event-based approach and uses single mail-box and performs load-balancing. Finally, the calling threaddispatcher has no thread associated with it and the thread ofthe message sender executes the message. Kilim is an actorframework for Java. It provides lightweight event-based ac-tors. Kilim scheduler is a bundle composed of a thread-pool,a scheduling policy and collection of runnable actors. By de-fault, actors in the collection are scheduled in a round-robinfashion. Scala Actors allow creation of thread-based andevent-based actors. Thread-based actors are assigned dedi-cated JVM threads, whereas event-based actors share JVM

1 Main promise of capsules is that they can enable modular reasoning aboutconcurrent programs, but that direction is explored elsewhere [11].

threads associated with the task-pool that processes event-based actor messages. For the purpose of fair scheduling,task-pool uses round-robin scheduling. ActorFoundry inter-nally uses kilim and Actors Guild follows similar strategy asScala Actors. SALSA allows creation of heavy-weight andlight-weight actors using Stage. Stage is a bundle composedof a single message queue and a JVM thread. Heavy-weightactors are assigned individual stages and light-weight actorsshare stages. However, mapping actors to stages is left to theapplication developers.

In all these actor languages and frameworks, applica-tion developers define actors to threads mappings and finetune the mapping until the desired performance is achieved.Whereas, our technique performs automatic mapping of ac-tors to threads.

Several works on performance improvement of non-JVMactor frameworks exists. Francesquini et al.[13] proposed atechnique implemented in Erlang [27] runtime that placesErlang actors on multi-core efficiently. Their techniqueshowed that by placing frequently communicating actors(hub-and-affinity) together, over two times improvement inthe application performance can be achieved. However, pro-grammers need to identify hub and its affinity actors and an-notate the program for runtime to perform the desired map-ping. Our technique uses many more actor characteristicsalong with hub-and-affinity and performs essential programanalyses to automatically determine the mappings.

Mapping application on to multi-core is well studiedproblem. The application is represented as task graph and themapping problem is defined as how to map different tasksto CPU cores to minimize application runtime. A recent sur-vey by Singh et al. [22] lists different static, dynamic andhybrid techniques that map task graph to multi-core withperformance, energy consumption and temperature as dif-ferent goals of determining optimal mapping. Researchershave explored the problem of mapping application tasks thatcommunicate via both message passing and shared mem-ory on homogeneous and heterogeneous cores [24]. Thesetechniques are not directly applicable to JVM-based actorframeworks, because threads to cores mapping is left to OSscheduler and only actors to threads mapping can be opti-mized. However, actors to threads mapping technique canutilize solutions proposed for general task graph mappingproblem. In our heuristic-based mapping technique, we uti-lize actor characteristics and interaction behaviors similarto task characteristics and task graph in general task graphmapping problem.

Note that the mapping problem in actor programs is dif-ferent from the mapping problem in general multi-threadedprograms. In multi-threaded programs, the mapping problemis defined as scheduling and load-balancing of threads onmulti-cores. This also involves binding of threads to phys-ical cores. However in actor programs, the mapping prob-lem is two-fold: mapping actors to threads and scheduling

30

of threads on multi-cores. Tousimojarad et al. [26]’s workproposes efficient strategies for mapping threads to cores forOpenMP multi-threaded programs. When compared to thiswork, our technique maps actors to threads and not threadsto cores. Threads to cores mapping is handled by OS sched-uler in JVM-based actor frameworks.

3. Actors to JVM Threads Mapping ProblemTraditionally actors tend to have a dedicated thread that pro-cesses messages in their queue. As the popularity of actorprogramming paradigm grows, actor languages are used todevelop large scale distributed applications. These applica-tions spawn a large number of actors and hence dedicating athread for every actor is not scalable. Actor languages haverealized this limitation and allowed programmer to share athread across multiple actors. However, it is the responsibil-ity of the programmer to correctly map actors to threads.

Figure 1. BenchErl serialmsg actor communication graph.

Figure 2. Panini histogram actor communication graph.

Figure 3. Performance of default-thread and default-taskmappings for BenchErl serialmsg and Panini histogrambenchmark programs (x-axis: 2, 4, 8, and 12-core settingsand y-axis: runtime in seconds).

To illustrate the problem consider mapping actors tothreads for two actor programs shown in Figure 1 and Fig-ure 2. Our first program shown in Figure 1 is the serialmsg

program from BenchErl benchmark suite [9]. This bench-mark is about message proxying through a dispatcher andspawns a certain number of receivers, one dispatcher, and acertain number of generators. The dispatcher forwards themessages that it receives from generators to the appropri-ate receiver. Each generator sends a number of messages toa specific receiver. The second program shown in Figure 2is the histogram program from Panini [1, 20]. The goal ofthis problem is to count the number of times each ASCIIcharacter occurs on a page of text.

For deciding initial actors to threads mapping program-mers can use default-thread or default-task mappings. Indefault-thread mapping, every actor instance is assigned adedicated thread and in default-task mapping, actors aremapped to a taskpool containing fixed number of threads(usually equals to number of CPU cores). Figure 3 showsperformance of two benchmark programs on 2, 4, 8, and12-core machines. The two main concerns are visible in theperformance results. These are i) performance is not consis-tent across programs indicating that a single default mappingstrategy (thread or task) does not work across programs, andii) performance degrades upon adding more cores.

Now, the programmer can use default-thread or default-task mappings, profile the default mappings and iterativelyfine tune the actors to threads mappings. However, whenmultiple actors are mapped to a JVM thread, profiling thethread and understanding the bottlenecks is difficult. We be-lieve that some aspects of the actor model can be directlyused to decide the initial actors to threads mappings. For in-stance, in the illustrative program shown in Figure 1, gener-ators and receivers can be assigned to single thread to avoidmessage passing overheads. For addressing the scalabilityproblem, choosing the right number of JVM threads plays animportant role. The solution of limiting number of threadsworks for applications that have smaller number of actors.The challenge remains for applications with large number ofactors. Hence the fundamental problem is finding an initialoptimal mapping that has consistent performance and scaleswell when additional cores are allocated.

3.1 Problem FormulationWe formulate the problem of mapping actors to JVM threadsas selecting different execution policies for actors in thesystem. Execution policy defines how actor messages areprocessed. We define four different execution policies asfollows,

• THREAD, in this execution policy, a dedicated thread isassigned for processing the messages from the actor’smessage queue and executing the corresponding behavior(works similar to akka’s pinned dispatcher).

• TASK, in this execution policy, the actor messages areprocessed by the shared thread of the taskpool. Thetaskpool may contain one or more actors that abide toTASK execution policy. The order in which the messages

31

from different actors message queue has to be processedcould vary. One simple policy is to process one messagefrom each actor to avoid starvation of other actors.

• SEQ/MONITOR, in case of SEQ and MONITOR, thepolicy is that the calling actor that sends the messageneeds to execute the defined behavior at the callee actorthat received the message (works similar to akka’s callingthread dispatcher).

Actor communication graph (ACG). Given an actorprogram, actor communication graph defines various actorsin the program and the interaction between them. ACG is adirected graph G(V,E) where,

• V = A0,A1, ...,An is a set of nodes, each representing anactor.

• E is a set of edges (Ai,A j) for all i,j such that there iscommunication from Ai to A j.

Mapping function. Given actor program and ACG wedefine actors to threads mapping as assigning execution poli-cies from the set of possible execution policies, M(P× ACG)7→ EP where,

• P is actor program,• ACG is actor communication graph and,• EP = {THREAD, TASK, SEQ, MONITOR}.

4. ApproachIn this section we describe our heuristic-based mappingtechnique that assigns execution policies to actors. We firstdescribe different aspects of actor programs that are relevantfor our heuristics and formulate them as a Actor Character-istics Vector (cVector). We then describe a set of heuristicsthat given cVector predicts the execution policy. Finally, wedescribe our mapping function followed by a set of examplesillustrating the application of our mapping technique.

4.1 Analyzing ApplicationsSome aspects of the actor applications can be directly usedto decide the execution policies for actors. The knowledgeof the actor behavior and their communication graph thatdefines the relationship between the actors can be easilyextracted. This subsection presents several of these aspects.

• blocking, an actor has blocking behavior if it has exter-nally blocking behaviors using I/O, socket or databaseblocking primitives.

• stateful/stateless, actors may have state variables that aremodified in multiple actor behaviors.

• inherent parallelism, actors may use blocking send prim-itives and receive results or use asynchronous send prim-itives. Actors may or may not require the results immedi-ately.

• computationally intensive, actors may have differentcomputational requirements.

• communication behavior (or message rate), actor systemmay contain leaf actors that do not send messages to otheractors, or routing actors that sends exactly one messagefor every message it receives, or broadcast actors thatsends multiple messages for every message it receives.

• hub-affinity, some actors communicates more with a setof actors than other actors in the system. These actorsform hub-affinity group.

• data rate, some actors in the system may send/receivehigh volume of data.

• contention, some actors may have a tendency to receivemessages from multiple competing actors.

4.2 cVector: Actor Characteristics VectorBased on the different aspect about actor behaviors and theircommunication graph every actor is assigned a Characteris-tics vector. The Characteristics vector has five fields,

〈〈 BLK, STATE, PAR, COMM, CPU 〉〉

1. BLK = {true, false} represents blocking behavior,

2. STATE = {true, false} represents stateful/stateless behav-ior,

3. PAR = {low, med, high} represents inherent parallelismand values are assigned as follows,• low, if actor sends a synchronous message and waits

for the result or consumes the result right away,• high, if actors sends an asynchronous message and

does not require result,• med, otherwise

4. COMM = {low, med, high} represents communicationbehavior and it is determined as follows,• low, does not send messages to other actors,• high, sends more than one message to the connected

actors,• med, sends exactly one message to the connected ac-

tor,

5. CPU = {low, high}, represents computational workloadof the actor and it is determined as follows,• high, when recursive, loops with unknown bounds,

makes high cost library calls,• low, otherwise

Actor characteristics vector is computed using actor pro-gram (P) and actor communication graph (ACG) by perform-ing a number of program analyses. For determining block-ing, we look for usage of external blocking primitives. Weperform intra-procedural analysis of actor behavior defini-tions to determine stateful/stateless behavior, inherent paral-

32

Figure 4. Flow diagram of our mapping function that assigns actors one of the four execution policies.

lelism (PAR), communication behavior (COMM) and com-putational requirement (CPU). We mark an actor stateful ifmore than one actor behavior accesses any of its state vari-able. Both PAR and COMM requires analysis of statementsthat involves sending messages. Note that in determiningvarious fields of cVector the program analyses requires ac-tor definition (actor code) and the communication behaviorsof outgoing messages rather than incoming messages. Thisrequirement helps our analyses to be less-strict about avail-ability of complete ACG at compile time.

4.3 Mapping HeuristicsEarlier we presented four different execution policies thatcan be assigned to actors. This section examines a number ofheuristics for predicting the execution policy. The heuristicsmake use of the actor characteristics vector cVector.

Blocking Actors Heuristics. This heuristic states thatany actor that has external blocking behavior, as representedby BLK field in cVector, should be assigned thread (Th)execution policy. Any other execution policy for blockingactors would lead to blocking of the executing thread andmay lead to actor starvation and deadlocks. The cVector forsuch actors is <true,_,_,_,_>. If BLK is true, other fields ofcVector are ignored in making the execution policy decision.

Heavy Actors Heuristics. This heuristic states that anyactor that is non-blocking with high inherent parallelism,high communication and high computation should be as-signed thread execution policy. The cVector of such an actoris <false,_,high,high,high>. The rationale behind this decision isthat the assigned thread can perform its CPU intensive com-putations in parallel with other threads without voluntarilyinterruption.

HighCPU Actors Heuristics. Actors that have high in-herent parallelism with high CPU needs but low communica-tion frequency are assigned task (Ta) execution policy. Theseactors have cVector <false,_,high,low,high>. By assigning taskexecution policy, these actors can utilize any load-balancingstrategies applied to the task-pool.

LowCPU Actors Heuristics. LowCPU actors that havecharacteristics vector cVector <false,_,high,low,low> should beassigned monitor (M) execution policy. These actors do notneed special attention and hence are processed by the callingactor (actor that sends messages).

Hub Actors Heuristics. This heuristic states that hub ac-tors should be assigned task (Ta) execution policy. Hub ac-tors are represented using cVector either <false,_,high,high,low>

or <false,_,low/med,high,high>. The rationale behind this decisionis that affinity actors (actors that hub actor communicates of-ten) can be executed by the shared thread that is executingthe hub actor task in the task-pool.

Affinity Actors Heuristics. This heuristic states thataffinity actors should be assigned monitor (M) executionpolicy. Affinity actors have following cVector <false,_,low/med

,low/med,low>. By assigning monitor execution policy, the hubactor (of these affinity actors) is forced to execute the affinityactors.

Master Actors Heuristics. This heuristic states thatmaster actors should be assigned thread (Th) execution pol-icy. The cVector of master actors is <false,_,low/med,high,low>.Master actors have the property that they delegate the workto slave actors and often wait for the result. Hence, by as-signing a dedicated thread it does not block the execution ofslave actors.

33

Slave Actors Heuristics. This heuristic states that slaveactors should be assigned task (Ta) execution policy. ThecVector of slave actors is <false,_,low/med,low/med,high>. Simi-lar to HighCPU actors these actors can utilize any load-balancing strategies applied to the task-pool.

4.4 Mapping functionFigure 4 describes the flow of our mapping function. Forevery actor in the system, mapping function assigns one ofthe four execution policies. By following the flow it is easyto see that the strategy is complete. Because, every actor witha cVector is assigned an execution policy. It can also be seenthat actors are never assigned multiple execution policies.

4.5 ExamplesWe have implemented our technique in Panini Capsules (anactor flavor), hence we first briefly describe Panini Cap-sules. We then present several example panini programs anddemonstrate the application of our mapping technique to de-cide execution policies. The panini source code of these pro-grams is available in [6].

4.5.1 Panini CapsulesCapsules are an actor-like abstraction implemented in theprogramming language Panini [1, 11, 20]. Figure 5 presentsan example HelloWorld program in this language. In thisprogram there are three actors (capsules) HelloWorld, Greeterand Console and they are connected as HelloWorld→ Greeter→Console.

1 signature Stream { //A signature declaration2 void write(String s) ;3 }

5 capsule Console () implements Stream { //Capsule declaration6 void write(String s) { // Capsule procedure7 System.out.println(s) ;8 }9 }

11 capsule Greeter (Stream s) { //Requires an instance of Stream to work12 String message = "Hello World!"; // State declaration13 void greet() { // Capsule procedure14 s.write ("Panini: " + message); // Inter−capsule procedure call15 long time = System.currentTimeMillis();16 s.write ("Time is now: " + time);17 }18 }

20 capsule HelloWorld() {21 design { // Design declaration22 Console c; // Capsule instance declaration23 Greeter g; // Another capsule instance declaration24 g(c) ; // Wiring, connecting capsule instance g to c25 }26 void run() { // An autonomous procedure27 g.greet() ; // Inter−capsule procedure call28 }29 }

Figure 5. HelloWorld Program in Panini

In Panini’s programming model, capsules are indepen-dently acting entities. Capsules provide interface to com-municate to other capsules via capsule procedures. When a

capsule wants to communicate with other capsule it does sousing inter-capsule procedure calls. In the HelloWorld pro-gram described above, g.greet() in line 27 is an inter-capsuleprocedure call between HelloWorld capsule and Greeter cap-sule. If a capsule requires return result of inter-capsular callthen the caller receives a future as a proxy for the actualreturn value (void return values are allowed). If the valueis not used immediately, the caller can continue execution.Inter-capsule procedure calls are processed using messagepassing mechanism similar to actors. There are two kinds ofcapsules: those that define autonomous behavior by declar-ing a run procedure, and those that respond to request fromother capsules.

4.5.2 BenchErl BangBang benchmark simulates many-to-one message passing.It spa-wns one receiver and multiple senders that flood thereceiver with messages. There are 440 instances of Sender andone instance of Receiver. Each Sender sends 440 messages toReceiver. The actor communication graph (ACG) is shown inFigure 6.

Figure 6. ACG of BenchErl Bang program.

Sender capsule has the characteristics vector (cv): <false,_,

high,high,low>, because it is non-blocking, it has high inherentparallelism as inter-capsular call does not expect the results,it highly communicates with Receiver capsule and it does nothave CPU intensive computation in it. Hence, by applyingour mapping function shown in Figure 4, we determine thatSender capsule instances should be assigned Task executionpolicy. The cV for Receiver is <false,_,low,low,low> and by fol-lowing the mapping strategy, it is assigned Monitor execu-tion policy. A general intuition is to assign Task executionpolicy to Receiver, because it processes large number of mes-sages from Sender capsules. However, if assigned Task ex-ecution policy, the thread processing Receiver task will en-counter large overhead due to excessive communication andlow computation behavior of Receiver. By assigning Monitorexecution policy to Receiver this overhead can be reduced.

4.5.3 FileSearch Program from Actor CollectionsFileSearch program performs document indexing and search-ing. The different capsules in this program are: FileCrawler

, FileScanner, Indexer and Searcher. FileCrawler recursively visitseach sub-directory in the input file path and sends a messageto FileScanner whenever it finds a file. FileScanner processes

34

the file sent by the FileCrawlerand asks next available Indexer

from the list of indexers to index the file. Indexer performshash-based indexing and stores the result. Upon visiting ev-ery file in the directories/sub-directories of the input pathFileCrawler notifies FileScanner and FileScanner notifies Searcher.Searcher takes search string from the command-line and re-quests each Indexer to look for the search string in their storedresults and return the file path if found.

Capsules cVector and the execution policies are shownin the table below. The ACG for this program is shown inFigure 7.

Capsule cVector PolicyFileCrawler <false,_,high,high,high> ThreadFileScanner <false,_,high,high,low> TaskIndexer <false,_,low,low,low> MonitorSearcher <true,_,_,_,_> Thread

Figure 7. Actor Collection FileSearch

FileCrawler has cVector <false,_,high,high,high> because it per-forms heavy recursive task, hence it is assigned Thread ex-ecution policy. FileScanner communicates highly with a set ofindexers and acts as a delegator. The cVector of FileScanner

is <false,_,high,high,low> and it is assigned Task execution pol-icy. There are eleven Indexer capsule instances with cVector<false,_,low,low,low>. These are leaf actors with low computa-tions, hence are assigned Monitor policy. Searcher blocks un-til FileScanner notifies to begin searching. Since it is a block-ing capsule, we assigned Thread execution policy to it. It isimportant that FileCrawler is assigned Thread execution pol-icy, because it requires a dedicated thread to perform un-interrupted processing. Assigning Thread execution policyto blocking Searcher capsule ensures that it does not leads tothe starvation of other actors. Our decision to assign Taskexecution policy to FileScanner is critical to further improvethe performance of the program. FileScanner in this programcan be a performance bottleneck, because it receives largenumber of requests from FileCrawler and it should delegate thework to Indexer capsules without delaying. Also FileScanner is astateless capsule and its requests can be simultaneously pro-cessed by multiple threads.

4.5.4 BenchErl SerialmsgBenchErl serialmsg is about message proxying through adispatcher. The benchmark spawns a certain number of re-ceivers, one dispatcher, and a certain number of genera-tors. The dispatcher forwards the messages that it receivesfrom generators to the appropriate receiver. Each generatorsends a number of messages to a specific receiver. This pro-gram has 120 instances of Generator capsule, 120 instancesof Receiver capsule and one instance of Dispatcher capsule. Thetable below lists cVector and assigned execution policy forvarious capsules in this program. The ACG for this programis shown in Figure 8.

Capsule cVector PolicyGenerator <false,_,high,high,low> TaskDispatcher <false,_,low,low,low> MonitorReceiver <false,_,low,low,low> Monitor

Figure 8. BenchErl Serialmsg

The communication between each Generator with its Receiver

is high. Hence, for tightly binding them we assign Mon-itor execution policy to Dispatcher and Receiver so that thethread that is processing Generator will also process Receiver

. If Dispatcher is assigned Task execution policy, it leads to theperformance bottleneck, because it receiver large number ofmessages from Generator capsules and it should immediatelyroute the messages to appropriate receivers.

5. Evaluation5.1 Benchmark selectionWe have selected actor programs that have data, task, andpipeline parallelism in them at fine and coarse granularitylevels. These applications show super-linear, linear and sub-linear speedups and they may or may not scale well whenadditional cores are allocated to them. Our idea is to eval-uate a wide range of actor programs rather than be repet-itive. We have selected representative programs from Er-lang BenchErl [9] suite, Computer Language BenchmarksGame [3], Actor Collections [4], StreamIt Benchmarks [7],JavaGrande [23] and Panini Examples [1, 20]. While select-ing the representative programs from different benchmark

35

B Capsule cVector Policy

bang Sender <false,_,high,high,low> TaskReceiver <false,_,low,low,low> Monitor

ehbGroup <false,_,high,high,low> TaskSender <false,_,high,high,low> TaskReceiver <false,_,low,low,low> Monitor

mbrot Worker <false,_,high,high,low> TaskMandel <false,_,low,low,low> Monitor

serialmsgGenerator <false,_,high,high,low> TaskDispatcher <false,_,high,low,low> MonitorReceiver <false,_,low,low,low> Monitor

Figure 9. Details of BenchErl benchmark programs.

suites, we have included programs that consists of differ-ent types of actors and their interactions are not straightfor-ward. We have translated a total of fourteen different actorprograms to Panini for evaluation. Our rewriting translatedonly concurrency-related code and structure from the origi-nal programs and did not alter or optimize code not related toconcurrency. We now list the details about each benchmarkprogram such as capsules, cVectors and execution policiesthat are assigned to capsules by our mapping technique. Thedetails about the mappings and its impact on the programperformance will be discussed in §5.3.

5.1.1 BenchErl BenchmarksBenchErl is a publicly available scalability benchmark suitefor applications written originally in Erlang. Unlike otherbenchmark suites, which are usually designed to report aparticular performance point, BenchErl aims to assess scala-bility, i.e., a set of performance points that show how an ap-plication’s performance changes when additional resources(e.g. CPU cores, schedulers, etc.) are added. We have se-lected bang, ehb, mbrot and serialmsg programs for evalu-ation. The description of actors (capsules) and the assignedexecution policies is shown in Figure 9.

5.1.2 Computer Language Benchmarks GameThis suite of programs is used to compare the perfor-mance of different languages and libraries. We have se-lected fannkuchredux, fasta and knucleotide programs. Thedescription of actors (capsules) and the assigned executionpolicies is shown in Figure 10.

5.1.3 Programs from Actor CollectionsThis suite lists a collection of Akka/Scala actor applica-tions in the Github repository. We have selected FileSearch,ScratchPad and PolynomialIntegral actor programs. The de-scription of actors (capsules) and the assigned executionpolicies is shown in Figure 11.

5.1.4 StreamIt ProgramsThe set of benchmarks are available with StreamIt softwareversion 2.1.1. Most benchmarks are from the signal process-

B Capsule cVector Policy

Fannkuchred fannkuchred <false,_,high,low,high> TaskCollector <false,_,low,low,low> Monitor

fasta

RandomFasta <false,_,low,high,high> TaskRepeatFasta <false,_,low,high,high> TaskFloatProbFreq <false,_,low,low,low> MonitorWriter <false,_,low,low,low> Monitor

KnucleotideSequenceGen <false,_,low,low,high> ThreadNucleotide <false,_,low,low,high> TaskCollector <false,_,low,low,low> Monitor

Figure 10. Details of CLBG benchmark programs.

B Capsule cVector Policy

FileSearch

FileCrawler <false,_,high,high,high> ThreadFileScanner <false,_,high,high,low> TaskIndexer <false,_,low,low,low> MonitorSearcher <true,_,_,_,_> Thread

ScratchPad

FilesystemWalker <false,_,high,high,high> ThreadLocAnalyser <false,_,high,high,low> TaskLocCounter <false,_,high,low,high> TaskAccumulator <false,_,high,low,low> MonitorResultAccumulator <false,_,low,low,low> Monitor

PolynomialDelegateActor <false,_,low,low,low> MonitorDispatcherActor <false,_,high,high,low> TaskComputerActor <false,_,low,low,low> Monitor

Figure 11. Details of Actor Collections benchmarks.

B Capsule cVector Policy

Bea

mFo

rmer

AnonFilter_a1 <false,_,high,low,low> MonitorAnonFilter_a0 <false,_,high,low,low> MonitorInputGenerate <false,_,high,low,high> TaskCoarseBeamFirFilter <false,_,high,low,high> TaskBeamFirFilter <false,_,high,low,high> TaskAnonFilter_a3 <false,_,high,high,low> Monitor

DC

T

FileReader <true,_,_,_,_> ThreadiDCT8x8_ieee <false,_,high,low,low> MonitoriDCT_2D_reference_fine <false,_,high,low,low> MonitorAnonFilter_a0 <false,_,high,low,low> MonitoriDCT_1D_Y_reference_fine <false,_,high,low,low> MonitorYSplitter <false,_,high,high,low> TaskiDCT_1D_reference_fine <false,_,high,low,high> TaskiDCT_1D_X_reference_fine <false,_,high,low,low> MonitorFileWriter <false,_,low,low,low> Monitor

Figure 12. Description of various capsules, their respectivecVectors and assigned execution policies for Streamit Beam-former and DCT benchmarks.

ing domain. We have selected BeamFormer and DCT as tworepresentative applications. The description of actors (cap-sules) and the assigned execution policies is shown in Fig-ure 12.

36

5.1.5 RayTracer from JavaGrandeThis benchmark measures the performance of a 3D ray-tracer. The description of various capsules, their respectivecVectors and assigned execution policies are shown in tablebelow.

Capsule cVector PolicyRunner <false,_,low,high,low> ThreadRayTracer <false,_,low,low,high> Task

5.1.6 Histogram from Panini ExamplesThis actor program implements classic histogram problem.The goal of this problem is to count the number of timeseach ASCII character occurs on a page of text.

Capsule cVector PolicyReader <true,_,_,_,_> ThreadBucket <false,_,high,low,low> MonitorPrinter <false,_,low,low,low> Monitor

5.2 MethodologyWe compare our heuristic-based mapping technique againstdefault thread and default task mapping techniques. Therationale behind comparing against default thread map-pings is that most actor languages and frameworks supportthreaded actors such that programmers can debug/profiletheir threaded actor program to fine tune the performance.The default task mappings are supported for event-basedprogramming of actors. We measure the runtime of pro-grams for three different mappings when steady-state perfor-mance is reached. Following the methodology of Georges etal.[14], steady-state performance is reached when the coef-ficient of variation of the most recent three iteration timesof a benchmark fall below 0.02. We compare the executiontime for each benchmark program for the three mappingstrategies. We also measure the performance of three dif-ferent mappings on 2, 4, 8, and 12- cores settings (Linuxtaskset utility is used for altering core settings on 12-coresystem). The experiments are conducted on 12-core system(2 Six-Core AMD Opteron R© 2431 Processors) with 24GBof memory running the Linux version 3.5.5 and Java version1.7.0_06. A Java VM size of 2GB is sufficient to run all ofour experiments.

5.3 Performance and ScalabilityFor comparing the performance of our heuristic-based map-ping against default-thread and default-task mappings, wedefine following two metrics:

• Ith is the improvement over default-thread mapping. It isthe percentage reduction in runtime over default-threadmappings, and

• Ita is the improvement over default-task mapping. It is thepercentage reduction in runtime over default-task map-pings.

We compute Ith and Ita for each program on 2, 4, 8, and12 core settings. Figure 13 shows the results. We also com-pute average Ith and Ita to determine overall improvementof heuristic-based mapping over default-thread and default-task mapping.

Results. Over fourteen evaluated benchmarks, averageIth is 51% and average Ita is 50%. Average Ith and averageIta on different core settings is shown in table below.

#cores Ith Ita2 49% 50%4 51% 50%8 51% 51%

12 53% 50%

Outliers. Mainly for three programs our heuristic-basedmapping technique achieved small or no improvements.These programs are, fannkuchredux (Ith: 3% & Ita: 1%),RayTracer (Ith: -5% & Ita: 10%), and dct (Ith: 3% & Ita:13%). On the other hand, our technique achieved large im-provements for five programs. These programs are, bencherlmbrot (Ith: 95% & Ita: 95%), bencherl serialmsg (Ith: 70%& Ita: 65%), polynomialintegral (Ith: 72% & Ita: 72%), fasta(Ith: 85% & Ita: 77%), and histogram (Ith: 91% & Ita: 99%).We will now discuss these outliers in detail along with otherinteresting results.

Analysis. For fourteen evaluated benchmarks on aver-age 50% improvement over default mappings suggests thatour heuristic-based mapping could be used as a better initialmapping strategy than default-thread and default-task map-pings. On average 50% improvement on various core set-tings indicates that our heuristic-based mapping is consistentacross machines with different resource (CPU cores) avail-abilities.

Our heuristic-based mapping technique achieved smallimprovements for three programs (RayTracer, fannkuchre-dux, and DCT). These programs are mainly data-parallelapplications with embarrassingly parallel behavior. The re-sults support our earlier intuition that for embarrassinglyparallel applications it is easy to determine actors to threadsmappings, as actors independently perform the tasks. For in-stance, in RayTracer program Runner acts as master that dis-tributes the work to a set of RayTracer worker actors. RayTracer

worker actors perform independent computations. For thisprogram, mapping is intuitive. Runner could be assignedThread execution policy and each RayTracer worker could beassigned Task execution policy. Hence, it is easy to map ac-tors to threads and there is very little opportunity for furtherimproving the initial mapping.

Our heuristic-based mapping technique achieved largeimprovements for five programs (mbrot, serialmsg, polyno-mialintegral, fasta, histogram). These programs have sub-

37

Figure 13. Results show Ith (improvement over default-thread mapping) and Ita (improvement over default-task mapping) forthe benchmark programs on 2, 4, 8 and 12-core settings.

linear performance benefits. In other words, in these pro-grams balancing computation over communication is a dif-ficult task. BenchErl-mbrot program is a Mandelbrot simu-lator. This program generates a set of pixels that correspondsto a 2-D image of a specific resolution. These pixels are dis-tributed to a set of workers (Worker) which communicateswith Madel capsule to check if the pixel belongs to Mandelset or not. By assigning Monitor execution policy to Madel

capsule large communication overhead between Worker cap-sules and Madel is reduced. Also, Madel capsule busy-waits un-til the Worker capsules requests Madel to perform Mandel setcheck. The assignment of Monitor execution policy to Madel

eliminates this busy-waits because the Worker itself performsMadel set check. In BenchErl-serialmsg program bindingGenerator capsules to Receiver capsules by assigning Monitorexecution policy to Dispatcher large communication overheadbetween Generator and Receiver is reduced.

In Polynomial there are 500 instances of ComputeActor

which performs small computations and do not require ded-icated thread. By assigning Monitor execution policy threadprocessing DispatchActor processes ComputeActor capsules. Infasta, the two instances of RandomFasta capsules commu-nicate highly with their respective FloatProbFreq instances.FloatProbFreq is leaf-capsule and has low computation. Byassigning Monitor execution policy we eliminate commu-nication overheads. Writer capsule busy-waits until completeset of sequences are generated by RandomFasta and RepeatFasta

capsules. By assigning Monitor execution policy communi-

cation overhead is reduced. In Histogram program, Reader

that has external I/O blocking is assigned Thread executionpolicy. Bucket and Printer perform very small computations.The communication between Reader and 128 instances ofBucket is very large. By assigning Monitor execution policy,Reader and Buckets are processed by same thread.

Details. So far we have discussed only outliers andtheir performance. We will now investigate the remainingbenchmark programs to see what mappings were crucial inproducing good performance. In bang, by assigning Moni-tor execution policy for Receiver, it is now processed by thethread that is processing the Sender. This reduced the com-munication overhead between Sender and Receiver and im-proved the performance of the program. Similarly in ehbprogram, by assigning Monitor execution policy to Receiver

we have reduced communication overhead between Sendersand Receivers in each group. In ScratchPad program run-ning FileSystemWalker and LocAnalyzer concurrently is impor-tant. FileSystemWalker performs high computations recursivelyand communicates the intermediate results to LocAnalyzer.LocAnalyzer delegates its work to a set of LocCounter instances.Both Accumulator that collects and processes the intermediateresults and ResultAccumulator that collects final set of resultsbusy-waits until all the requests from FileSystemWalker are pro-cessed by LocCounter instances. Hence, assigning Monitor ex-ecution policies to busy-wait capsules greatly reduces thecommunication overhead. knucleotide has SequenceGen thatreads input DNA sequence and uses blocking read opera-

38

Figure 14. Comparing scalability of our heuristic-based mapping against default-thread and default-task mapping (x-axis: 2,4, 8, and 12-core settings and y-axis: runtime in seconds).

tion. Hence it is assigned Thread execution policy. Thereare 46 instances of Nucleotide capsule. Nucleotide performs CPUintensive computations and sends the result to Collector cap-sule. Collector capsule busy-waits until the results from all 46Nucleotide capsules is received. By assigning Monitor execu-tion policy to Collector we eliminate this busy-wait that greatlyimproves the performance of this program.

In summary, for these programs a heuristic-based map-ping such as the one we proposed yields better perfor-mance over default mapping techniques. These programsalso shows the characteristics that are hard to fine tune forprogrammers to gain additional performance benefits.

Figure 14 evaluates the scalability of three mapping tech-niques. The individual charts show the effect of addingcores on program runtime. It can be seen that our heuristic-based mapping technique suffers less from scalability is-sues when compared to the default mappings. When addi-tional cores are allocated to the program, additional JVMthreads are available for the program to use. The defaultmappings utilize the additional threads without properly ad-justing the mappings. In case of highly parallel programs,additional threads balance the workload and helps to im-prove the overall performance of the program. Whereas, inother program additional threads reduces the computation-

to-communication ratio and adds more overhead. This be-havior is evident when we profile the default mappingson additional cores we see the number of context-switchesrapidly increases. However, in our heuristic-based mapping,when additional threads are available, threads are utilized bykeeping the critical mappings intact. For instance, we do notseparate hub actors from its affinity actors when additionalthreads are available for use. This helps us introduce lessoverheads. To further illustrate, consider fasta program. Thisprogram has two instances of RandomFasta, RepeatFasta, onetwo FloatProbFreq and one Writer. When additional threads areavailable, it is important that the mapping technique doesnot assign separate threads for RandomFasta and FloatProbFreq

capsules. This is not ensured in default mappings.Predictive Power of cVector. Our mapping function

shown in Figure 4 indicates that BLK is the most powerfulcVector field. If BLK is true, we ignore other fields in cVectorand assign Thread execution policy. In the remaining cases,it is mandatory that other fields in cVector should be checkedto assign appropriate execution policy. STATE field in cVec-tor is unused for deciding the execution policy. However, thisfield is used to further enhance the mapping as follows. Ifany actor is assigned Task execution policy and if STATE isfalse it indicates that multiple threads can process messages

39

in the actor’s message queue simultaneously. For instance,consider FileSearch program shown in Figure 7, FileScanner

capsule has STATE value false. By allocating more threadsto FileScanner task, overall performance of the program can befurther improved.

Handling Dynamism. Many actor languages and frame-works allow dynamic creation of actors and actor communi-cation graph may not be available statically. In our mappingtechnique, when an actor instance is created dynamically, itis assigned an execution policy based on the actor type (oractor definition). Hence, every actor created statically or dy-namically is assigned an execution policy. When the actorcommunication graph cannot be determined statically, werun the program (using default thread mapping) on sampleinputs to fully or partially determine actor communicationgraph and we use the computed graph for mapping actorsto threads efficiently. In summary, we believe that the dy-namism in actor applications can be accounted in our actorsto threads mapping technique.

5.4 Threats to ValidityA threat to validity is our evaluation that uses benchmarkstranslated to Panini. However, we compare the performanceof three different mapping techniques using the same paniniprogram and not three different implementations of the pro-gram. The three mapping techniques are implemented inPanini’s runtime such that we achieve different actors tothreads mapping.

6. ConclusionIn this paper we investigated different aspects of actor-based programs that influence the mapping of actors to JVMthreads. We proposed a heuristic-based approach that givenan actor program and its communication graph produces ac-tor characteristics vectors for actors. Using the actor charac-teristics vectors, we apply a set of heuristics in a systematicmanner to assign execution policies for actors. The execu-tion policies for actors defines how actors are mapped toJVM threads. We proposed using our heuristic-based map-ping in place of default mappings. Our evaluation over awide range of actor programs shows that our heuristic-basedmapping achieves on average 50% improvement over de-fault mappings. Further, our heuristic-based mapping tech-nique assigns execution policies to actors automatically atcompile-time. This helps to reduce programmers time andefforts to arrive at initial optimal mappings.

At a higher-level, we believe that better abstractions thatenable improved modularity are important for concurrentprogramming [17–19]. However, a problem with abstrac-tions in practice is that the abstraction boundaries are of-ten breached for performance reasons. By providing an auto-matic technique for improving mapping of concurrency ab-stractions to threads, we hope to mimimize such breach ofabstraction and improve portability.

AcknowledgmentsThis work was supported in part by the NSF under grantsCCF-08-46059, CCF-11-17937, and CCF-14-23370,.

References[1] Panini Programming Language. http://paninij.org/.

[2] Tim Jansen. Actors guild. https://code.google.com/p/actorsguildframework/,2009.

[3] Computer language benchmarks game. http://benchmarksgame.alioth.debian.org/.

[4] Actor Collection. http://actor-applications.cs.illinois.edu/.

[5] Mike Rettig. Jetlang. http://code.google.com/p/jetlang/,2008-09.

[6] Panini Benchmark Programs. design.cs.iastate.edu/strategy/.

[7] StreamIt. http://groups.csail.mit.edu/cag/streamit/index.shtml/.

[8] G. A. Agha. Actors: a model of concurrent computation indistributed systems. 1985.

[9] S. Aronis, N. Papaspyrou, K. Roukounaki, K. Sagonas,Y. Tsiouris, and I. E. Venetis. A scalability benchmark suitefor erlang/otp. In Proceedings of Erlang ’12, pages 33–42.

[10] M. Astley. The actor foundry: A java-based actor program-ming environment. In Open Systems Laboratory, Universityof Illinois at Urbana-Champaign, 1998-99.

[11] M. Bagherzadeh and H. Rajan. Panini: A concurrent program-ming model for solving pervasive & oblivious interference.Technical Report 14-09, Iowa State University, August 2014.

[12] T. Desell and C. A. Varela. Salsa lite: A hash-based actorruntime for efficient local concurrency.

[13] E. Francesquini, A. Goldman, and J.-F. Méhaut. Actorscheduling for multicore hierarchical memory platforms. InProceedings of Erlang’13 workshop, pages 51–62. ACM.

[14] A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigor-ous java performance evaluation. pages 57–76.

[15] M. Gupta. Akka Essentials. Packt Publishing Ltd, 2012.

[16] P. Haller and M. Odersky. Actors that unify threads andevents. In Proceedings of Coordination Models and Lan-guages 2007.

[17] Y. Long, S. L. Mooney, T. Sondag, and H. Rajan. Implicitinvocation meets safe, implicit concurrency. In GPCE ’10:Ninth International Conference on Generative Programmingand Component Engineering, October 2010.

[18] H. Rajan. Building scalable software systems in the multicoreera. In 2010 FSE/SDP Workshop on the Future of SoftwareEngineering, Nov. 2010.

[19] H. Rajan, S. M. Kautz, and W. Rowcliffe. Concurrency bymodularity: Design patterns, a case in point. In 2010 Onward!Conference, October 2010.

[20] H. Rajan, S. M. Kautz, E. Lin, S. L. Mooney, Y. Long, andG. Upadhyaya. Capsule-oriented programming in the Paninilanguage. Technical Report 14-08, 2014.

40

[21] H. Sasaki, T. Tanimoto, K. Inoue, and H. Nakamura.Scalability-based manycore partitioning. In Proceedings ofthe 21st int. Conf. on PACT, pages 107–116, 2012.

[22] A. K. Singh, M. Shafique, A. Kumar, and J. Henkel. Mappingon multi/many-core systems: survey of current and emergingtrends. In Proceedings of DAC’13, page 1. ACM.

[23] L. Smith, J. Bull, and J. Obdrizalek. A parallel Java Grandebenchmark suite. In ACM/IEEE Conf. on Supercomputing,pages 6–6, 2001.

[24] T. Sondag and H. Rajan. Phase-based tuning for better uti-lization of performance-asymmetric multicore processors. In

International Symposium on Code Generation and Optimiza-tion (CGO), April 2011.

[25] S. Srinivasan and A. Mycroft. Kilim: Isolation-typed actorsfor java. In Proceedings of ECOOP’08.

[26] A. Tousimojarad and W. Vanderbauwhede. An efficient threadmapping strategy for multiprogramming on manycore proces-sors. arXiv preprint arXiv:1403.8020, 2014.

[27] J. Zhang. Characterizing the scalability of erlang vm on many-core processors. 2011.

41