Empirical Study of Usage and Performance of JavaCollections

Diego CostaInstitute of Computer Science

Heidelberg University,Germany

diego.costa@informatik.uni-heidelberg.de

Artur AndrzejakInstitute of Computer Science

Heidelberg University,Germany

artur@uni-hd.de

Janos SeboekInstitute of Computer Science

Heidelberg University,Germany

seboek@cl.uni-heidelberg.de

David LoSchool of Information Systems

Singapore ManagementUniversity, Singapore

davidlo@smu.edu.sg

ABSTRACTCollection data structures have a major impact on the per-formance of applications, especially in languages such asJava, C#, or C++. This requires a developer to select anappropriate collection from a large set of possibilities, in-cluding different abstractions (e.g. list, map, set, queue),and multiple implementations. In Java, the default imple-mentation of collections is provided by the standard JavaCollection Framework (JCF). However, there exist a largevariety of less known third-party collection libraries whichcan provide substantial performance benefits with minimalcode changes.

In this paper, we first study the popularity and usagepatterns of collection implementations by mining a codecorpus comprised of 10,986 Java projects. We use the re-sults to evaluate and compare the performance of the sixmost popular alternative collection libraries in a large vari-ety of scenarios. We found that for almost every scenarioand JCF collection type there is an alternative implementa-tion that greatly decreases memory consumption while of-fering comparable or even better execution time. Memorysavings range from 60% to 88% thanks to reduced overheadand some operations execute 1.5x to 50x faster.

We present our results as a comprehensive guideline tohelp developers in identifying the scenarios in which an al-ternative implementation can provide a substantial perfor-mance improvement. Finally, we discuss how some codingpatterns result in substantial performance differences of col-lections.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.

ICPE’17, April 22-26, 2017, L’Aquila, Italyc© 2017 ACM. ISBN 978-1-4503-4404-3/17/04. . . $15.00

DOI: http://dx.doi.org/10.1145/3030207.3030221

Keywordsempirical study, collections, performance, memory, execu-tion time, java

1. INTRODUCTIONCollection libraries and frameworks offer programmers data

structures for handling groups of objects in an abstract andpotentially efficient way. The popularity and importance ofsuch libraries and frameworks is indicated by the fact thatprogramming languages such as Java, C#, Python, Ruby,or C++ include them as a part of the core language envi-ronment (STL is a de facto standard in case of C++.)

Typically in such libraries each of the major abstract col-lection types (e.g., lists, sets and maps) has multiple imple-mentations adhering to the same API. Each implementationattempts to fulfill the requirements of different executionscenarios and thus provides different trade-offs. For exam-ple, Java’s ArrayList has a smaller memory footprint than aJava LinkedList, yet it has a higher asymptotic complexitywhen it comes to inserting or deleting elements at randompositions.

Given that modern programs use collections in thousandsof program locations [23], selecting appropriate type and im-plementation is a crucial aspect of developing efficient ap-plications. Choosing a wrong collection may result in per-formance bloat - the excessive use of memory and computa-tional time for accomplishing simple tasks. Numerous stud-ies have identified the inappropriate use of collections as themain cause of runtime bloat [5, 16, 27, 28]. Even in produc-tion systems, the memory overhead of individual collectionscan be as high as 90% [16].

The problem of selecting an appropriate collection be-comes more complex when accounting for third party li-braries. These allow for many more choices compared tothe standard Java Collection Framework (JCF), from simplealternatives to existing JCF implementations to collectionswith extra features such as immutability and primitive-typesupport.

Despite its importance, we found a gap in experimentalstudies comparing execution and memory performance ofnon-JCF collections. Partial benchmarks, especially on the

389

websites of the libraries, are common enough; they, however,do not give a performance comparison for different libraries,nor do they provide an evaluation under a large set of sce-narios. This paper attempts to fill this gap in experimentalstudies and derive a set of guidelines for developers on howcan they improve performance with little code refactoring.

To this aim (i) we study the usage of collections in realJava code via repository mining and (ii) evaluate the mem-ory consumption and execution performance of collectionclasses offered by six most popular collection libraries. Thekey question to be answered by our study is: ”Can we im-prove performance of applications in typical scenarios bysimply replacing collection implementations, and if yes, towhich degree?”. In particular, we explore alternatives to JCFimplementations under the same collection abstraction.

Our results show a considerable variance in performancein different implementations of the same data structure.Moreover, the best alternative implementations outperformthe collections provided by the standard JCF library acrossmany workload profiles, making them a better choice interms of memory consumption and runtime for most appli-cations in many circumstances. Improvements range from20% to 50 times in execution time, and saving up to 88% inmemory.

The contributions of this paper are as follows:

• We analyze the popularity and usage patterns of col-lection libraries based on mining a dataset with 10,986Java projects (Section 3).

• We propose a framework for systematic evaluation ofcollection performance. This framework can be furtherextended to cover new libraries, scenarios and imple-mentations (Section 4).

• We evaluate the performance of JCF and six majoralternative libraries in terms of execution and memoryunder a variety of scenarios. As a part of the study,we investigate the performance of primitive collections(Section 5).

• We provide a guideline for developers on replacingstandard JCF collections by alternative implementa-tions based on performance characteristics (Section 6).

• We explain the causes of performance differences of col-lection implementations by analyzing the source-codeand benchmark indicators for some relevant scenarios(Section 6).

2. BACKGROUNDCollections are data structures that group multiple ele-

ments into a single unit. A collection uses metadata totrack, access and manipulate its elements via specified APIs.Naturally, the metadata incurs extra memory consumption(collection overhead) in addition to the memory used by itselements (element footprint). Collections come in severalabstraction types, primarily list, map, set, and queue.

The standard implementation of the collections in Javais the Java Collections Framework, or JCF. It provides im-plementations of all major abstraction types in several vari-ants, e.g. ArrayList, HashMap, HashSet and others. JCF isimplemented purely as object collection, i.e. the metadatacontains only references to Java objects, but no data itself(Figure 1).

CollectionOverhead

ElementsFootprint

ElementsFootprint

CollectionOverhead

Object[]

int[]

Integer

4 bytes

16 bytes

Object collection Primitive collection

Object Collection Footprint

Pri

mit

ive

Co

llect

ion

Fo

otp

rin

t

Figure 1: Conceptual view of object collections vs.primitive collections with the example of ArrayList,and terms related to memory usage.

The JCF offers programmers a stable and reliable collec-tions framework. However, there exist many alternative col-lection libraries that can outperform standard JCF abstrac-tion types, and additionally target features not supportedby the JCF, such as immutable collections, multisets andmultimaps.

If a collection contains simple objects like Integer, Long,Double, Character, etc., primitive collections can be used toreduce the memory footprint. The key difference is that aprimitive collection stores data directly in an array of prim-itives (int, long, double, char), instead of using an array ofreferences to data objects. Figure 1 illustrates an ArrayListof integers as an object collection (left) and a primitive col-lection (right). The primitive variant reduces the collectionfootprint in two ways. First, the primitive collection needsonly a single reference to an array of primitives instead ofan array of references. Second, each primitive data element(type int) requires only 4 bytes instead of 16 bytes of anobject Integer. This example indicates that alternative col-lections can slash memory footprint of collections by a largefactor.

3. ANALYSIS OF COLLECTION USAGEIn this section we describe the usage patterns of collections

found in a large code corpus. Understanding such patternshelps us to identify the most popular collection types andimplementations. This study will help us devise a bench-mark presented later in the paper.

3.1 Data and Static AnalysisFor our analysis we use the GitHub Java Corpus [1], a

dataset from GitHub containing 10,986 Java projects, in to-tal more than 268 millions lines of code. This corpus consistsonly of projects with more than one fork to eliminate smallor inactive projects, which prevail at GitHub [12]. In ad-dition, the creators of GitHub Java Corpus analyzed andpruned this dataset manually in order to eliminate projectclones.

We categorize the whole data into two sets of reposi-tories, namely all projects provided by Java GitHub Cor-pus (FullSet) and a subset of the 50 most popular projects(Top50 ). The popularity of the latter set is based on rank-ing of projects provided by GitHub corpus [1], and it is illus-

390

trated by the fact that Top50 contains projects like EclipseIDE1, Clojure2, and Elastic Search3. By analyzing both setswe can investigate whether there is an impact of projectmaturity (approximated by popularity) on the patterns ofcollection usage.

We need to extract from the code every declaration of acollection and every site of a collection instantiation. Thisis needed to count the usages of collection classes, recordthe types of held elements, and count how often the initialcapacity is specified. We use JavaParser4 to extract eachvariable declaration and instantiation site in the code. Wethen filter the collection instances by applying a heuristic asdefined by the following regular expression:

.*List|.*Map|.*Set|.*Queue|.*Vector| .

This pattern finds all collection implementations of inter-est but also retrieves some false positives, e.g., java.util.BitSetis not a general purpose collection but is retrieved by ourheuristic regardless. We rank the retrieved types and man-ually inspect and filter out false positive for the top 99% ofthe retrieved data, ranked by occurrence.

3.2 Collections Usage in Real Code

Collection Types. We analyze the instantiations sites andextract the most used collections in the code (see Figure 2).Unsurprisingly, list is the most commonly used collectionabstraction, followed by map and finally by set. Our rankingis dominated by JCF as developers only rarely opted forothers, not nearly often enough to make it into our charts.

Among the abstraction types, lists are the most common(≈ 56%), followed by maps (≈ 28%) and sets (≈ 15%).Contrary to this, queues are rather rare (1.23% of usages).Among lists, ArrayList makes up the bulk of all collectionsusages, namely 47%. The LinkedList is surprisingly com-mon, with ≈ 5% and is ranked as the fourth most com-mon implementation. In maps, HashMap is most commonlyused, representing ≈ 23% of all collections instantiation, fol-lowed by LinkedHashMap (≈ 2%) and the TreeMap (≈ 1%).Set instances follow a similar pattern to maps, HashSet ismost used (≈ 10%), followed by TreeSet and LinkedHashSetwith both ≈ 1% of occurrences.

In summary, the top four most frequently used collectionsare JCF implementations: ArrayList, HashMap, HashSetand LinkedList. Together they account for approximately83% of all collections declared in the repository. We founda similar distribution in Top50, which have a slightly higherusage of concurrent collections and a lower variety of types.

Element Types. By element type we mean the object typesheld by collections. Understanding their distribution helpsus to cover in our benchmark the most frequent real sce-narios. To simplify, we group the element types into fourcategories:

• Strings: String, String[], and String[][].

• Primitives: The primitive-wrappers such as Double,Float, Long, Short, Integer, as well as Boolean, Char-acter, and their respective arrays.

1 https://github.com/eclipse/eclipse.jdt/2https://github.com/clojure/clojure3https://github.com/elastic/elasticsearch4https://github.com/javaparser/javaparser

• Collections: Collection types that can be identified byour heuristic from Section 3.1.

• Other: All the classes and data types not fitting anyof the mentioned categories.

The result of our analysis, presented in Figure 3, shows aclear similarity between the results in FullSet and Top50. Apattern that emerges is the similar usage of primitive wrap-pers throughout the categories, always ranging from ≈ 5%to ≈ 8% of the declarations. This category is particularlyinteresting because it contains the collections that can bereplaced by primitive collections with simple code refactor-ing.

The remaining categories show a distinct pattern for eachof lists, maps and sets. Lists have a higher variability ofelement types, but hold strings ≈ 20% of the time. Setshold strings more often than lists, ≈ 31% of the time. Mapsalso hold strings very often: ≈ 70% of all declared maps usethe String as a key and ≈ 28% as a value. Map values areoften a collection (≈ 15%) pointing to a common usage ofmaps as a collection holder.

Initial Capacity Specification. Defining an appropriateinitial capacity of a collection is a simple but effective methodfor optimizing runtime and memory. We sample 400 ofthe instantiation sites of ArrayList, HashSet and HashMap,which give us 5% confidence interval. Then we manuallycategorize whether a developer specifies an initial capacity,copies this instance from another collection, or uses defaultconstructor values.

Figure 4 shows that programmers specify initial capacitiesfor ArrayList only in ≈ 19% of the declarations. HashMapand HashSet have their capacity specified in ≈ 7% and≈ 8% of collection declarations, respectively. Our anal-ysis with the Top50 provides very similar results, with aslightly higher percentage of initial capacity in ArrayListdeclarations. Interestingly, HashSet is the category morecommonly created through copy instruction (in ≈ 11% ofcases in FullSet and ≈ 14% in Top50 ).

Project Maturity and Collections Usage. We found sim-ilar results in our analysis of FullSet and Top50. This indi-cates that, regardless of the project maturity, programmersinstantiate collections in a similar fashion. They mostly se-lect standard collections types, and rarely specify the initialcapacity in a collection instantiation.

4. EXPERIMENTAL DESIGNIn this section we describe the design of experiments for

evaluating performance of collection implementations in termsof execution time and memory usage. In particular:

1. We identify a suitable set of collection libraries for thisevaluation through ranking of libraries in terms of theirpopularity.

2. We describe a benchmark framework capable of mea-suring the steady-state performance of collections withhigh precision.

3. We design an experimental plan which covers a largeset of usage scenarios based on the findings of usagepatterns reported in Section 3.

391

ArrayList47%

LinkedList5%

CopyOnWriteArrayList<1%

Other Lists4%

HashSet10%

TreeSet1%

LinkedHashSet1%

Other Sets3%

HashMap21%

LinkedHashMap2%

TreeMap1%

Other Maps4%

LinkedBlockingQueue<1%

Other Queues1%

Lists

Sets

Maps

Queue

Figure 2: Distribution of instantiated collectiontypes in FullSet. Every specific collection in thischart is from JCF, as standard implementations arethe most prominent ones.

0%

20%

40%

60%

80%

100%

Lists Sets MapKeys

MapValues

FullSet

Lists Sets MapKeys

MapValues

Top50

String

Numeric

Collections

Other

Figure 3: Distribution of element types by collec-tion for both the FullSet and the Top50 set.

ArrayList HashMap HashSet

Top50

Specified

Default

Copy

0%

20%

40%

60%

80%

100%

ArrayList HashMap HashSet

FullSet

Figure 4: Statistics on the style of specifying ini-tial capacity of collections for FullSet and Top50.”Specified” denotes that the developer explicitlyset the initial capacity, and ”Copy” that a copyconstructor was used.

The results of rigorous statistical analysis of each evaluatedscenario are presented then in Section 5.

Note that we compare the performance within the sameabstraction type (i.e. list, map, set). Our experimental pa-rameters include the number of elements held in a collection,and a particular usage scenario (Section 4.2).

4.1 Selection of Collection LibrariesWe search the Web for alternative collection libraries im-

plemented in Java and find a total of 14 libraries. Fromthese, we use two different approaches to extract a suitableset for our experimental evaluation. First, we rank themby popularity of their project in GitHub, see Table 1. Sec-ond, we analyze how many times they have been includedin existing benchmarks (Table 2).

The GitHub metrics such as number of stars and numberof watches provide a simple but effective method for rank-ing software projects. Such metrics have also been used asa criterion in other studies [19, 22]. To account for librariesthat are not on GitHub, we retrieve a set of collection bench-marksfrom the web, and count for each collection library inhow many benchmarks it was evaluated.

Furthermore, we filter libraries that do not provide imple-mentations that can serve as a replacement to JCF collec-tions. This excludes Javaslang as it provides only immutablecollections for lambda function usage. In the final step weselect the top five libraries from each ranking and mergethem into a single list. This yields seven unique libraries(three of them occur in both rankings) to be included in ourexperimental evaluation. For this study we use the JCF ofJava 8 (jdk1.8.0 65.)

4.2 Benchmark DesignA managed runtime environment such as the Java Vir-

Table 1: Ranking of collection libraries by GitHubpopularity. We select the top 5 collections from thisranking. Data was obtained on 28 September 2015.

Rank Library # Stars # Watches1 Guava 5067 6412 GS-Collection 1293 1963 Koloboke 369 69* Javaslang 309 314 HPPC 189 315 Fastutil 69 66 HPPC-RT 8 3

tual Machine poses a challenge for performance measure-ment. Numerous factors can affect performance: the effectsof the garbage collector, the Just in Time compilation (JIT),the heap size, the method sampling optimization, and manyother unpredictable system effects [3, 4, 18]. To reduce theimpact of these factors we follow the suggestions by Georgeset al. [4] and implement a four-step methodology for evalu-ating a steady state performance of collection implementa-tions:

S1 For each scenario we execute ten warm-up iterationsto achieve a steady performance 5.

S2 We execute 30 iterations while measuring our responsevariables. Each iteration executes the same operation(sample) in an uninterrupted fashion for five seconds.After reaching the timeout, we calculate the averageof all samples as a result. Each result also contains

5Our preliminary analysis showed that the execution timeof the experiment converges after seven warm-up iterations

392

Table 2: Ranking of collection libraries by numberof occurrences in benchmarks. We select the top 5collections from this ranking.

Rank Library # Occurrences1 JCF 132 Trove 103 HPPC 64 Koloboke 55 Fastutil 46 GS-Collection 47 Javolution 48 Guava 39 Mahout 310 Commons 211 Brownies 112 Colt 113 Javaslang 013 HPPC-RT 0

the experimental error of the iterations at a 99% con-fidence interval.

S3 We execute steps S1 and S2 twice to avoid circumstan-tial external influence.

S4 We analyze the results of 2 x 30 iterations using rigor-ous statistical methods. In detail, we analyze the vari-ance with ANOVA [17] and compare multiple meansaccounting for their experimental errors.

We perform this experimental methodology on our bench-mark suite called CollectionsBench. CollectionsBench is builtupon JMH6, a Java harness framework for building, run-ning and pre-analyzing benchmarks. JMH provides supportfor benchmark forks, warm-up iterations, and can give thebenchmark results with nanosecond precision.

To reliably measure performance using CollectionsBench,we perform the following steps: First, we guarantee a ho-mogeneous benchmark behavior throughout different collec-tion implementations. The Template design pattern helpsus reuse the same test code in all JCF compliant libraries(see Table 3) and to delegate the collection creation to thelibrary specific code. Second, we consume every non-voidreturn to avoid undesired dead-code optimization. All thesesteps help to minimize sources of non-determinism in theJVM.

Finally, to only measure time and memory on the collec-tions operations, we divide our experiment into two distinctsetup and benchmark phases: In the setup phase we allocateall the elements and insert them into an instance of the eval-uated collection. The values of each element are generatedrandomly through a uniform distribution, and we then reusethe random seed to ensure the same conditions in all tests.In the benchmark phase we execute the collection operationsand measure the following response variables:

• Execution time: Time spent executing the benchmarkin nanoseconds, using JMH native support.

• Memory allocation: The sum of memory requestedduring the benchmark execution. Since we allocatethe elements in the setup phase, this variable showsonly the memory requested for the collection overhead(including the collection object header and eventual

6http://openjdk.java.net/projects/code-tools/jmh/

memory padding), but does not consider the elementfootprint. We extract this information through theJMH GC profiler.

We also collect some performance indicators through thePerf7 profiler, such as the number of instructions executed,cache miss rate and branch misprediction rate. In Section 6we analyze those indicators along with the source-code tounderstand the reasons for performance differences of im-plementations.

The memory allocated during the benchmark is sensitiveto buffers and temporarily allocated objects. For instance,when expanding the ArrayList allocates a new and largerbuffer to accommodate more elements. The previously al-located buffer and the new one will be measured by thememory allocation variable. Therefore, the memory alloca-tion alone cannot provide an accurate view of the memoryusage of a collection.

To account for this problem, we evaluate the memory us-age of a collection by computing their collection overhead.We use the tool Java Object Layout (JOL)8 to retrieve thecollection overhead of each implementation. We then ana-lyze both memory allocation and overhead together, to givea detailed perspective of a collection’s memory consump-tion.CollectionsBench is open-source and is fully availableonline9.

4.3 Experimental PlanningOur experimental plan takes into consideration the find-

ings on collections usage presented in Section 3. First, we fo-cus on the most frequently used collection types while inves-tigating possible alternatives (from third-party libraries) tothe JCF collections. Therefore we select the four most usedcollection types, ArrayList (AL), LinkedList (LL), HashMap(HM), HashSet (HS), which accounts for more than 84% ofall declared collections.

Most of the libraries do not implement an alternative toLinkedList, so we opted to compare JCF LinkedList imple-mentations against ArrayList alternatives, as they are in-terchangeable. We also profile primitive alternatives to thetop three collection types, as they provide a small-footprintoption to collections that hold primitive wrappers (see Ta-ble 3).

Second, we pay a special attention to collections holdingthe element type String. Strings are particularly interestingin Java due to their extensive use, and have been studiedby various authors [13]. To expand on the results, we eval-uate the collection implementations for Integer and Longobjects as well. We opt for these two numeric objects asthey are representative: primitive-wrappers are the secondmost common category identified. Moreover, they mainlyrepresent objects with 32 bytes and 64 bytes respectively, acommon object size.

Third, we do not specify the initial capacity in our bench-mark because from our static analysis results, specifying theinitial capacity is done rarely in real code. Since we aim toprovide results that are useful to the widest range of pro-grammers, we evaluate collections with their default initialcapacity.

Fourth, given C as a collection populated with N elements

7https://perf.wiki.kernel.org/index.php/Main Page8http://openjdk.java.net/projects/code-tools/jol/9https://gitlab.com/DiegoCosta/collections-bench

393

Table 3: The evaluated collection implementations. The primitive implementations are marked with (p). Thecolumn JCF indicates whether the libraries provide implementations compatible with the Java CollectionFramework (only for object collections). In total we evaluate 33 implementations.

Library Version JCF List(AL/LL) HashMap (HM) HashSet (HS)

JCF [20] 8.0 65 yesArrayList

HashMap HashSetLinkedList

Guava (Gu) [8] 18.0 no HashMultimap HashMultiset

Fastutil (Fu) [24] 7.0.10yes ObjectArrayList Object2ObjectOpenHashMap ObjectOpenHashSet– IntArrayList(p) Int2IntOpenHashMap(p) IntOpenHashSet(p)

Koloboke (Ko) [2] 0.6.8yes HashObjObjMap HashObjSet– HashIntIntMap(p) HashIntSet(p)

HPPC (HP) [21] 0.7.1no ObjectArrayList ObjectObjectHashMap ObjectHashSet– IntArrayList(p) IntIntHashMap(p) IntHashSet(p)

GSCollections (GS) [7] 6.2.0yes FastList UnifiedMap UnifiedSet– IntArrayList(p) IntIntHashMap(p) IntHashSet(p)

Trove (Tr) [6] 3.0.3yes THashMap THashSet– TIntArrayList(p) TIntIntHashMap(p) TIntHashSet(p)

in the setup phase, we evaluate the collections under thefollowing scenarios:

• populate: add N random elements into an empty col-lection.

• iterate: iterate through all elements of C.

• contains: check whether C contains an existing ran-dom element.

• add : add a random element to C. To keep C with thesame size throughout the experiment, we remove theadded element at the end of the scenario.

• get : get a random existing element from C.

• remove: find and remove a random element from C.Analogously to add, we add the removed element in Cat the end of the scenario.

• copy : copy C into an empty collection using the copyconstructor.

We could not obtain the size of collections through staticcode analysis. To compensate for this fact and to accountfor various scenarios, we run our benchmarks with multiplevalues for N ranging from 100 to 1M, with the interval setas a power of ten (five categories of size).

In summary, we perform a factorial experiment [17] withthree element types and five collection sizes, in a total of 15different configurations. This 15-configuration experiment isrun through the seven scenarios for each of the 33 collectiontypes (see Table 3). We execute a total of 3,465 experiments,where each experiment lasts five minutes including replica-tions and forks. Running the full benchmark takes 12 daysto finish.

We conduct our experiments on a machine with a E5-16603.3GHz CPU, 64GB RAM using Linux 3.16.0-53. We use 64-bits JVM HotSpot, and the jdk1.8.0 65 as our Java version.All tests were executed in a single-threaded environment, asour target collections were built for such settings.

5. RESULTSOur goal is to find a superior alternative that can out-

perform JCF in terms of execution time and/or memoryconsumption in several scenarios, while introducing no orminimal penalties in others. To achieve this, we ask thefollowing research questions:

RQ1. Does the type of elements stored in collections influ-ence execution time?

RQ2. Are there superior alternatives to the most used JCFcollections with regard to execution time?

RQ3. Do primitive collections perform better than JCF col-lections with regard to execution time?

RQ4. Are there superior alternatives to the most used JCFcollections with regard to memory consumption?

RQ1 addresses the impact of element type in the exe-cution time of collections; RQ2 and RQ3 show the anal-ysis of alternative object and primitive-collections on exe-cution time; RQ4 reports superior object-collection alter-natives on memory consumption. We omit the analysis ofprimitive-collections on memory consumption, as this ques-tion is rather well-explored and exemplified in Section 2.

RQ1. Does the type of elements stored in col-lections influence execution time?

Key result: The element type has a significant influencein the execution time in all major collection abstractions.For lists, however, this impact is homogeneous through-out all implementations.

We analyze the impact of the element type in the object-collections execution time to verify whether our results canbe generalized beyond the types evaluated. We use theANOVA method [17], with a 95% confidence interval andanalyze the influence of element type on lists, sets and maps.

The element type influences execution time significantlyin our three main categories (p < 0.05). For lists, however,the element type factor does not interact with the list im-plementation (p = 0.76), indicating that it influences theperformance regardless of the list type. Thus, we expectthat our results for lists can be generalized beyond the typeused. We report the results for sets and maps only for theString element type as it represents the largest amount ofcollection instantiations.

394

Libs populate iterate contains add get remove copyFU

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 3 3

GS1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

HP1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 6 7 14 14 9

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

Libs populate iterate contains add get remove copy

FU1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

-4 -4 -4 -3 -3

GS1 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 -1 0 0 0 0 0 0

HP1 2 2 1 2 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 0 0 0 0 0 0 -1 -1 -1 0

-6 -7 -14-14 -9100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

(a) ArrayList alternatives

Libs populate iterate contains add get remove copyFU

1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1

GS1 1 1 1 1 2 3 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Gu3 2 2 2 2 7 10 7 8 7 1 1 1 1 1 3 2 2 3 2 2 2 2 1 1 2 2 2 3 2 9 10 9 13 9

HP2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2

Ko2 6 4 4 2 3 3 2 1 1 1 2 3 3 1 1 1 1 1 1 1 2 3 3 1 2 11 13 15 4 0 0 0 0 0

Tr3 2 2 1 1 7 9 5 4 3 1 1 1 1 1 3 2 2 3 2 1 1 1 1 1 2 2 2 1 1 4 4 6 5 7

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

Libs populate iterate contains add get remove copyFU

0 -1 -1 0 1 -1 -2 -1 0 1 -1 0 0 0 1 0 0 0 0 -1 -1 0 0 0 0 -1 -1 -1 -1 0 -1 -1 -1 -2 -1

GS-1 -1 -1 -1 0 -2

-3-2 -1 0 -1 0 -1 0 0 0 0 0 0 0 -1 0 -1 0 0 0 0 0 1 1 0 -1 -1 -1 -1

Gu -3-2 -2 -2 -2

-7 -9 -7 -8 -70 0 -1 0 0

-3-2 -2

-3-2 -2 -2 -2 -1 0 -2 -2 -2

-3-2 -9 -10 -9 -13 -9

HP-2 -2 -1 -1 0 -1 -1 0 0 1 -1 0 0 0 0 0 0 0 0 -1 -1 0 0 0 0 -1 -1 -1 -1 0 -1 -2 -2 -2 -2

Ko-2

-6 -4 -4 -2 -3 -3-2 -1 1 -1 -2

-3 -3-1 0 -1 -1 -1 -1 -1 -2

-3 -3-1 -2

-11-13-15 -4 4 6 9 11 11

Tr -3-2 -2 -1 -1

-7 -9 -5 -4 -31 1 1 1 2

-3-2 -2

-3-2 0 1 1 1 2 -2 -2 -2 -1 -1 -4 -4 -6 -5 -7

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

(b) HashMap alternatives (element type = String)

Libs populate iterate contains add get remove copyJCF

1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0

FU1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0

GS1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0

HP1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 0 0 0 0 1 0 0 0 1 1 1 1 1 1

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

Libs populate iterate contains add get remove copyJCF

-1 0 0 0 0 0 0 0 1 1 2 2 2 2

31 0 0 1 0

* * * * *1

4 6 32

11 12 16 11 12

FU0 1 1 0 0 0 0 0 1 1 2 2 2 2

31 1 1 1 1

* * * * *1

4 6 32

3 3 4 3 4

GS0 1 1 0 0 0 0 0 1 1 2 2 2 2

31 1 1 1 1

* * * * *1

4 6 32

10 13 16 12 12

HP1 2 2 1 2 0 0 -1 -1 0 1 2 2 2 2 2 1 1 1 1

* * * * *1

3 6 32 2 2 1 -1 1

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

(c) ArrayList alternatives to LinkedList

Libs populate iterate contains add remove copyFU

1 1 1 1 2 1 0 1 1 0 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 0 1

GS1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1

Gu1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1

HP2 2 1 1 2 1 1 1 1 0 1 1 1 1 1 1 1 2 1 1 1 1 2 1 1 1 1 1 1 1

Ko1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 0 0 0 0 0

Tr2 2 2 1 2 2 2 1 1 1 1 1 1 1 1 2 2 3 2 2 2 2 2 1 2 1 2 2 1 2

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

populate iterate contains add remove copyFU

0 -1 -1 0 -2 1 2 2 1

30 0 0 1 1 0 0 -2 0 0 0 -1 -1 1 1 2 1 0 2 1

GS-1 -1 0 0 -2 -1 1 0 0 0 -1 -1 0 0 0 0 0 0 0 0 0 0 0 2 1 1 0 1 2 1

Gu0 0 0 0 -1 -2 0 -1 -1 -1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 -1 -2 -2 0

HP-2 -2 -1 0 -2 0 2 1 0

30 0 0 1 1 0 0 -2 0 0 0 -1 -2 1 0 1 -1 -1 1 0

Ko0 0 0 1 0 0 2 2 1

30 0 0 1 1 0 0 -2 0 0 0 -1 -1 1 1

13 16 29 29 51

Tr-2 -2 -2 0 -2 -2 -2 0 0 2 0 0 0 0 1 -2 -2 -2 -2 -2 -2 -2 -2 -1 -2 0 -2 -2 1 -2

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

(d) HashSet alternatives (element type = String)

Speedup

1

> + 2x

1

+ 1.5x

1

--

2

- 1.5

2

> - 2x

Figure 5: Heatmap showing speedup/slowdown (as defined in Equation 1) of alternative collections comparedto JCF, per scenario and size. Green cells indicate a speedup, red cells represent a slowdown. Performanceratios larger than factor 2x are rounded to the nearest integer and printed inside the cell. LinkedList getratios are substantially higher and not shown.

0

0.4

0.8

1.2

100 1000 10000 100000 1000000

JCF

NO

RM

ALI

ZED

TIM

E

N

JCF FU HP GS

Array List populate

(a) populate

0

4

8

12

16

20

100 1000 10000 100000 1000000

JCF

NO

RM

ALI

ZED

TIM

E

N

JCF FU HP GS

ArrayList - copy

(b) copy

Figure 6: Execution time profiles of Array Lists forthe populate and copy scenarios.

RQ2. Are there superior alternatives to the mostused JCF collections with regard to executiontime?

Key result: GSCollections provides a superior alterna-tive to JCF ArrayList. LinkedList is outperformed byany other ArrayList implementation, and both the Hash-Set of Koloboke and Fastutil are a solid improvementover the standard one. We found no superior alternativeto the JCF HashMap, which provides a stable and fastimplementation throughout all the scenarios.

We report our results as a comparison to the JCF im-plementation instead of the absolute performance. First,we extract only comparisons where the errors (99% confi-dence interval) do not overlap. Then we calculate the im-pact of an alternative collection over JCF using the followingspeedup/slowdown S definitions:

S =

Tjcf

Talt, if Tjcf > Talt

− Talt

Tjcf, otherwise

(1)

where Tjcf and Talt are the time obtained with using theJCF implementation and the time using an alternative im-plementation respectively.

We present our results in a broad heatmap analysis inFigure 5, labeled by color. We make an in-depth analysisfor each category in the remainder of the section.

Linked List - addAll

0

0.25

0.5

0.75

1

1.25

1.5

100 1000 10000 100000 1000000

LIN

KED

LIST

JCF

NO

RM

ALI

ZED

TIM

E

N

LinkedList JCF ArrayList JCF ArrayList GS

(a) populate

Linked List - remove

0

0.25

0.5

0.75

1

1.25

1.5

100 1000 10000 100000 1000000

LIN

KED

LIST

JCF

NO

RM

ALI

ZED

TIM

E

N

LinkedList JCF ArrayList JCF ArrayList GS

(b) remove

Figure 7: Execution time profiles of LinkedList forpopulate and remove scenarios.

ArrayList. In the populate scenario occurrences of JCF Ar-rayList can be replaced by the implementation from GSCol-lections to achieve faster population without compromisingthe speed of any other operation. In fact, all evaluated al-ternatives are faster than JCF when populating the list (seeFigure 6.) The HPPC implementation is, however, slowerwhen iterating the elements in both the iterate and the con-tains scenario. In addition, both Fastutil and HPPC copytheir lists from 3 to 14 times more slowly.

LinkedList. JCF LinkedList is outperformed by all ArrayListimplementations by a large margin, even in scenarios whereLinkedList has a theoretical advantage (Figure 7). Thisis the case for the remove scenario, where we search andremove the element through the remove(Object) method.Despite the asymptotic advantage, LinkedList was outper-formed by a large margin forN ≥ 1k. Furthermore, LinkedListhas a comparable performance in the populate scenario, evenwithout having to reallocate its buffer (as ArrayList does).The LinkedList is up to three times slower when searchingfor a random element (N = 1M), and it was also outper-formed in the iteration scenario by some ArrayList variants,but only for N > 10k. Note that in our benchmark designthe LinkedList is unfragmented as the elements are insertedwithout any removal. Hence, we consider these results abest-case scenario for LinkedList.

HashMap. JCF HashMap provides solid performance andcannot be easily replaced by any alternative in terms of ex-

395

HashSet iterate

0

0.25

0.5

0.75

1

1.25

100 1000 10000 100000 1000000

JCF

NO

RM

ALI

ZED

TIM

E

N

JCF FU Ko

(a) iterate

HashSet copy

0

0.25

0.5

0.75

1

1.25

100 1000 10000 100000 1000000

JCF

NO

RM

ALI

ZED

TIM

E

N

JCF FU Ko

(b) copy

Figure 8: Execution time profiles of HashSets hold-ing String elements for the iterate and copy scenar-ios.

ecution time improvement. As shown in Figure 5, someimplementations have a comparable performance, like Fas-tutil, GSCollections and HPPC. Standard HashMap is out-performed only in the copy scenario, where Koloboke is ableto copy up to 11x faster. However, Koloboke is substantiallyslower in most of the other remaining scenarios.

HashSet. Standard HashSet is outperformed by many al-ternatives, mostly in the iterate and copy scenarios (Fig-ure 8). Koloboke is a superior alternative to JCF, outper-forming JCF HashSet for large set iteration, and copying atleast 10x faster. Fastutil and GSCollections can also be se-lected as good alternatives: both are slower when populatingbut faster in the copy and and iterate scenarios. The JCFimplementation is faster than the majority of alternativeswhen adding elements to the set, but is often outperformedin the remaining scenarios for large workloads (N > 100k).

RQ3. Do primitive collections perform betterthan JCF collections with regard to executiontime?

Key Result: For all three abstraction types (list, map,set) we found a set of primitive implementations thatare superior to JCF implementations. GSCollectionsand Koloboke provide the fastest primitive-based alter-natives, followed by Fastutil and HPPC. Trove is oftenoutperformed by JCF implementations in this context.

Our results for analyzing the experiments are summarizedin a colored heatmap in Figure 9.

Primitive ArrayList. Primitive lists provide a superior al-ternative to JCF ArrayList (Figure 9a). The speedup insome cases reaches four times better than the baseline whenchecking or removing an element from the list. GSCollec-tions consistently outperforms the JCF, followed by Fastu-til, which is slightly slower when iterating through the list.HPPC and Trove are much slower when copying their lists,and should be avoided if the workload demands this opera-tion often.

Primitive HashMap. As a consequence of the standardJCF HashMap’s good performance, in some scenarios thereis no underlying benefit gained from changing to a primitive-type map implementation (Figure 9b). Iterations are espe-cially good with the standard HashMap, up to five timesfaster than the alternatives. For the contains, add, get andremove scenarios, most primitive implementations can pro-

Libs populate iterate contains add get remove copy

FU0 0 0 0 0 2 2 2 1 2 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1

GS0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1

HP1 1 1 1 0 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 5 7 8 9 7

Tr0

0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 3 8 9 7 4

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

Libs populate iterate contains add get remove copy

FU 3 3 32 2 -2 -2 -2 -1 -2 1 2

3 3 30 1 1 0 0 0 0 0 0 0 1 2

3 4 30 0 0 0 0

GS 3 3 32 2 1 1 1 1 1 2 2

3 3 40 0 1 0 0 0 0 0 0 0 1 2 2

3 30 0 0 0 0

HP2 1 2 2 2 1 1 1 1 2 1 2

3 3 31 1 1 1 1 0 0 0 0 0 1 2

3 4 3 -5 -7 -8 -9 -7

Tr 3 3 32 2 1 1 1 1 1 0 0 0 1 1 0 1 1 0 1 0 0 0 0 0 1 2 2

4 3 -3 -8 -9 -7 -4100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

(a) ArrayList primitive alternatives

Libs populate iterate contains add get remove copy

FU1 1 1 1 0 4 4 4 4 2 1 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 2 2 2 3 1

GS1 1 1 1 0 3 2 5 4 2 1 1 1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0 0 0 0 0 0

HP3 2 1 1 0 2 2 3 3 2 1 1 1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 2 3 3 4 2

Ko1 1 1 1 0 2 2 3 3 2 1 1 1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 0 0 0 0

Tr2 2 2 1 1 3 3 4 3 2 1 1 1 1 0 2 2 2 2 1 1 1 1 1 1 2 2 2 2 1 4 3 4 4 3

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

Libs populate iterate contains add get remove copy

FU-1 1 0 -1 3

-4 -4 -4 -4-2 0 1 2 2

51 1 2 2

40 1 1 2

30 0 0 1 2 -2 -2 -2

-3-1

GS-1 1 0 0 3

-3 -2 -5 -4-2 -1 1 2 2

31 1 1 0 1 0 1 2 2

31 1 2 1 2 2

42 2

6HP -3

-2 -1 -1 2 -2 -2

-3 -3-2 0 1 1 2

41 1 1 1 2 0 1 1 2 2 -1 -1 0 1 2 -2

-3 -3 -4-2

Ko-1 2 1 0 3 -2 -2

-3 -3-2 0 1 2 2

31 1 1 1 2 0 1 1 2

3-1 0 0 1 2 2

4 52

4Tr

-2 -2 -2 -1 1

-3 -3 -4 -3-2 -1 0 1 1

3-2 -2 -2 -2 1 -1 0 1 1 2 -2 -2 -2 -2 1

-4 -3 -4 -4 -3100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

(b) HashMap primitive alternatives

Libs populate iterate contains add remove copyFU

1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1 1 0

GS1 1 1 1 0 1 0 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0

Tr1 1 1 1 0 1 1 1 1 0 1 1 1 1 0 2 2 2 2 1 1 2 1 1 0 2 3 2 2 1

HP2 1 1 1 0 1 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 0

Ko1 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0

100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

Improvement

1

> + 100%

1

+ 50%

1

0%

2

- 50%

2

> - 100%

Libs populate iterate contains add remove copyFU

1 2 1 2

3-1 1 1 0

30 0 1 1 2 1 0 0 0 1 1 0 0 1 2 2

32 1

3GS

1 2 0 2

41 2 1 1

40 0 1 1

40 -1 0 -1 2 1 0 2 2

4 9 27151533Tr

-1 -1 0 2 2 -1 1 1 1

30 0 1 1 2 -2 -2 -2 -2 1 -1 -2 -1 -1 2 -2

-3-2 -2 -1

HP-2 -1 1 1

31 2 2 1

41 1 1 2

41 0 1 1

31 0 0 2

42 0 1 0 2

Ko1 2 2 2

4-1 1 1 1

31 0 1 1 2 1 0 1 1 2 0 0 0 1

3 3 7 32

7100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M 100 1K 10K 100K 1M

Speedup

1

> + 2x

1

+ 1.5x

1

--

2

- 1.5x

2

> - 2x

(c) HashSet primitive alternatives

Figure 9: Heatmap showing the speedup/slowdown(Equation 1) of primitive collections (int) comparedto the JCF collections holding Integers, per scenarioand workload.

vide an improvement. When also considering the copy sce-nario, GSCollections and Koloboke provide the highest per-formance gain among the alternatives.

Primitive HashSet. Primitive-based sets can be beneficialfor the application’s performance when it comes to a largenumber of elements (Figure 9c). In fact, the speedup com-monly reaches four times as fast across many scenarios withN = 1M . GSCollections, Fastutil and Koloboke providesuperior alternatives and can copy their instances up to27 times faster. HPPC is also a good alternative but itdoes not provide the same copy operation speed gain as theother mentioned libraries. Trove’s implementation was theonly primitive-based HashSet outperformed by the standardHashSet in the add, remove and copy scenarios.

RQ4. Are there superior alternatives to the mostused JCF collections with regard to memoryconsumption?

Key result: We found no superior alternative to theJCF ArrayList in terms of memory overhead, but Fastu-til allocates less memory when populating its own list im-plementation. Numerous alternatives offer a superior al-ternative to the JCF HashMap and HashSet. LinkedListhas a higher overhead than any considered ArrayList al-ternative.

In this section, we analyze the difference in the overheadof collections as the main goal for a superior collection re-placement. As a supplementary analysis, we also looked atan aspect often neglected by micro-benchmarks: memoryallocation. A collection that allocates more memory thanindicated by its overhead requests memory for two reasons:(i) allocation as a buffer for future operations, (ii) alloca-tion for temporary objects. The latter reason has a negativeimpact on performance and can be used as an indicator ofhow often a collection implementation can trigger the action

396

Table 4: Comparison of collection overhead andmemory allocation of various implementations. Theoverhead is given in bytes in the form α×N+β, whereN is the number of elements in the collection, andα and β are implementation-specific factors. Theaverage allocated is given in bytes per element.

Category LibsCollection Avg AllocatedOverhead populate copy

ArrayLists

JCF 4 ×N + 24 14.83 4.01Fu 4 ×N + 24 10.07 4.02GS 4 ×N + 24 13.56 4.01HP 4 ×N + 48 15.10 4.01

LinkedList

JCF 24 ×N + 32 24.07 28.11

HashMaps

JCF 36 ×N + 48 49.93 41.05Fu 8 ×N + 64 35.52 42.18GS 8 ×N + 64 59.87 24.62Gu 96 ×N + 64 96.90 132.36HP 8 ×N + 96 36.41 18.16Ko 8 ×N + 232 35.47 17.96Tr 8 ×N + 72 47.23 18.37

HashSets

JCF 36 ×N + 80 48.85 40.43Fu 4 ×N + 40 24.43 8.43GS 4 ×N + 32 34.44 14.21Gu 52 ×N + 112 64.77 58.25HP 4 ×N + 88 16.88 8.43Ko 4 ×N + 208 16.84 8.43Tr 4 ×N + 64 22.25 8.35

of the Garbage Collector (GC). Collections that require fre-quent intervention of the GC can reduce the performance ofan application in a long run, a behavior difficult to observewith micro-benchmarks.

Since the elements are pre-allocated in the setup phase,memory allocation only comes into play when elements areadded to the collection, namely by populate and copy oper-ations.

ArrayList. All implementations have a similar overhead (seeTable 4). Due to its buffer reallocation, each implementa-tion allocates on average three times its own overhead in thepopulate scenario. Regarding only memory allocation, Fas-tutil can be a good alternative, saving 30% of allocations inthe populate scenario.

LinkedList. We show in Table 4 that the standard LinkedListimplementation has an overhead of 24 bytes per element, asopposed to the 4 bytes required in each ArrayList implemen-tation. This is a consequence of the pointers to neighbor el-ements for each LinkedList entry. Essentially, an ArrayListsaves 83% of the memory overhead. Despite the buffer ex-pansion of ArrayList, LinkedList still allocates twice as muchmemory in the populate scenario, and it allocates five timesmore memory when copying from a previous instance.

HashMap. We can observe in Table 4 that the JCF im-plementation has a considerably higher overhead. StandardHashMap has an overhead of 36 bytes per each entry in themap while almost every other alternative consumes only 8bytes. This difference occurs because JCF uses a Node ob-ject for each entry, a structure that contains three references(12 bytes) and a primitive (4 bytes), but, being an object,also has an overhead of 12 bytes of header and 4 bytes lostdue to alignment. The alternatives do not define each entry

as an object and instead use two arrays where each key andvalue pair are stored. As a consequence, they save 77% ofmemory overhead. Regarding allocations however, the dif-ference of JCF and the alternatives are not quite as drasticas in the collection overhead. This means that even thoughthe alternatives have five times less overhead, the memoryinvolved in the map operations is of a similar size.

HashSet. Similarly to Maps, JCF HashSet is implementedwith a larger overhead than almost any of the potential al-ternatives. In fact, it uses a HashMap internally to store theelements, causing the same additional overhead of 36 bytesper element, due to the HashMap’s Node object. The al-ternatives, on the other hand, implement the HashSet as asimple single array. This allows them to save 88% of memoryoverhead compared to the JCF HashSet. Unlike HashMap,this difference holds in the memory allocation as well, whereJCF allocates twice as much memory in the populate and fivetimes as much memory in the copy scenario.

6. DISCUSSION

6.1 Guideline for Collection ReplacementEach additional alternative collection implementation used

in a software project increases its complexity and mainte-nance effort. Consequently, developers should consider re-placing collection implementations only if such a change willresult in a significant benefit. In order to support program-mers in such decisions, we present a guideline (Table 5)showing the potential benefits and drawbacks of using alter-native implementations for several relevant scenarios and/oroptimization objectives.

We illustrate the effect of JCF collection replacement onexecution time for collections with one thousand (small) andone million elements (large). Note that in case of objectcollections the memory savings come solely from a reducedoverhead as the elements themselves are not modified. Incase of the primitive collections the programmer must re-place the object element type by its respective primitive.However, the impact on memory savings is higher as thememory footprint of elements is reduced as well.

Table 5 presents seven recommendations for replacementsleading to a substantial improvements. RecommendationR1 describes the replacement of a LinkedList by an Ar-rayList from one of three libraries, each offering a consis-tent improvement in time and memory. In R2 we recom-mend Koloboke’s alternative to JCF HashSet, as it providesa substantial improvement on execution time and memoryoverhead. For HashMaps it is possible to aim for a mem-ory overhead reduction with a small execution time penalty(R3), or to reduce memory and improve the execution timeof copy operations (R4).

An important result is the universal superiority of primi-tive collections. It is possible to considerably improve bothmemory footprint (overhead + elements) and execution timefor all three major collections: ArrayList (R5), HashSet (R6)and HashMap (R7). Note that in all cases the replacementis particularly beneficial for large collections as the savingsof memory and execution time are, in general, bigger.

6.2 Reasons for Performance DifferencesOur investigation of the source code of collections revealed

some implementation patterns responsible for performance

397

Table 5: A guideline showing the impact of replacing JCF collections with alternative collection implemen-tations for some relevant scenarios/optimization objectives. For primitive collections, the memory savingsinclude overhead reduction as well as smaller element footprint (results assume replacing Integer objects bythe primitive int).

Overhead Speedup/SlowdownSavings 1k elements 1M elements

To reduce JCF LinkedList overhead and improve time performance

R1) Replace it by JCF/Fasutils/GSCollections ArrayList84%

12x

+2x contains

+4x remove

+12x/+3x/+13x copy12x 12x

+3x contains

+2x remove

+4x/+2x+4x copy12x

To reduce JCF HashSet overhead and improve time performance

R2) Replace it by Koloboke HashSet88%

12x

+1.5x iterate

-1.5x remove

+16x copy12x 12x

+3x iterate

+1.5x contains

+51x copy12x

To reduce JCF HashMap overhead with smallest time penalty

R3) Replace it by GSCollections/Fastutils HashMap78%

-1.5x populate

-1.5x/-3x iterate

-1.5x copy -1.5x copy

To reduce JCF HashMap overhead and improve copy performance

R4) Replace it by Koloboke HashMap78%

+6x copy

-6x populate

-11x remove

+11x copy

-2x populate

-4x remove

Footprint Speedup/SlowdownSavings 1k elements 1M elements

To reduce JCF collections footprint and improve time performance

R5) Replace ArrayList by GSCollections primitive-collection 60%

12x

+3x populate

+2x contains

+2x remove12x 12x

+2x populate

+4x contains

+2x remove12x

R6) Replace HashMap by GSCollections/Koloboke primitive-collection76%

12x

+1.5x/2x populate

-2x iterate

+2x remove12x 12x

+2x populate

-2x iterate

+6x/+4x copy12x

R7) Replace HashSet by GSCollections/Koloboke primitive-collection84%

12x

+2x populate

+2x/+1.5x iterate

+27x/7x copy12x 12x

+4x populate

+4x/+3x iterate

+33x/+7x copy12x

differences in collections. We discuss these in the following.

Distinct API calls. The implementations often differ in us-age of the API call for copying an array. Here two meth-ods are used: System.arraycopy() and Arrays.copyOf().The latter is just a wrapper for System.arraycopy() whichrequires only the original array, as opposed to the Systemversion, where the target must be passed as a parameter.Our experiments showed that System.arraycopy() is 25%faster than Arrays.copyOf() for arrays up to 1 million el-ements. This is one of the reasons why GSCollections Ar-rayList (which calls System.arraycopy()) is faster than JCFcounterpart, since populate scenario is dominated by thebuffer expansion cost.

Add copy versus Memory Copy. Collections copy is thescenario with the largest discrepancy in performance of ourexperiments. We found that libraries implement two mainapproaches: they either add all elements one by one to anew created collection instance, or they perform a memorycopy. Needless to say, the memory copy is faster as addingelements one by one has the burden of manipulating objectsindividually. This explains why Koloboke executes up to 50xfaster than JCF when copying a set with 1 million elements.

Nevertheless, libraries often opt for adding the elementsinto the new instance for the simplicity of the workflow. Thecopy constructor receives a collection reference as a parame-ter and must be able to polymorphically handle all kinds of

collection types. A memory copy is restricted only to copiesfrom the same type. GSCollections and JCF use memorycopy for ArrayLists but rely on addAll for HashSet andHashMap. Koloboke is the only library that use memorycopy for HashSet/HashMap. It is important to address thatcopy is called 11% of the HashSet instantiations. Thereforeapplications can strongly benefit from memory copy imple-mentation.

Sub-optimal primitive API. Iterating elements of FastList’sArrayList primitive collection is slower than JCF ArrayList,and far more time-consuming than any other primitive im-plementation. To understand this, note that each primitivelibrary provides a forEach(IntProcedure) method to iter-ate and process the list. However, FastList provides a forE-

ach() method defined by JCF AbstractCollection, whichaccepts an Object and implicitly converts each primitiveto its wrapper, degrading the performance by a factor of5x. This case illustrates the complexity of a collection API,which often provides multiple ways of performing the sametask, but with distinct performance costs.

Sub-optimal loop implementation. The contains() methodof Trove primitive ArrayList is 3 times slower than any otherprimitive implementation (see Fig 9a). The code inspectionreveals that contrary to other primitive alternatives, Trovedefined the array loop using an unconventional for loop,namely for ( int i = offset; i- > 0; ). This loop, al-

398

beit correct, produces three times more branches and 30%more branch misses than a backwards loop defined with thedecrement in third position of the for-statement. In fact,after refactoring the contains()-method in an obvious wayin our own extension of Trove’s primitive ArrayList the per-formance bottleneck was fixed.

6.3 Threats to ValidityThere are several issues that may affect the validity of

our work. As a benchmark study, our results are subjectto a number of external factors. Even though we mitigatetheir influence with our methodology (see Section 4), thehardware specs may have an impact on collections time per-formance [11]. In particular, the speedup/slowdown indica-tors might be biased towards our test machine configuration.The memory footprint results, however, can be generalizedbeyond the hardware specification.

We try to reuse the same code to provide a homogeneousbenchmark for the different collection libraries. Some im-plementations are not JCF compliant though, e.g. HPPCimplementations and most primitive collections. For thosecases we carefully adapt our code to their respective API,introducing only subtle code differences. Still, this may im-pact time performance of evaluated collections.

Lastly, we do not consider workloads with a convolutedset of operations. For such workloads, the memory overheadand execution time can be impacted by complex interactionsamong operations. Such workloads are harder to generalizethough. That is why we based our micro-benchmark onsingle operations so that a useful guideline can be derivedfrom the results.

7. RELATED WORKThere is a substantial body of research devoted to under-

standing the impact of collections on code performance.

Empirical Studies of Java Collections. To the best ofour knowledge our evaluation is the most comprehensivestudy of the space and time profiles of alternative collec-tion implementations in Java. However, the benchmarks weevaluated partially cover some aspects featured in our ex-periments. In [25] the author evaluates different HashMapslibraries in more convoluted scenarios. Lewis [15] comparesJCF implementations and suggests a new collection for aspecific GUI update scenario. Another interesting paper [14]exemplifies the time and memory trade-offs of HashMaps.Our work differs from these in some key aspects. First, weevaluate a wider range of implementations, presenting theperformance profile of lists, sets and maps, including primi-tive alternatives, while those works focus more commonly onHashMap alternatives. Second, we based our experimentalplanning on patterns found on real code (Section 3). Thisallows us to evaluate scenarios more commonly adopted bydevelopers in real applications, and provide more representa-tive results. Lastly, we investigate and report some patternsin real code that explain some performance differences, anaspect rarely explored in the benchmarks we have found.

More recently, some works have studied the collections se-lection problem in relation to energy consumption of appli-cations [10]. In particular, Hasam [9] presented an inspiringwork which experimentally explores the impact of collectionson energy consumption and concludes that some implemen-tations can increase the energy usage of an application up

to 300%. Their work does not consider time/memory pro-file as we do, and we believe that time, memory and energyconsumption profiles can be combined to give programmersa more complete view of collections performance.

Collection inefficiencies. Several authors have designedmethods to detect collection inefficiencies. Xu et al, [28]use both static and dynamic analysis to detect overpopu-lated and underutilized collections. Yang et al, [29] tracksand records the containers and the flow of its elements,and uses an offline analysis to pinpoint some inefficiencies.Those works target performance problems related to collec-tion without exploring the effect of different collection types.Thus the authors attempt to solve an orthogonal problemto collection selection.

Performance optimization by collection replacement.Recently, several studies have attempted to advise develop-ers on selecting collection abstraction type which improvesthe performance in a context-dependent way. Shacham etal., [23] optimizes applications by monitoring and replacingthe collections types using a user-defined model. Coco [26]also utilizes a user defined model to replace the collectionsat runtime. Both works differ from our approach as theydeploy solely asymptotic analysis to select the most suitablecollection type, without accounting for the real performanceprofile. Consequently, the authors do not consider alterna-tive collection libraries, and focus only on JCF.

The approach of Brainy [11] is most closely related to ourwork. Brainy is a C++ framework which generates a largeamount of synthetic applications, models their performancevia machine learning, and suggests appropriate collectionreplacements. This framework does not consider alterna-tive implementations, but rather, focuses on a set of specificcross-abstraction transformations which might be difficult toapply to real code. Our work targets understanding the per-formance of alternative libraries, and we provide a guidelinefor replacing collection implementations for scenarios with ahigh impact on performance.

8. CONCLUSIONS AND FUTURE WORKWe analyzed usage patterns of collections by mining a

code corpus of 10,986 Java projects and found that evenin high-quality code, the usage of alternative collections li-braries is rare. To investigate the potential for performanceimprovement in Java applications with only moderate pro-gramming effort, we have conducted a rigorous performanceevaluation study of the standard Java Collection Frameworkand six most popular alternative collection libraries. Wefound that such alternative libraries can offer a programmera significant reduction of both execution time and memoryconsumption.

We summarized the impact of collection replacement onruntime improvements and memory savings for some rele-vant scenarios. The resulting guideline assists programmersin deciding whether in a given setting an alternative imple-mentation can offer a substantial performance improvement,and if so, which implementation should be used. Some ofthese replacements are easy to automate and can improvethe performance profile of collections through simple coderefactoring.

As future work, we target implementation of such auto-

399

mated refactoring tools. We also intend to evaluate ourrecommendations in more complex scenarios, including astudy of their performance impact in large, mature softwareprojects.

9. REFERENCES[1] M. Allamanis and C. Sutton. Mining source code

repositories at massive scale using language modeling.In Proceedings of the 10th Working Conference onMining Software Repositories, MSR ’13, pages207–216, Piscataway, NJ, USA, 2013. IEEE Press.

[2] Chronicle. Koloboke. http://chronicle.software/products/koloboke-collections/,Oct. 2015.

[3] A. Georges, D. Buytaert, and L. Eeckhout.Statistically rigorous java performance evaluation.SIGPLAN Not., 42(10):57–76, Oct. 2007.

[4] A. Georges, L. Eeckhout, and D. Buytaert. Javaperformance evaluation through rigorous replaycompilation. SIGPLAN Not., 43(10):367–384, Oct.2008.

[5] J. Y. Gil and Y. Shimron. Smaller footprint for javacollections. In Proceedings of the ACM InternationalConference Companion on Object OrientedProgramming Systems Languages and ApplicationsCompanion, OOPSLA ’11, pages 191–192, New York,NY, USA, 2011. ACM.

[6] GNU Trove. Trove.http://trove.starlight-systems.com/, May 2015.

[7] Goldman Sachs Group, Inc. GS Collections.https://github.com/goldmansachs/gs-collections, June2015.

[8] Google. Guava. https://github.com/google/guava,Aug. 2014.

[9] S. Hasan, Z. King, M. Hafiz, M. Sayagh, B. Adams,and A. Hindle. Energy profiles of java collectionsclasses. In Proceedings of the 38th InternationalConference on Software Engineering, ICSE ’16, pages225–236, New York, NY, USA, 2016. ACM.

[10] N. Hunt, P. S. Sandhu, and L. Ceze. Characterizingthe performance and energy efficiency of lock-free datastructures. In 2011 15th Workshop on Interactionbetween Compilers and Computer Architectures, pages63–70, Feb 2011.

[11] C. Jung, S. Rus, B. P. Railing, N. Clark, andS. Pande. Brainy: Effective selection of datastructures. SIGPLAN Not., 46(6):86–97, June 2011.

[12] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer,D. M. German, and D. Damian. The promises andperils of mining github. In Proceedings of the 11thWorking Conference on Mining Software Repositories,MSR 2014, pages 92–101, New York, NY, USA, 2014.ACM.

[13] K. Kawachiya, K. Ogata, and T. Onodera. Analysisand reduction of memory inefficiencies in java strings.SIGPLAN Not., 43(10):385–402, Oct. 2008.

[14] R. Leventov. Time - Memory Tradeoff With theExample of Java Maps. https://dzone.com/articles/time-memory-tradeoff-example.

[15] L. Lewis. Java Collection Performance. https://dzone.com/articles/java-collection-performance,July 2011.

[16] N. Mitchell and G. Sevitsky. The causes of bloat, thelimits of health. In Proceedings of the 22Nd AnnualACM SIGPLAN Conference on Object-orientedProgramming Systems and Applications, OOPSLA ’07,pages 245–260, New York, NY, USA, 2007. ACM.

[17] D. C. Montgomery. Design and Analysis ofExperiments. Wiley, Hoboken, NJ, 8 edition edition,Apr. 2012.

[18] T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F.Sweeney. Evaluating the accuracy of java profilers. InProceedings of the 31st ACM SIGPLAN Conferenceon Programming Language Design andImplementation, PLDI ’10, pages 187–197, New York,NY, USA, 2010. ACM.

[19] H. A. Nguyen, R. Dyer, T. N. Nguyen, and H. Rajan.Mining preconditions of apis in large-scale codecorpus. In Proceedings of the 22Nd ACM SIGSOFTInternational Symposium on Foundations of SoftwareEngineering, FSE 2014, pages 166–177, New York,NY, USA, 2014. ACM.

[20] Oracle. Java development kit.https://www.oracle.com/java/index.html, Sept. 2015.

[21] S. Osinski and D. Weiss. HPPC: High PerformancePrimitive Collections for Java.http://labs.carrotsearch.com/hppc.html, Jan. 2015.

[22] B. Ray, D. Posnett, V. Filkov, and P. Devanbu. Alarge scale study of programming languages and codequality in github. In Proceedings of the 22Nd ACMSIGSOFT International Symposium on Foundationsof Software Engineering, FSE 2014, pages 155–165,New York, NY, USA, 2014. ACM.

[23] O. Shacham, M. Vechev, and E. Yahav. Chameleon:Adaptive selection of collections. In Proceedings of the30th ACM SIGPLAN Conference on ProgrammingLanguage Design and Implementation, PLDI ’09,pages 408–418, New York, NY, USA, 2009. ACM.

[24] S. Vigna. Fastutil. http://fastutil.di.unimi.it/l, Jan.2016.

[25] M. Vorontsov. Large HashMap Overview. http://java-performance.info/hashmap-overview-jdk-fastutil-goldman-sachs-hppc-koloboke-trove-january-2015/,Feb. 2015.

[26] G. Xu. Coco: Sound and adaptive replacement of javacollections. In Proceedings of the 27th EuropeanConference on Object-Oriented Programming,ECOOP’13, pages 1–26, Berlin, Heidelberg, 2013.Springer-Verlag.

[27] G. Xu and A. Rountev. Precise memory leak detectionfor java software using container profiling. InProceedings of the 30th International Conference onSoftware Engineering, ICSE ’08, pages 151–160, NewYork, NY, USA, 2008. ACM.

[28] G. Xu and A. Rountev. Detecting inefficiently-usedcontainers to avoid bloat. In Proceedings of the 31stACM SIGPLAN Conference on ProgrammingLanguage Design and Implementation, PLDI ’10,pages 160–173, New York, NY, USA, 2010. ACM.

[29] S. Yang, D. Yan, G. Xu, and A. Rountev. Dynamicanalysis of inefficiently-used containers. In Proceedingsof the Ninth International Workshop on DynamicAnalysis, WODA 2012, pages 30–35, New York, NY,USA, 2012. ACM.

400