Corey: An Operating System for Many Cores

Silas Boyd-Wickizer∗ Haibo Chen† Rong Chen† Yandong Mao†Frans Kaashoek∗ Robert Morris∗ Aleksey Pesterev∗ Lex Stein‡ Ming Wu‡

Yuehua Dai◦ Yang Zhang∗ Zheng Zhang‡∗MIT †Fudan University ‡Microsoft Research Asia ◦Xi’an Jiaotong University

ABSTRACTMultiprocessor application performance can be limitedby the operating system when the application uses theoperating system frequently and the operating systemservices use data structures shared and modified by mul-tiple processing cores. If the application does not needthe sharing, then the operating system will become anunnecessary bottleneck to the application’s performance.

This paper argues that applications should controlsharing: the kernel should arrange each data structureso that only a single processor need update it, unlessdirected otherwise by the application. Guided by thisdesign principle, this paper proposes three operatingsystem abstractions (address ranges, kernel cores, andshares) that allow applications to control inter-core shar-ing and to take advantage of the likely abundance ofcores by dedicating cores to specific operating systemfunctions.

Measurements of microbenchmarks on the Corey pro-totype operating system, which embodies the new ab-stractions, show how control over sharing can improveperformance. Application benchmarks, using MapRe-duce and a Web server, show that the improvements canbe significant for overall performance: MapReduce onCorey performs 25% faster than on Linux when using16 cores. Hardware event counters confirm that theseimprovements are due to avoiding operations that are ex-pensive on multicore machines.

1 INTRODUCTIONCache-coherent shared-memory multiprocessor hard-ware has become the default in modern PCs as chip man-ufacturers have adopted multicore architectures. Chipswith four cores are common, and trends suggest thatchips with tens to hundreds of cores will appear withinfive years [2]. This paper explores new operating systemabstractions that allow applications to avoid bottlenecksin the operating system as the number of cores increases.

Operating system services whose performance scalespoorly with the number of cores can dominate applica-tion performance. Gough et al. show that contention forLinux’s scheduling queues can contribute significantly to

total application run time on two cores [12]. Veal andFoong show that as a Linux Web server uses more coresdirectory lookups spend increasing amounts of time con-tending for spin locks [29]. Section 8.5.1 shows thatcontention for Linux address space data structures causesthe percentage of total time spent in the reduce phase ofa MapReduce application to increase from 5% at sevencores to almost 30% at 16 cores.

One source of poorly scaling operating system ser-vices is use of data structures modified by multiple cores.Figure 1 illustrates such a scalability problem with a sim-ple microbenchmark. The benchmark creates a numberof threads within a process, each thread creates a file de-scriptor, and then each thread repeatedly duplicates (withdup) its file descriptor and closes the result. The graphshows results on a machine with four quad-core AMDOpteron chips running Linux 2.6.25. Figure 1 showsthat, as the number of cores increases, the total number ofdup and close operations per unit time decreases. Thecause is contention over shared data: the table describ-ing the process’s open files. With one core there are nocache misses, and the benchmark is fast; with two cores,the cache coherence protocol forces a few cache missesper iteration to exchange the lock and table data. Moregenerally, only one thread at a time can update the sharedfile descriptor table (which prevents any increase in per-formance), and the increasing number of threads spin-ning for the lock gradually increases locking costs. Thisproblem is not specific to Linux, but is due to POSIX se-mantics, which require that a new file descriptor be vis-ible to all of a process’s threads even if only one threaduses it.

Common approaches to increasing scalability includeavoiding shared data structures altogether, or designingthem to allow concurrent access via fine-grained lockingor wait-free primitives. For example, the Linux com-munity has made tremendous progress with these ap-proaches [18].

A different approach exploits the fact that some in-stances of a given resource type need to be shared, whileothers do not. If the operating system were aware of anapplication’s sharing requirements, it could choose re-source implementations suited to those requirements. A

0

500

1000

1500

2000

2500

3000

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

10

00

s o

f d

up

+ c

lose

per

sec

on

d

Cores

Figure 1: Throughput of the file descriptor dup and close mi-crobenchmark on Linux.

limiting factor in this approach is that the operating sys-tem interface often does not convey the needed informa-tion about sharing requirements. In the file descriptorexample above, it would be helpful if the thread couldindicate whether the new descriptor is private or usableby other threads. In the former case it could be createdin a per-thread table without contention.

This paper is guided by a principle that generalizesthe above observation: applications should control shar-ing of operating system data structures. “Application”is meant in a broad sense: the entity that has enoughinformation to make the sharing decision, whether thatbe an operating system service, an application-level li-brary, or a traditional user-level application. This prin-ciple has two implications. First, the operating systemshould arrange each of its data structures so that by de-fault only one core needs to use it, to avoid forcingunwanted sharing. Second, the operating system inter-faces should let callers control how the operating systemshares data structures between cores. The intended re-sult is that the operating system incurs sharing costs (e.g.,cache misses due to memory coherence) only when theapplication designer has decided that is the best plan.

This paper introduces three new abstractions for ap-plications to control sharing within the operating system.Address ranges allow applications to control which partsof the address space are private per core and which areshared; manipulating private regions involves no con-tention or inter-core TLB invalidations, while explicit in-dication of shared regions allows sharing of hardwarepage tables and consequent minimization of soft pagefaults (page faults that occur when a core referencespages with no mappings in the hardware page table butwhich are present in physical memory). Kernel cores al-low applications to dedicate cores to run specific kernelfunctions, avoiding contention over the data those func-tions use. Shares are lookup tables for kernel objectsthat allow applications to control which object identifiersare visible to other cores. These abstractions are imple-

mentable without inter-core sharing by default, but allowsharing among cores as directed by applications.

We have implemented these abstractions in a proto-type operating system called Corey. Corey is organizedlike an exokernel [9] to ease user-space prototyping ofexperimental mechanisms. The Corey kernel providesthe above three abstractions. Most higher-level servicesare implemented as library operating systems that use theabstractions to control sharing. Corey runs on machineswith AMD Opteron and Intel Xeon processors.

Several benchmarks demonstrate the benefits of thenew abstractions. For example, a MapReduce applica-tion scales better with address ranges on Corey than onLinux. A benchmark that creates and closes many shortTCP connections shows that Corey can saturate the net-work device with five cores by dedicating a kernel coreto manipulating the device driver state, while 11 coresare required when not using a dedicated kernel core. Asynthetic Web benchmark shows that Web applicationscan also benefit from dedicating cores to data.

This paper should be viewed as making a case for con-trolling sharing rather than demonstrating any absoluteconclusions. Corey is an incomplete prototype, and thusit may not be fair to compare it to full-featured operatingsystems. In addition, changes in the architecture of fu-ture multicore processors may change the attractivenessof the ideas presented here.

The rest of the paper is organized as follows. Usingarchitecture-level microbenchmarks, Section 2 measuresthe cost of sharing on multicore processors. Section 3describes the three proposed abstractions to control shar-ing. Section 4 presents the Corey kernel and Section 5its operating system services. Section 6 summarizes theextensions to the default system services that implementMapReduce and Web server applications efficiently. Sec-tion 7 summarizes the implementation of Corey. Sec-tion 8 presents performance results. Section 9 reflects onour results so far and outlines directions for future work.Section 10 relates Corey to previous work. Section 11summarizes our conclusions.

2 MULTICORE CHALLENGESThe main goal of Corey is to allow applications to scalewell with the number of cores. This section details somehardware obstacles to achieving that goal.

Future multicore chips are likely to have large totalamounts of on-chip cache, but split up among the manycores. Performance may be greatly affected by how wellsoftware exploits the caches and the interconnect be-tween cores. For example, using data in a different core’scache is likely to be faster than fetching data from mem-ory, and using data from a nearby core’s cache may befaster than using data from a core that is far away on theinterconnect.

3/6.77 121/1.27

14/4.44 50/3.27

273/1.46

282/0.84

L3

C1 C2 C3C0

L1

L2DRAM

201/0.86

DRAM

DRAM

DRAM

255/1.75

327/1.33

Figure 2: The AMD 16-core system topology. Memory access latencyis in cycles and listed before the backslash. Memory bandwidth is inbytes per cycle and listed after the backslash. The measurements reflectthe latency and bandwidth achieved by a core issuing load instructions.The measurements for accessing the L1 or L2 caches of a different coreon the same chip are the same. The measurements for accessing anycache on a different chip are the same. Each cache line is 64 bytes, L1caches are 64 Kbytes 8-way set associative, L2 caches are 512 Kbytes16-way set associative, and L3 caches are 2 Mbytes 32-way set asso-ciative.

These properties can already be observed in currentmulticore machines. Figure 2 summarizes the memorysystem of a 16-core machine from AMD. The machinehas four quad-core Opteron processors connected by asquare interconnect. The interconnect carries data be-tween cores and memory, as well as cache coherencebroadcasts to locate and invalidate cache lines, and point-to-point cache coherence transfers of individual cachelines. Each core has a private L1 and L2 cache. Fourcores on the same chip share an L3 cache. Cores on onechip are connected by an internal crossbar switch and canaccess each others’ L1 and L2 caches efficiently. Mem-ory is partitioned in four banks, each connected to oneof the four chips. Each core has a clock frequency of2.0 GHz.

Figure 2 also shows the cost of loading a cache linefrom a local core, a nearby core, and a distant core. Wemeasured these numbers using many of the techniquesdocumented by Yotov et al. [31]. The techniques avoidinterference from the operating system and hardware fea-tures such as cache line prefetching.

Reading from the local L3 cache on an AMD chip isfaster than reading from the cache of a different core onthe same chip. Inter-chip reads are slower, particularlywhen they go through two interconnect hops.

Figure 3 shows the performance scalability of locking,an important primitive often dominated by cache misscosts. The graph compares Linux’s kernel spin locks and

0

500

1000

1500

2000

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

nan

ose

con

ds

per

acq

uir

e +

rel

ease

Cores

Spin lockMCS lock

Figure 3: Time required to acquire and release a lock on a 16-coreAMD machine when varying number of cores contend for the lock.The two lines show Linux kernel spin locks and MCS locks (on Corey).A spin lock with one core takes about 11 nanoseconds; an MCS lockabout 26 nanoseconds.

scalable MCS locks [21] (on Corey) on the AMD ma-chine, varying the number of cores contending for thelock. For both the kernel spin lock and the MCS lockwe measured the time required to acquire and release asingle lock. When only one core uses a lock, both typesof lock are fast because the lock is always in the core’scache; spin locks are faster than MCS locks because theformer requires three instructions to acquire and releasewhen not contended, while the latter requires 15. Thetime for each successful acquire increases for the spinlock with additional cores because each new core adds afew more cache coherence broadcasts per acquire, in or-der for the new core to observe the effects of other cores’acquires and releases. MCS locks avoid this cost by hav-ing each core spin on a separate location.

We measured the average times required by Linux2.6.25 to flush the TLBs of all cores after a change to ashared page table. These “TLB shootdowns” are consid-erably slower on 16 cores than on 2 (18742 cycles versus5092 cycles). TLB shootdown typically dominates thecost of removing a mapping from an address space thatis shared among multiple cores.

On AMD 16-core systems, the speed difference be-tween a fast memory access and a slow memory accessis a factor of 100, and that factor is likely to grow with thenumber of cores. Our measurements for the Intel Xeonshow a similar speed difference. For performance, ker-nels must mainly access data in the local core’s cache.A contended or falsely shared cache line decreases per-formance because many accesses are to remote caches.Widely-shared and contended locks also decrease perfor-mance, even if the critical sections they protect are only afew instructions. These performance effects will be moredamaging as the number of cores increases, since the costof accessing far-away caches increases with the numberof cores.

core a core b

stack a stack b results a results b

address space

(a) A single address space.

core a core b

stack a stack bresults a results b

address space a address space b

(b) Separate address spaces.

core a core b

stack a stack b

results bresults a

root address range a

shared address range a shared address range b

root address range b

(c) Two address spaces with shared result mappings.

Figure 4: Example address space configurations for MapReduce exe-cuting on two cores. Lines represent mappings. In this example a stackis one page and results are three pages.

3 DESIGNExisting operating system abstractions are often difficultto implement without sharing kernel data among cores,regardless of whether the application needs the shared se-mantics. The resulting unnecessary sharing and resultingcontention can limit application scalability. This sectiongives three examples of unnecessary sharing and for eachexample introduces a new abstraction that allows the ap-plication to decide if and how to share. The intent is thatthese abstractions will help applications to scale to largenumbers of cores.

3.1 Address rangesParallel applications typically use memory in a mixtureof two sharing patterns: memory that is used on justone core (private), and memory that multiple cores use(shared). Most operating systems give the applicationa choice between two overall memory configurations: asingle address space shared by all cores or a separateaddress space per core. The term address space hererefers to the description of how virtual addresses mapto memory, typically defined by kernel data structuresand instantiated lazily (in response to soft page faults) inhardware-defined page tables or TLBs. If an application

chooses to harness concurrency using threads, the resultis typically a single address space shared by all threads.If an application obtains concurrency by forking multipleprocesses, the result is typically a private address spaceper process; the processes can then map shared segmentsinto all address spaces. The problem is that each of thesetwo configurations works well for only one of the sharingpatterns, placing applications with a mixture of patternsin a bind.

As an example, consider a MapReduce application [8].During the map phase, each core reads part of the appli-cation’s input and produces intermediate results; map oneach core writes its intermediate results to a different areaof memory. Each map instance adds pages to its addressspace as it generates intermediate results. During the re-duce phase each core reads intermediate results producedby multiple map instances to produce the output.

For MapReduce, a single address-space (see Fig-ure 4(a)) and separate per-core address-spaces (see Fig-ure 4(b)) incur different costs. With a single addressspace, the map phase causes contention as all cores addmappings to the kernel’s address space data structures.On the other hand, a single address space is efficientfor reduce because once any core inserts a mapping intothe underlying hardware page table, all cores can use themapping without soft page faults. With separate addressspaces, the map phase sees no contention while addingmappings to the per-core address spaces. However, thereduce phase will incur a soft page fault per core per pageof accessed intermediate results. Neither memory con-figuration works well for the entire application.

We propose address ranges to give applications highperformance for both private and shared memory (seeFigure 4(c)). An address range is a kernel-providedabstraction that corresponds to a range of virtual-to-physical mappings. An application can allocate addressranges, insert mappings into them, and place an addressrange at a desired spot in the address space. If multi-ple cores’ address spaces incorporate the same addressrange, then they will share the corresponding pieces ofhardware page tables, and see mappings inserted by eachothers’ soft page faults. A core can update the mappingsin a non-shared address range without contention. Evenwhen shared, the address range is the unit of locking; ifonly one core manipulates the mappings in a shared ad-dress range, there will be no contention. Finally, deletionof mappings from a non-shared address range does notrequire TLB shootdowns.

Address ranges allow applications to selectively shareparts of address spaces, instead of being forced to makean all-or-nothing decision. For example, the MapReduceruntime can set up address spaces as shown in 4(c). Eachcore has a private root address range that maps all privatememory segments used for stacks and temporary objects

and several shared address ranges mapped by all othercores. A core can manipulate mappings in its private ad-dress ranges without contention or TLB shootdowns. Ifeach core uses a different shared address range to storethe intermediate output of its map phase (as shown Fig-ure 4(c)), the map phases do not contend when addingmappings. During the reduce phase there are no soft pagefaults when accessing shared segments, since all coresshare the corresponding parts of the hardware page ta-bles.

3.2 Kernel coresIn most operating systems, when application code on acore invokes a system call, the same core executes thekernel code for the call. If the system call uses sharedkernel data structures, it acquires locks and fetches rel-evant cache lines from the last core to use the data.The cache line fetches and lock acquisitions are costlyif many cores use the same shared kernel data. If theshared data is large, it may end up replicated in manycaches, potentially reducing total effective cache spaceand increasing DRAM references.

We propose a kernel core abstraction that allows ap-plications to dedicate cores to kernel functions and data.A kernel core can manage hardware devices and exe-cute system calls sent from other cores. For example,a Web service application may dedicate a core to inter-acting with the network device instead of having all coresmanipulate and contend for driver and device data struc-tures (e.g., transmit and receive DMA descriptors). Mul-tiple application cores then communicate with the kernelcore via shared-memory IPC; the application cores ex-change packet buffers with the kernel core, and the ker-nel core manipulates the network hardware to transmitand receive the packets.

This plan reduces the number of cores available to theWeb application, but it may improve overall performanceby reducing contention for driver data structures and as-sociated locks. Whether a net performance improvementwould result is something that the operating system can-not easily predict. Corey provides the kernel core ab-straction so that applications can make the decision.

3.3 SharesMany kernel operations involve looking up identifiers intables to yield a pointer to the relevant kernel data struc-ture; file descriptors and process IDs are examples ofsuch identifiers. Use of these tables can be costly whenmultiple cores contend for locks on the tables and on thetable entries themselves.

For each kind of lookup, the operating system inter-face designer or implementer typically decides the scopeof sharing of the corresponding identifiers and tables.For example, Unix file descriptors are shared among thethreads of a process. Process identifiers, on the other

hand, typically have global significance. Common im-plementations use per-process and global lookup tables,respectively. If a particular identifier used by an applica-tion needs only limited scope, but the operating systemimplementation uses a more global lookup scheme, theresult may be needless contention over the lookup datastructures.

We propose a share abstraction that allows applica-tions to dynamically create lookup tables and determinehow these tables are shared. Each of an application’scores starts with one share (its root share), which is pri-vate to that core. If two cores want to share a share, theycreate a share and add the share’s ID to their private rootshare (or to a share reachable from their root share). Aroot share doesn’t use a lock because it is private, but ashared share does. An application can decide for eachnew kernel object (including a new share) which sharewill hold the identifier.

Inside the kernel, a share maps application-visibleidentifiers to kernel data pointers. The shares reachablefrom a core’s root share define which identifiers the corecan use. Contention may arise when two cores manipu-late the same share, but applications can avoid such con-tention by placing identifiers with limited sharing scopein shares that are only reachable on a subset of the cores.

For example, shares could be used to implement file-descriptor-like references to kernel objects. If only onethread uses a descriptor, it can place the descriptor in itscore’s private root share. If the descriptor is shared be-tween two threads, these two threads can create a sharethat holds the descriptor. If the descriptor is sharedamong all threads of a process, the file descriptor canbe put in a per-process share. The advantage of sharesis that the application can limit sharing of lookup tablesand avoid unnecessary contention if a kernel object is notshared. The downside is that an application must oftenkeep track of the share in which it has placed each iden-tifier.

4 COREY KERNEL

Corey provides a kernel interface organized around fivetypes of low-level objects: shares, segments, addressranges, pcores, and devices. Library operating sys-tems provide higher-level system services that are imple-mented on top of these five types. Applications imple-ment domain specific optimizations using the low-levelinterface; for example, using pcores and devices they canimplement kernel cores. Figure 5 provides an overviewof the Corey low-level interface. The following sectionsdescribe the design of the five types of Corey kernel ob-jects and how they allow applications to control sharing.Section 5 describes Corey’s higher-level system services.

System call Descriptionname obj get name(obj) return the name of an objectshareid share alloc(shareid, name, memid) allocate a share objectvoid share addobj(shareid, obj) add a reference to a shared object to the specified sharevoid share delobj(obj) remove an object from a share, decrementing its reference countvoid self drop(shareid) drop current core’s reference to a sharesegid segment alloc(shareid, name, memid) allocate physical memory and return a segment object for itsegid segment copy(shareid, seg, name, mode) copy a segment, optionally with copy-on-write or -readnbytes segment get nbytes(seg) get the size of a segmentvoid segment set nbytes(seg, nbytes) set the size of a segmentarid ar alloc(shareid, name, memid) allocate an address range objectvoid ar set seg(ar, voff, segid, soff, len) map addresses at voff in ar to a segment’s physical pagesvoid ar set ar(ar, voff, ar1, aoff, len) map addresses at voff in ar to address range ar1ar mappings ar get(ar) return the address mappings for a given address rangepcoreid pcore alloc(shareid, name, memid) allocate a physical core objectpcore pcore current(void) return a reference to the object for current pcorevoid pcore run(pcore, context) run the specified user contextvoid pcore add device(pcore, dev) specify device list to a kernel corevoid pcore set interval(pcore, hz) set the time slice periodvoid pcore halt(pcore) halt the pcoredevid device alloc(shareid, hwid, memid) allocate the specified device and return a device objectdev list device list(void) return the list of devicesdev stat device stat(dev) return information about the devicevoid device conf(dev, dev conf) configure a devicevoid device buf(dev, seg, offset, buf type) feed a segment to the device objectlocality matrix locality get(void) get hardware locality information

Figure 5: Corey system calls. shareid, segid, arid, pcoreid, and devid represent 64-bit object IDs. share, seg, ar, pcore, obj, and dev represent〈share ID, object ID〉 pairs. hwid represents a unique ID for hardware devices and memid represents a unique ID for per-core free page lists.

4.1 Object metadataThe kernel maintains metadata describing each object.To reduce the cost of allocating memory for object meta-data, each core keeps a local free page list. If the archi-tecture is NUMA, a core’s free page list holds pages fromits local memory node. The system call interface allowsthe caller to indicate which core’s free list a new object’smemory should be taken from. Kernels on all cores canaddress all object metadata since each kernel maps all ofphysical memory into its address space.

The Corey kernel generally locks an object’s metadatabefore each use. If the application has arranged things sothat the object is only used on one core, the lock and useof the metadata will be fast (see Figure 3), assuming theyhave not been evicted from the core’s cache. Corey usesspin lock and read-write lock implementations borrowedfrom Linux, its own MCS lock implementation, and ascalable read-write lock implementation inspired by theMCS lock to synchronize access to object metadata.

4.2 Object namingAn application allocates a Corey object by calling thecorresponding alloc system call, which returns aunique 64-bit object ID. In order for a kernel to use anobject, it must know the object ID (usually from a sys-tem call argument), and it must map the ID to the addressof the object’s metadata. Corey uses shares for this pur-

pose. Applications specify which shares are available onwhich cores by passing a core’s share set to pcore run(see below).

When allocating an object, an application selectsa share to hold the object ID. The application uses〈share ID, object ID〉 pairs to specify objects to systemcalls. Applications can add a reference to an object in ashare with share addobj and remove an object froma share with share delobj. The kernel counts refer-ences to each object, freeing the object’s memory whenthe count is zero. By convention, applications maintain aper-core private share and one or more shared shares.

4.3 Memory managementThe kernel represents physical memory using the seg-ment abstraction. Applications use segment alloc toallocate segments and segment copy to copy a seg-ment or mark the segment as copy-on-reference or copy-on-write. By default, only the core that allocated the seg-ment can reference it; an application can arrange to sharea segment between cores by adding it to a share, as de-scribed above.

An application uses address ranges to define its ad-dress space. Each running core has an associated rootaddress range object containing a table of address map-pings. By convention, most applications allocate a rootaddress range for each core to hold core private map-

pings, such as thread stacks, and use one or more addressranges that are shared by all cores to hold shared segmentmappings, such as dynamically allocated buffers. An ap-plication uses ar set seg to cause an address range tomap addresses to the physical memory in a segment, andar set ar to set up a tree of address range mappings.

4.4 ExecutionCorey represents physical cores with pcore objects. Onceallocated, an application can start execution on a physi-cal core by invoking pcore run and specifying a pcoreobject, instruction and stack pointer, a set of shares, andan address range. A pcore executes until pcore haltis called. This interface allows Corey to space-multiplexapplications over cores, dedicating a set of cores to agiven application for a long period of time, and lettingeach application manage its own cores.

An application configures a kernel core by allocat-ing a pcore object, specifying a list of devices withpcore add device, and invoking pcore run withthe kernel core option set in the context argument. Akernel core continuously polls the specified devices byinvoking a device specific function. A kernel core pollsboth real devices and special “syscall” pseudo-devices.

A syscall device allows an application to invoke sys-tem calls on a kernel core. The application communi-cates with the syscall device via a ring buffer in a sharedmemory segment.

5 SYSTEM SERVICESThis section describes three system services exported byCorey: execution forking, network access, and a buffercache. These services together with a C standard librarythat includes support for file descriptors, dynamic mem-ory allocation, threading, and other common Unix-likefeatures create a scalable and convenient application en-vironment that is used by the applications discussed inSection 6.

5.1 cforkcfork is similar to Unix fork and is the main ab-straction used by applications to extend execution to anew core. cfork takes a physical core ID, allocatesa new pcore object and runs it. By default, cforkshares little state between the parent and child pcore. Thecaller marks most segments from its root address range ascopy-on-write and maps them into the root address rangeof the new pcore. Callers can instruct cfork to sharespecified segments and address ranges with the new pro-cessor. Applications implement fine-grained sharing us-ing shared segment mappings and more coarse-grainedsharing using shared address range mappings. cforkcallers can share kernel objects with the new pcore bypassing a set of shares for the new pcore.

5.2 NetworkApplications can choose to run several network stacks(possibly one for each core) or a single shared networkstack. Corey uses the lwIP [19] networking library. Ap-plications specify a network device for each lwIP stack.If multiple network stacks share a single physical device,Corey virtualizes the network card. Network stacks ondifferent cores that share a physical network device alsoshare the device driver data, such as the transmit descrip-tor table and receive descriptor table.

All configurations we have experimented with run aseparate network stack for each core that requires net-work access. This design provides good scalability butrequires multiple IP addresses per server and must bal-ance requests using an external mechanism. A potentialsolution is to extend the Corey virtual network driver touse ARP negotiation to balance traffic between virtualnetwork devices and network stacks (similar to the LinuxEthernet bonding driver).

5.3 Buffer cacheAn inter-core shared buffer cache is important to systemperformance and often necessary for correctness whenmultiple cores access shared files. Since cores share thebuffer cache they might contend on the data structuresused to organize cached disk blocks. Furthermore, un-der write-heavy workloads it is possible that cores willcontend for the cached disk blocks.

The Corey buffer cache resembles a traditional Unixbuffer cache; however, we found three techniques thatsubstantially improve multicore performance. The firstis a lock-free tree that allows multiple cores to locatecached blocks without contention. The second is a writescheme that tries to minimize contention on shared datausing per-core block allocators and by copying applica-tion data into blocks likely to be held in local hardwarecaches. The third uses a scalable read-write lock to en-sure blocks are not freed or reused during reads.

6 APPLICATIONSIt is unclear how big multicore systems will be used, butparallel computation and network servers are likely ap-plication areas. To evaluate Corey in these areas, we im-plemented a MapReduce system and a Web server withsynthetic dynamic content. Both applications are data-parallel but have sharing requirements and thus are nottrivially parallelized. This section describes how the twoapplications use the Corey interface to achieve high per-formance.

6.1 MapReduce applicationsMapReduce is a framework for data-parallel executionof simple programmer-supplied functions. The program-mer need not be an expert in parallel programming to

IPC IPC IPC IPC IPC IPC

appappapp

IPC

kernelcore

IPC

kernelcoreNIC NIC

webd webd

buffer cache

Figure 6: A Corey Web server configuration with two kernel cores,two webd cores and three application cores. Rectangles represent seg-ments, rounded rectangles represents pcores, and circles represent de-vices.

achieve good parallel performance. Data-parallel appli-cations fit well with the architecture of multicore ma-chines, because each core has its own private cache andcan efficiently process data in that cache. If the runtimedoes a good job of putting data in caches close to thecores that manipulate that data, performance should in-crease with the number of cores. The difficult part isthe global communication between the map and reducephases.

We started with the Phoenix MapReduce implemen-tation [22], which is optimized for shared-memory mul-tiprocessors. We reimplemented Phoenix to simplify itsimplementation, to use better algorithms for manipulat-ing the intermediate data, and to optimize its perfor-mance. We call this reimplementation Metis.

Metis on Corey exploits address ranges as described inSection 3. Metis uses a separate address space on eachcore, with private mappings for most data (e.g. local vari-ables and the input to map), so that each core can up-date its own page tables without contention. Metis usesaddress ranges to share the output of the map on eachcore with the reduce phase on other cores. This arrange-ment avoids contention as each map instance adds pagesto hold its intermediate output, and ensures that the re-duce phase incurs no soft page faults while processingintermediate data from the map phase.

6.2 Web server applicationsThe main processing in a Web server includes low-levelnetwork device handling, TCP processing, HTTP proto-col parsing and formatting, application processing (fordynamically generated content), and access to applica-tion data. Much of this processing involves operatingsystem services. Different parts of the processing requiredifferent parallelization strategies. For example, HTTPparsing is easy to parallelize by connection, while appli-

cation processing is often best parallelized by partition-ing application data across cores to avoid multiple corescontending for the same data. Even for read-only ap-plication data, such partitioning may help maximize theamount of distinct data cached by avoiding duplicatingthe same data in many cores’ local caches.

The Corey Web server is built from three compo-nents: Web daemons (webd), kernel cores, and applica-tions (see Figure 6). The components communicate viashared-memory IPC. Webd is responsible for process-ing HTTP requests. Every core running a webd front-end uses a private TCP/IP stack. Webd can manipu-late the network device directly or use a kernel core todo so. If using a kernel core, the TCP/IP stack of eachcore passes packets to transmit and buffers to receive in-coming packets to the kernel core using a syscall device.Webd parses HTTP requests and hands them off to acore running application code. The application core per-forms the required computation and returns the results towebd. Webd packages the results in an HTTP responseand transmits it, possibly using a kernel core. Applica-tions may run on dedicated cores (as shown in Figure 6)or run on the same core as a webd front-end.

All kernel objects necessary for a webd core to com-plete a request, such as packet buffer segments, networkdevices, and shared segments used for IPC are referencedby a private per-webd core share. Furthermore, mostkernel objects, with the exception of the IPC segments,are used by only one core. With this design, once IPCsegments have been mapped into the webd and applica-tion address ranges, cores process requests without con-tending over any global kernel data or locks, except asneeded by the application.

The application running with webd can run in twomodes: random mode and locality mode. In randommode, webd forwards each request to a randomly cho-sen application core. In locality mode, webd forwardseach request to a core chosen from a hash of the nameof the data that will be needed in the request. Localitymode increases the probability that the application corewill have the needed data in its cache, and decreases re-dundant caching of the same data.

7 IMPLEMENTATIONCorey runs on AMD Opteron and the Intel Xeon proces-sors. Our implementation is simplified by using a 64-bitvirtual address space, but nothing in the Corey design re-lies on a large virtual address space. The implementationof address ranges is geared towards architectures withhardware page tables. Address ranges could be ported toTLB-only architectures, such as MIPS, but would pro-vide less performance benefit, because every core mustfill its own TLB.

The Corey implementation is divided into two parts:the low-level code that implements Corey objects (de-scribed in Section 4) and the high-level Unix-like envi-ronment (described in Section 5). The low-level code(the kernel) executes in the supervisor protection do-main. The high-level Unix-like services are a library thatapplications link with and execute in application protec-tion domains.

The low-level kernel implementation, which includesarchitecture specific functions and device drivers, is11,000 lines of C and 150 lines of assembly. The Unix-like environment, 11,000 lines of C/C++, provides thebuffer cache, cfork, and TCP/IP stack interface as wellas the Corey-specific glue for the uClibc [28] C standardlibrary, lwIP, and the Streamflow [24] dynamic memoryallocator. We fixed several bugs in Streamflow and addedsupport for x86-64 processors.

8 EVALUATIONThis section demonstrates the performance improve-ments that can be obtained by allowing applications tocontrol sharing. We make these points using several mi-crobenchmarks evaluating address ranges, kernel cores,and shares independently, as well as with the two appli-cations described in Section 6.

8.1 Experimental setupWe ran all experiments on an AMD 16-core system (seeFigure 2 in Section 2) with 64 Gbytes of memory. Wecounted the number of cache misses and computed theaverage latency of cache misses using the AMD hard-ware event counters. All Linux experiments use DebianLinux with kernel version 2.6.25 and pin one thread toeach core. The kernel is patched with perfctr 2.6.35 toallow application access to hardware event counters. ForLinux MapReduce experiments we used our version ofStreamflow because it provided better performance thanother allocators we tried, such as TCMalloc [11] andglibc 2.7 malloc. All network experiments were per-formed on a gigabit switched network, using the server’sIntel Pro/1000 Ethernet device.

We also have run many of the experiments with Coreyand Linux on a 16-core Intel machine and with Windowson 16-core AMD and Intel machines, and we draw con-clusions similar to the ones reported in this section.

8.2 Address rangesTo evaluate the benefits of address ranges in Corey, weneed to investigate two costs in multicore applicationswhere some memory is private and some is shared. First,the contention costs of manipulating mappings for pri-vate memory. Second, the soft page-fault costs for mem-ory that is used on multiple cores. We expect Corey tohave low costs for both situations. We expect other sys-tems (represented by Linux in these experiments) to have

0

20000

40000

60000

80000

100000

120000

140000

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

cycl

es p

er p

age

Cores

Linux singleLinux separate

Corey address ranges

(a) memclone

0

100

200

300

400

500

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Tim

e (m

illi

seco

nd

s)

Cores

Linux singleLinux separate

Corey address ranges

(b) mempass

Figure 7: Address ranges microbenchmark results.

low cost for only one type of sharing, but not both, de-pending on whether the application uses a single addressspace shared by all cores or a separate address space percore.

The memclone [3] benchmark explores the costs ofprivate memory. Memclone has each core allocate a100 Mbyte array and modify each page of its array. Thememory is demand-zero-fill: the kernel initially allo-cates no memory, and allocates pages only in responseto page faults. The kernel allocates the memory fromthe DRAM system connected to the core’s chip. Thebenchmark measures the average time to modify eachpage. Memclone allocates cores from chips in a round-robin fashion. For example, when using five cores,memclone allocates two cores from one chip and onecore from each of the other three chips.

Figure 7(a) presents the results for three situations:Linux with a single address space shared by per-corethreads, Linux with a separate address space (process)per core but with the 100 Mbyte arrays mmaped into eachprocess, and Corey with separate address spaces but withthe arrays mapped via shared address ranges.Memclone scales well on Corey and on Linux with

separate address spaces, but poorly on Linux with a sin-gle address space. On a page fault both Corey and Linux

verify that the faulting address is valid, allocate and cleara 4 Kbyte physical page, and insert the physical page intothe hardware page table. Clearing a 4 Kbyte page incurs64 L3 cache misses and copies 4 Kbytes from DRAM tothe local L1 cache and writes 4 Kbytes from the L3 backto DRAM; however, the AMD cache line prefetcher al-lows a core to clear a page in 3350 cycles (or at a rate of1.2 bytes per cycle).

Corey and Linux with separate address spaces each in-cur only 64 cache misses per page fault, regardless of thenumber of cores. However, the cycles per page grad-ually increases because the per-chip memory controllercan only clear 1.7 bytes per cycle and is saturated whenmemclone uses more than two cores on a chip.

For Linux with a single address space cores contendover shared address space data structures. With six coresthe cache-coherency messages generated from lockingand manipulating the shared address space begin to con-gest the interconnect, which increases the cycles requiredfor processing a page fault. Processing a page fault takes14 times longer with 16 cores than with one.

We use a benchmark called mempass to evaluatethe costs of memory that is used on multiple cores.Mempass allocates a single 100 Mbyte shared buffer onone of the cores, touches each page of the buffer, andthen passes it to another core, which repeats the processuntil every core touches every page. Mempassmeasuresthe total time for every core to touch every page.

Figure 7(b) presents the results for the same configu-rations used in memclone. This time Linux with a sin-gle address space performs well, while Linux with sep-arate address spaces performs poorly. Corey performswell here too. Separate address spaces are costly withthis workload because each core separately takes a softpage fault for the shared page.

To summarize, a Corey application can use addressranges to get good performance for both shared and pri-vate memory. In contrast, a Linux application can getgood performance for only one of these types of mem-ory.

8.3 Kernel coresThis section explores an example in which use of theCorey kernel core abstraction improves scalability. Thebenchmark application is a simple TCP service, whichaccepts incoming connections, writing 128 bytes to eachconnection before closing it. Up to 15 separate client ma-chines (as many as there are active server cores) generatethe connection requests.

We compare two server configurations. One of them(called “Dedicated”) uses a kernel core to handle all net-work device processing: placing packet buffer pointersin device DMA descriptors, polling for received pack-ets and transmit completions, triggering device transmis-

0

20000

40000

60000

80000

100000

120000

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

con

nec

tio

ns

per

sec

on

d

Cores

PollingDedicated

Linux

(a) Throughput.

0

50

100

150

200

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

L3

mis

ses

per

co

nn

ecti

on

Cores

PollingDedicated

Linux

(b) L3 cache misses.

Figure 8: TCP microbenchmark results.

sions, and manipulating corresponding driver data struc-tures. The second configuration, called “Polling”, uses akernel core only to poll for received packet notificationsand transmit completions. In both cases, each other coreruns a private TCP/IP stack and an instance of the TCPservice. For Dedicated, each service core uses shared-memory IPC to send and receive packet buffers with thekernel core. For Polling, each service core transmitspackets and registers receive packet buffers by directlymanipulating the device DMA descriptors (with lock-ing), and is notified of received packets via IPC from thePolling kernel core. The purpose of the comparison is toshow the effect on performance of moving all device pro-cessing to the Dedicated kernel core, thereby eliminatingcontention over device driver data structures. Both con-figurations poll for received packets, since otherwise in-terrupt costs would dominate performance.

Figure 8(a) presents the results of the TCP benchmark.The network device appears to be capable of handlingonly about 900,000 packets per second in total, whichlimits the throughput to about 110,000 connections persecond in all configurations (each connection involves 4input and 4 output packets). The dedicated configurationreaches 110,000 with only five cores, while Polling re-

quires 11 cores. That is, each core is able to handle moreconnections per second in the Dedicated configurationthan in Polling.

Most of this difference in performance is due to theDedicated configuration incurring fewer L3 misses. Fig-ure 8(b) shows the number of L3 misses per connec-tion. Dedicated incurs about 50 L3 misses per connec-tion, or slightly more than six per packet. These includemisses to manipulate the IPC data structures (shared withthe core that sent or received the packet) and the de-vice DMA descriptors (shared with the device) and themisses for a core to read a received packet. Polling in-curs about 25 L3 misses per packet; the difference isdue to misses on driver data structures (such as indexesinto the DMA descriptor rings) and the locks that protectthem. To process a packet the device driver must acquireand release an MCS lock twice: once to place the packeton DMA ring buffer and once to remove it. Since eachL3 miss takes about 100 nanoseconds to service, missesaccount for slightly less than half the total cost for thePolling configuration to process a connection (with twocores). This analysis is consistent with Dedicated pro-cessing about twice as many connections with two cores,since Dedicated has many fewer misses. This approxi-mate relationship holds up to the 110,000 connection persecond limit.

The Linux results are presented for reference. InLinux every core shares the same network stack and acore polls the network device when the packet rate ishigh. All cores place output packets on a software trans-mit queue. Packets may be removed from the softwarequeue by any core, but usually the polling core trans-fers the packets from the software queue to the hardwaretransmit buffer on the network device. On packet recep-tion, the polling core removes the packet from the ringbuffer and performs the processing necessary to place thepacket on the queue of a TCP socket where it can be readby the application. With two cores, most cycles on bothcores are used by the kernel; however, as the number ofcores increases the polling core quickly becomes a bot-tleneck and other cores have many idle cycles.

The TCP microbenchmark results demonstrate that aCorey application can use kernel cores to improve scal-ability by dedicating cores to handling kernel functionsand data.

8.4 SharesWe demonstrate the benefits of shares on Corey by com-paring the results of two microbenchmarks. In the firstmicrobenchmark each core calls share addobj to adda per-core segment to a global share and then removesthe segment from the share with share delobj. Thebenchmark measures the throughput of add and removeoperations. As the number of cores increases, cores

0

1000

2000

3000

4000

5000

6000

7000

8000

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

10

00

s o

f ad

d +

del

per

sec

on

d

Cores

Global sharePer−core shares

(a) Throughput.

0

2

4

6

8

10

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

L3

mis

ses

per

op

erat

ion

Cores

Global sharePer−core shares

(b) L3 cache misses.

Figure 9: Share microbenchmark results.

contend on the scalable read-write lock protecting thehashtable used to implement the share. The second mi-crobenchmark is similar to the first, except each coreadds a per-core segment to a local share so that the coresdo not contend.

Figure 9 presents the results. The global share scalesrelatively well for the first three cores, but at four coresperformance begins to slowly decline. Cores contend onthe global share, which increases the amount of cache co-herence traffic and makes manipulating the global sharemore expensive. The local shares scale perfectly. At16 cores adding to and removing from the global sharecosts 10 L3 cache misses, while no L3 cache misses re-sult from the pair of operations in a local share.

We measured the L3 misses for the Linux file descrip-tor microbenchmark described in Section 1. On one corea duplicate and close incurs no L3 cache misses; how-ever, L3 misses increase with the number of cores and at16 cores a dup and close costs 51 L3 cache misses.

A Corey application can use shares to avoid bottle-necks that might be caused by contention on kernel datastructures when creating and manipulating kernel ob-jects. A Linux application, on the other hand, must usesharing policies specified by kernel implementers, which

0

10

20

30

40

50

60

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Tim

e (s

eco

nd

s)

Cores

CoreyLinux

(a) Corey and Linux performance.

0

5

10

15

20

25

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Imp

rov

emen

t (%

)

Cores

(b) Corey improvement over Linux.

Figure 10: MapReduce wri results.

might force inter-core sharing and unnecessarily limitscalability.

8.5 ApplicationsTo demonstrate that address ranges, kernel cores, andshares have an impact on application performance wepresents benchmark results from the Corey port of Metisand from webd.

8.5.1 MapReduceWe compare Metis on Corey and on Linux, using thewri MapReduce application to build a reverse index ofthe words in a 1 Gbyte file. Metis allocates 2 Gbytesto hold intermediate values and like memclone, Metisallocates cores from chips in a round-robin fashion.

On Linux, cores share a single address space. OnCorey each core maps the memory segments holding in-termediate results using per-core shared address ranges.During the map phase each core writes 〈word, index〉key-value pairs to its per-core intermediate results mem-ory. For pairs with the same word, the indices aregrouped as an array. More than 80% of the map phaseis spent in strcmp, leaving little scope for improve-ment from address ranges. During the reduce phase eachcore copies all the indices for each word it is responsible

buffer cache

filesum

webd

IPC IPC IPC

webd

filesum

NIC

IPC

Figure 11: A webd configuration with two front-end cores and twofilesum cores. Rectangles represent segments, rounded rectanglesrepresents pcores, and the circle represents a network device.

for from the intermediate result memories of all coresto a freshly allocated buffer. The reduce phase spendsmost of its time copying memory and handling soft pagefaults. As the number of cores increases, Linux corescontend while manipulating address space data whileCorey’s address ranges keep contention costs low.

Figure 10(a) presents the absolute performance of wrion Corey and Linux and Figure 10(b) shows Corey’s im-provement relative to Linux. For less than eight coresLinux is 1% faster than Corey because Linux’s soft pagefault handler is about 10% faster than Corey’s when thereis no contention. With eight cores on Linux, addressspace contention costs during reduce cause executiontime for reduce to increase from 0.47 to 0.58 seconds,while Corey’s reduce time decreases from 0.46 to 0.42seconds. The time to complete the reduce phase contin-ues to increase on Linux and decrease on Corey as morecores are used. At 16 cores the reduce phase takes 1.9seconds on Linux and only 0.35 seconds on Corey.

8.5.2 WebdAs described in Section 6.2, applications might be ableto increase performance by dedicating application datato cores. We explore this possibility using a webd appli-cation called filesum. Filesum expects a file nameas argument, and returns the sum of the bytes in that file.Filesum can be configured in two modes. “Random”mode sums the file on a random core and “Locality”mode assigns each file to a particular core and sums arequested file on its assigned core. Figure 11 presents afour core configuration.

We measured the throughput of webd with filesumin Random mode and in Locality mode. We configuredwebd front-end servers on eight cores and filesum oneight cores. This experiment did not use a kernel coreor device polling. A single core handles network inter-rupts, and all cores running webd directly manipulate thenetwork device. Clients only request files from a set ofeight, so each filesum core is assigned one file, which

0

10000

20000

30000

40000

50000

60000

409620481024512256128

con

nec

tio

ns

per

sec

on

d

File size (Kbytes, log scale)

RandomLocality

Figure 12: Throughput of Random mode and Locality mode.

it maps into memory. The HTTP front-end cores andfilesum cores exchange file requests and responses us-ing a shared memory segment.

Figure 12 presents the results of Random mode andLocality mode. For file sizes of 256 Kbytes and smallerthe performance of Locality and Random mode are bothlimited by the performance of the webd front-ends’ net-work stacks. In Locality mode, each filesum corecan store its assigned 512 Kbyte or 1024 Kbyte filein chip caches. For 512 Kbyte files Locality modeachieves 3.1 times greater throughput than Random andfor 1024 Kbyte files locality mode achieves 7.5 timesgreater throughput. Larger files cannot be fit in chipcaches and the performance of both Locality mode andRandom mode is limited by DRAM performance.

9 DISCUSSION AND FUTURE WORKThe results above should be viewed as a case for theprinciple that applications should control sharing ratherthan a conclusive “proof”. Corey lacks many featuresof commodity operating systems, such as Linux, whichinfluences experimental results both positively and nega-tively.

Many of the ideas in Corey could be applied to existingoperating systems such as Linux. The Linux kernel coulduse a variant of address ranges internally, and perhapsallow application control via mmap flags. Applicationscould use kernel-core-like mechanisms to reduce systemcall invocation costs, to avoid concurrent access to kerneldata, or to manage an application’s sharing of its owndata using techniques like computation migration [6, 14].Finally, it may be possible for Linux to provide share-like control over the sharing of file descriptors, entries inbuffer and directory lookup caches, and network stacks.

10 RELATED WORKCorey is influenced by Disco [5] (and its relatives [13,30]), which runs as a virtual machine monitor on a

NUMA machine. Like Disco, Corey aims to avoid ker-nel bottlenecks with a small kernel that minimizes shareddata. However, Corey has a kernel interface like an ex-okernel [9] rather than a virtual machine monitor.

Corey’s focus on supporting library operating systemsresembles Disco’s SPLASHOS, a runtime tailored forrunning the Splash benchmark [26]. Corey’s library op-erating systems require more operating system features,which has led to the Corey kernel providing a number ofnovel ideas to allow libraries to control sharing (addressranges, kernel cores, and shares).

K42 [3] and Tornado [10], like Corey, are designedso that independent applications do not contend over re-sources managed by the kernel. The K42 and Tornadokernels provide clustered objects that allow different im-plementations based on sharing patterns. The Corey ker-nel takes a different approach; applications customizesharing policies as needed instead of selecting from a setof policies provided by kernel implementers. In addition,Corey provides new abstractions not present in K42 andTornado.

Much work has been done on making widely-used op-erating systems run well on multiprocessors [4]. TheLinux community has made many changes to improvescalability (e.g., Read-Copy-Update (RCU) [20] locks,local runqueues [1], libnuma [16], improved load-balancing support [17]). These techniques are comple-mentary to the new techniques that Corey introduces andhave been a source of inspiration. For example, Coreyuses tricks inspired by RCU to implement efficient func-tions for retrieving and removing objects from a share.

As mentioned in the Introduction, Gough et al. [12]and Veal and Foong [29] have identified Linux scalingchallenges in the kernel implementations of schedulingand directory lookup, respectively.

Saha et al. have proposed the Multi-Core Run Time(McRT) for desktop workloads [23] and have exploreddedicating some cores to McRT and the application andothers to the operating system. Corey also uses spa-tial multiplexing, but provides a new kernel that runs onall cores and that allows applications to dedicate kernelfunctions to cores through kernel cores.

An Intel white paper reports use of spatial multiplex-ing of packet processing in a network analyzer [15], withperformance considerations similar to webd running inLocality or Random modes.

Effective use of multicore processors has many aspectsand potential solution approaches; Corey explores only afew. For example, applications could be assisted in cop-ing with heterogeneous hardware [25]; threads could beautomatically assigned to cores in a way that minimizescache misses [27]; or applications could schedule workin order to minimize the number of times a given datumneeds to be loaded from DRAM [7].

11 CONCLUSIONSThis paper argues that, in order for applications to scaleon multicore architectures, applications must controlsharing. Corey is a new kernel that follows this princi-ple. Its address range, kernel core, and share abstractionsensure that each kernel data structure is used by only onecore by default, while giving applications the ability tospecify when sharing of kernel data is necessary. Ex-periments with a MapReduce application and a syntheticWeb application demonstrate that Corey’s design allowsthese applications to avoid scalability bottlenecks in theoperating system and outperform Linux on 16-core ma-chines. We hope that the Corey ideas will help appli-cations to scale to the larger number of cores on futureprocessors. All Corey source code is publicly available.

ACKNOWLEDGMENTSWe thank Russ Cox, Austin Clements, Evan Jones, EmilSit, Xi Wang, and our shepherd, Wolfgang Schröder-Preikschat, for their feedback. The NSF partially fundedthis work through award number 0834415.

REFERENCES[1] J. Aas. Understanding the Linux 2.6.8.1

CPU scheduler, February 2005. http://josh.trancesoftware.com/linux/.

[2] A. Agarwal and M. Levy. Thousand-core chips: thekill rule for multicore. In Proceedings of the 44thAnnual Conference on Design Automation, pages750–753, 2007.

[3] J. Appavoo, D. D. Silva, O. Krieger, M. Auslander,M. Ostrowski, B. Rosenburg, A. Waterland, R. W.Wisniewski, J. Xenidis, M. Stumm, and L. Soares.Experience distributing objects in an SMMP OS.ACM Trans. Comput. Syst., 25(3):6, 2007.

[4] R. Bryant, J. Hawkes, J. Steiner, J. Barnes, andJ. Higdon. Scaling linux to the extreme. In Pro-ceedings of the Linux Symposium 2004, pages 133–148, Ottawa, Ontario, June 2004.

[5] E. Bugnion, S. Devine, and M. Rosenblum.DISCO: running commodity operating systems onscalable multiprocessors. In Proceedings of the16th ACM Symposium on Operating Systems Prin-ciples, pages 143–156, Saint-Malo, France, Octo-ber 1997. ACM.

[6] M. C. Carlisle and A. Rogers. Software caching andcomputation migration in Olden. In Proceedings ofthe 5th ACM SIGPLAN Symposium on Principlesand Practice of Parallel Programming, 1995.

[7] S. Chen, P. B. Gibbons, M. Kozuch, V. Liaskovi-tis, A. Ailamaki, G. E. Blelloch, B. Falsafi, L. Fix,N. Hardavellas, T. C. Mowry, and C. Wilkerson.Scheduling Threads for Constructive Cache Shar-ing on CMPs. In Proceedings of the 19th ACMSymposium on Parallel Algorithms and Architec-tures, pages 105–115. ACM, 2007.

[8] J. Dean and S. Ghemawat. MapReduce: simplifieddata processing on large clusters. Commun. ACM,51(1):107–113, 2008.

[9] D. R. Engler, M. F. Kaashoek, and J. W. O’Toole.Exokernel: An operating system architecture forapplication-level resource management. In Pro-ceedings of the 15th ACM Symposium on OperatingSystems Principles, pages 251–266, Copper Moun-tain, CO, December 1995. ACM.

[10] B. Gamsa, O. Krieger, J. Appavoo, and M. Stumm.Tornado: maximizing locality and concurrency ina shared memory multiprocessor operating system.In Proceedings of the 3rd Symposium on Operat-ing Systems Design and Implementation, pages 87–100, 1999.

[11] Google Performance Tools. http://goog-perftools.sourceforge.net/.

[12] C. Gough, S. Siddha, and K. Chen. Kernelscalability—expanding the horizon beyond finegrain locks. In Proceedings of the Linux Sympo-sium 2007, pages 153–165, Ottawa, Ontario, June2007.

[13] K. Govil, D. Teodosiu, Y. Huang, and M. Rosen-blum. Cellular Disco: resource management us-ing virtual clusters on shared-memory multiproces-sors. In Proceedings of the 17th ACM Symposiumon Operating Systems Principles, pages 154–169,Kiawah Island, SC, October 1999. ACM.

[14] W. C. Hsieh, M. F. Kaashoek, and W. E.Weihl. Dynamic computation migration in dsmsystems. In Supercomputing ’96: Proceedings ofthe 1996 ACM/IEEE conference on Supercomput-ing (CDROM), Washington, DC, USA, 1996.

[15] Intel. Supra-linear Packet Processing Performancewith Intel Multi-core Processors. ftp://download.intel.com/technology/advanced comm/31156601.pdf.

[16] A. Klein. An NUMA API for Linux, Au-gust 2004. http://www.firstfloor.org/˜andi/numa.html.

[17] Linux kernel mailing list. http://kerneltrap.org/node/8059.

[18] Linux Symposium. http://www.linuxsymposium.org/.

[19] lwIP. http://savannah.nongnu.org/projects/lwip/.

[20] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen,O. Krieger, and R. Russell. Read-copy update. InProceedings of the Linux Symposium 2002, pages338–367, Ottawa, Ontario, June 2002.

[21] J. M. Mellor-Crummey and M. L. Scott. Al-gorithms for scalable synchronization on shared-memory multiprocessors. ACM Trans. Comput.Syst., 9(1):21–65, 1991.

[22] C. Ranger, R. Raghuraman, A. Penmetsa, G. Brad-ski, and C. Kozyrakis. Evaluating MapReduce forMulti-core and Multiprocessor Systems. In Pro-ceedings of the 13th International Symposium onHigh Performance Computer Architecture, pages13–24. IEEE Computer Society, 2007.

[23] B. Saha, A.-R. Adl-Tabatabai, A. Ghuloum, M. Ra-jagopalan, R. L. Hudson, L. Petersen, V. Menon,B. Murphy, T. Shpeisman, E. Sprangle, A. Ro-hillah, D. Carmean, and J. Fang. Enabling scal-ability and performance in a large-scale CMP en-vironment. In Proceedings of the 2nd ACMSIGOPS/EuroSys European Conference on Com-puter Systems, pages 73–86, New York, NY, USA,2007. ACM.

[24] S. Schneider, C. D. Antonopoulos, and D. S.Nikolopoulos. Scalable Locality-Conscious Mul-tithreaded Memory Allocation. In Proceedings ofthe 2006 ACM SIGPLAN International Symposiumon Memory Management, pages 84–94, 2006.

[25] A. Schüpbach, S. Peter, A. Baumann, T. Roscoe,P. Barham, T. Harris, and R. Isaacs. Embracingdiversity in the Barrelfish manycore operating sys-tem. In Proceedings of the Workshop on ManagedMany-Core Systems (MMCS), Boston, MA, USA,June 2008.

[26] J. P. Singh, W.-D. Weber, and A. Gupta. SPLASH:Stanford Parallel Applications for Shared-Memory.Computer Architecture News, 20(1):5–44.

[27] D. Tam, R. Azimi, and M. Stumm. Thread cluster-ing: sharing-aware scheduling on SMP-CMP-SMTmultiprocessors. In Proceedings of the 2nd ACMSIGOPS/EuroSys European Conference on Com-puter Systems, pages 47–58, New York, NY, USA,2007. ACM.

[28] uClibc. http://www.uclibc.org/.

[29] B. Veal and A. Foong. Performance scalabilityof a multi-core web server. In Proceedings of the3rd ACM/IEEE Symposium on Architecture for net-working and communications systems, pages 57–66, New York, NY, USA, 2007. ACM.

[30] B. Verghese, S. Devine, A. Gupta, and M. Rosen-blum. Operating system support for improvingdata locality on CC-NUMA compute servers. InProceedings of the 7th international conference onArchitectural support for programming languagesand operating systems, pages 279–289, New York,NY, USA, 1996. ACM.

[31] K. Yotov, K. Pingali, and P. Stodghill. Auto-matic measurement of memory hierarchy parame-ters. SIGMETRICS Perform. Eval. Rev., 33(1):181–192, 2005.