Address Translation Aware Memory Consistency

..........................................................................................................................................................................................................................

ADDRESS TRANSLATION AWAREMEMORY CONSISTENCY

..........................................................................................................................................................................................................................

COMPUTER SYSTEMS WITH VIRTUAL MEMORY ARE SUSCEPTIBLE TO DESIGN BUGS

AND RUNTIME FAULTS IN THEIR ADDRESS TRANSLATION SYSTEMS. DETECTING BUGS

AND FAULTS REQUIRES A CLEAR SPECIFICATION OF CORRECT BEHAVIOR. A NEW

FRAMEWORK FOR ADDRESS TRANSLATION AWARE MEMORY CONSISTENCY MODELS

ADDRESSES THIS NEED.

......We expect computers to func-tion correctly, despite potential problemslike design bugs and physical faults. The con-sequences of incorrect functionality includesilent data corruptions and crashes. Thegoal of dependable computing is to reducethe probabilities of these events at the lowestcost in terms of performance loss, hardware,power consumption, and design and verifica-tion time. Here, we focus on detecting incor-rect behavior caused by design bugs orphysical faults before it leads to a data cor-ruption or crash.

To detect incorrect behavior, we mustfirst precisely specify what constitutes correctbehavior. For a processor core, the instruc-tion set architecture (ISA) specifies the exactsemantics of every instruction in the instruc-tion set. The ISA also specifies the architec-ture’s memory consistency model,1 whichdefines the legal software-visible orderingsof loads and stores performed by multiplethreads. After Lamport introduced the firstmemory consistency model (sequentialconsistency2), it became easier to staticallyor dynamically verify a memory system.Lamport’s key contribution to verificationwas a precise, microarchitecture-independentspecification of correct memory systembehavior.

Nevertheless, modern memory systemsstill have many bugs. We identified21 bugs related to address translation in pub-lished errata (see Table 1 for a small subset ofthese).3-5 And we believe that one underlyingcause of these bugs is a tendency to oversim-plify memory consistency. Memory consis-tency isn’t just one monolithic interfacebetween the hardware and the software, asFigure 1 shows. Rather, it’s a set of interfacesbetween the hardware and various levels ofsoftware, as in Figure 2. Although Adveand Gharachorloo explained memory consis-tency’s multilevel nature,1 architects don’talways adopt this more comprehensive defi-nition of memory consistency. For example,when we teach students about memory con-sistency models, we generally don’t specifywhether that model refers to virtual or phys-ical addresses.

In this article, we develop a framework forspecifying two critical levels of memory con-sistency: the physical address memory consis-tency (PAMC) model and the virtual addressmemory consistency (VAMC) model, whichdefine the behavior of operations (loads,stores, and memory barriers) on physicaland virtual addresses, respectively. As partof this specification framework, we discussand formalize address translation’s crucial

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 109

Bogdan F. Romanescu

Alvin R. Lebeck

Daniel J. Sorin

Duke University

0272-1732/11/$26.00 �c 2011 IEEE Published by the IEEE Computer Society

...................................................................

109

role in supporting a VAMC model. Withoutcorrect address translation, a system withvirtual memory can’t enforce any VAMCmodel. Furthermore, we develop addresstranslation-aware specifications of memoryconsistency not only to benefit design andverification teams, but also to enable archi-tects to design runtime checkers that detectincorrect behaviors.

Specifying address translation awarememory consistency

A memory consistency specification pro-vides two important functions: it serves as acontract between the system and theprogrammer, and it provides the formalframework necessary for verifying correctmemory system operation, either statisticallyor dynamically.

Levels of memory consistencyA computer system presents memory

interfaces (consistency models) at multiple

levels, as Figure 2 shows. We position hard-ware below all levels, because it providesmechanisms that we can use to enforceconsistency models at various levels (for ex-ample, the processor provides in-order in-struction commit). We consider four levelsrelevant to programmers. At each level, theconsistency model defines the legal orderingsof the memory operations available at thatlevel. These consistency models are necessaryinterfaces that are included in the specifica-tions of the instruction set architecture(ISA), application binary interface (ABI),and application programming interface(API). Here, we discuss these levels, startingat the lowest level.

� PAMC: Unmapped software, includingthe boot code and part of the systemsoftware that manages address transla-tion, relies on PAMC. ImplementingPAMC is the hardware’s responsibilityand, as such, is specified precisely inthe architectural manual.

� VAMC: All mapped software relies onVAMC, including mapped system soft-ware. VAMC builds upon PAMC andrequires support from address transla-tion software and the hardware.

� User process memory consistency(UPMC): UPMC could be identicalto VAMC, or it could differ, as in thecase of software transactional memoryor software distributed shared memory.

� High-level language consistency: User-level programmers see the consistencymodel specified by the high-level

[3B2-11] mmi2011010109.3d 21/1/011 10:29 Page 110

Software

Hardware

Memoryconsistency

model

Figure 1. Address translation oblivious

memory consistency. In this simplistic view

of consistency, there is a single interface

between software and hardware.

Table 1. A small sample of published address translation bugs.

Chip Bug Effect

AMD Athlon64/Opteron6 TLB (translation look-aside buffer) flush filter can

cause coherency problem in multicore systems.

Unpredictable system failure (possible use of

stale translations).

INVLPG instruction with address prefix doesn’t

correctly invalidate the translation requested.

Unpredictable system behavior (possible use

of stale translations).

Intel Core Duo7 One core updates a page table entry while the

other core uses the same translation entry,

which could lead to unexpected behavior.

Unexpected processor behavior.

Updating a PTE (page table entry) by changing

read/write, U/S or P bits without TLB shootdown

could cause unexpected processor behavior.

Unexpected processor behavior.

....................................................................

110 IEEE MICRO

...............................................................................................................................................................................................

TOP PICKS

language, such as Java’s consistencymodel.8 These high-level language con-sistency models are supported by thecompilers, runtime systems, and lowerlevel consistency models.

Existing consistency models—such as se-quential consistency, processor consistency,weak ordering, and release consistency—don’t distinguish between virtual and physi-cal addresses. Lamport’s original definitionof sequential consistency is typical in that itspecifies a total order of operations (loadsand stores). But it doesn’t specify whetherthe loads and stores are to virtual or physicaladdresses. We refer to these consistencymodels as being address translation oblivious.

We specifically focus on PAMC andVAMC and the hardware and softwareinvolved in supporting them.

Specifying PAMCWe can fairly easily adapt an address

translation oblivious consistency model asPAMC’s specification. For example, thePAMC model could be sequential consis-tency, in which case the interface would spec-ify that there must exist a total order of allloads and stores to physical addresses thatrespects each thread’s program order, andthat each load’s value is equal to the valueof the most recent store to that physical ad-dress in the total order.

We specify consistency models using atable-based scheme like those of Hill et al.9

and Arvind and Maessen.10 The table speci-fies which program orderings are enforced bythe consistency model. Thus, Tables 2 and 3define adaptations of sequential consistencyand weak ordering for PAMC, respectively.Some consistency models have atomicity

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 111

Considers synonymsets of virtualaddresses

Considers physicaladdresses

Hardware

Physical address memory consistency (PAMC)

Unmapped/fixed-map assembly code

Virtual address memory consistency (VAMC)

User process memory consistency (UPMC)

User-levelbinaries

Java virtualmachine

Compiler

Other unmappedsystem software

Mapped virtual memorysystem software

Other mappedsystem software

Java memorymodel

HLL memorymodel

Unmapped virtual memorysystem software

Figure 2. Address translation aware memory consistency. In this richer view of consistency,

there are multiple interfaces for supporting different types of software. This article focuses

on the shaded portions of this figure.

....................................................................

JANUARY/FEBRUARY 2011 111

constraints that can’t be expressed with just atable (for example, stores are atomic). We canspecify these models by augmenting the tablewith a specification of atomicity require-ments, as previous researchers have done.10

Specifying VAMCAlthough adapting an address translation

oblivious consistency model for PAMC isstraightforward, three challenges arise whenadapting an address translation obliviousconsistency model for VAMC: synonyms,mapping and permission changes, and loadand store side effects.

Synonyms. Synonyms are multiple virtualaddresses (VAs) that map to the same phys-ical address (PA). Consider the example inFigure 3, in which VA1 and VA2 map toPA1. Sequential consistency requires atotal order in which a load’s value equalsthe value of the most recent store to thesame address. Unfortunately, naively apply-ing sequential consistency at the VAMClevel allows an execution in which x ¼ 2and y ¼ 1. The programmer expects thatthe loads in both threads will be assigned

the value of the most recent update toPA1. However, a definition of VAMCthat didn’t consider address translationwould allow x to receive the most recentvalue of VA2 and y to receive the most re-cent value of VA1, without consideringthat they both map to PA1. To overcomethis challenge, we reformulate address trans-lation oblivious consistency models forVAMC by applying the model to synonymsets of (virtual) addresses rather than individ-ual addresses. For example, we define se-quential consistency for VAMC as follows:there must be a total order of all loadsand stores to virtual addresses that respectsprogram order and in which each loadgets the value of the most recent store toany virtual address in the same virtual addresssynonym set. We can make similar modifica-tions to adapt other address translationoblivious consistency models for VAMC.

Programmers that use synonyms generallyexpect ordering to be maintained betweenaccesses to synonymous virtual addresses.Incorporating synonyms explicitly in theconsistency model enables programmers toreason about the ordering of accesses tovirtual addresses.

Explicitly stating the ordering constraintsof synonyms is necessary for verification.An error in the address translation hardwarecould result in a violation of ordering amongsynonyms that might not be detected with-out the formal specification.

Mapping and permission changes. There is aricher set of memory operations at theVAMC level than at the PAMC level.User-level and system-level programmersat the VAMC interface are provided withOS software routines to map and remap

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 112

Table 2. Sequential consistency for

physical address memory

consistency (PAMC).

Op 2

Op

1

Load Store

Load X X

Store X X.......................................................*Loads and stores are to physical addresses.An ‘‘X’’ denotes an enforced ordering.

Table 3. Weak ordering for PAMC.

Op 2

Op

1

Load Store MemoryBarrier

Load A X

Store A A X

MemoryBarrier X X X.......................................................................................................*Loads and stores are to physical addresses. An ‘‘X’’ denotes an enforcedordering; an ‘‘A’’ denotes an ordering that’s enforced if the operations areto the same physical address. Empty entries denote no ordering.

Thread 1 Thread 2

Store VA1 = 1Store VA2 = 2Load y = VA1 // y = 1

Load x = VA2 // x = 2

Figure 3. Example of the synonym problem.

Assume sequential consistency. VA1 and

VA2 map to PA1, and PA1 is initially zero.

....................................................................

112 IEEE MICRO

...............................................................................................................................................................................................

TOP PICKS

or change permissions on virtual memoryregions, such as the mk_pte() (makenew page table entry) or pte_mkread()(make page table entry readable) functionsin Linux 2.6. We call these software rou-tines map/remap functions (MRFs).

The code snippet in the left side ofFigure 4 is written for a system implement-ing the Power ISA. The code snippet illus-trates the need to consider MRFs and theirordering. We expect that the load byThread1 should return the value C writtenby Thread2, because that appears to be thevalue of the most recent write (in causalorder, according to the Power ISA’s weak-or-dered memory model). However, this codesnippet doesn’t guarantee when the tlbie(translation look-aside buffer invalidateentry) instruction will be observed byThread2—thus, Thread2 could continue tooperate with the old translation of VA1 toPA1. Therefore, Thread2’s store to VA1could modify PA1. When Thread1 performsits load to VA1, it could access PA2 and thusobtain B’s old value.

The problem with the code is that itdoesn’t guarantee that the invalidation

generated by the tlbie instruction will ex-ecute on Thread2’s core before Thread2’sstore to VA1 accesses its translation in itsTLB. Understanding only the PAMCmodel isn’t sufficient for the programmerto reason about this code’s behavior. Theprogrammer must also understand howMRFs are ordered. We show a corrected ver-sion of the code on the right side of Figure 4.In this code, Thread1 executes a tlbsync(translation look-aside buffer synchronize)instruction that’s effectively a fence for theMRF. Specifically, the tlbsync guaranteesthat other cores have observed the tlbieinstruction executed by Thread1.

A runtime hardware error or design bugcould cause a TLB invalidation to bedropped or delayed, resulting in TLB inco-herence. A formal specification of MRForderings is required to develop proper veri-fication techniques, and PAMC is insuffi-cient for this purpose.

Load/store side effects. The third challengein specifying VAMC is that loads and storesto virtual addresses have side effects. Theaddress translation system includes status

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 113

Buggy Code Correct Code

Thread1 Thread2Thread2Thread1

MRF {map VA1 to PA2; MRF {map VA1 to PA2;

tlbie VA1; // invalidate // translation // (VA1→PA1)

tlbie VA1; // invalidate // translation

// (VA1→PA1)

} }

tlbsync // fence for MRF

sync; // memory barrier for // regular memory ops

sync; // memory barrier for // regular memory ops

Store VA2 = B Store VA2 = B

)B=!2AV( elihwcnys{spin}

)B=!2AV(elihwcnys{spin}

sync sync

Store VA1 = C Store VA1 = C

sync sync

Store VA2 = D Store VA2 = D

while (VA2 != D) {spin} while (VA2 != D) {spin}

sync sync

Load VA1 // can get C or A Load VA1 // can only get C

Figure 4. Power instruction set architecture (ISA) code snippets illustrate the need to consider the ordering of map/remap

functions (MRF). Initially, VA1 maps to PA1, and the value of PA1 is A.

....................................................................


bits (such as Accessed and Dirty bits) foreach page table entry. These status bits arepart of the architectural state. Thus, theordering of updates to those bits must bespecified in VAMC. We add two new oper-ations to the specification tables: Ld-sband St-sb (load and store impact on sta-tus bits).

Consider the example in Figure 5. With-out knowing how status updates are ordered,the operating system can’t be sure what statewill be visible in these bits. It’s possible thatthe load of the PTE (page table entry) occursbefore the first store’s Dirty bit update.The OS could incorrectly determine that awrite-back was unnecessary, resulting indata loss.

Without a precise specification of statusbit ordering, verification could miss a situa-tion analogous to the software examplewe’ve discussed (see Figure 5). A physicalfault could lead to an error in the orderingof setting a status bit, and this error couldbe overlooked by dynamic verification hard-ware and could lead to silent data corruption.

Putting it all together. Table 4 presents aVAMC adaptation of sequential consis-tency. This specification includes MRFs

and status bit updates, and loads and storesapply to synonym sets of virtual addresses(not individual virtual addresses). This spec-ification provides both a contract for pro-grammers and enables development oftechniques to verify correct memory systemoperation.

Gap between PAMC and VAMCOne of our research areas is dynamic

verification of VAMC (for more informa-tion, see the ‘‘Dynamic verification ofVAMC’’ sidebar). Because we haven’t yetdiscovered an efficient scheme for directdynamic verification of VAMC, we insteaddynamically verify VAMC by dynamicallyverifying PAMC (using an existing scheme)as well as the gap between PAMC andVAMC. This gap is the address translationsystem.

Specification of address translationWe present a framework for specifying

address translation systems.

Address translation assumptionsWe restrict our discussion to page-based

address translation systems. A translation isa tuple <mapping (VP, PP), permissions,

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 114

Store VA1=1;// VA1 maps to PA1

Load VA2;// VA2 maps to the page table entry of VA1

Figure 5. Code snippet illustrating the need to consider side effects. The load is used by

the VM system to determine if the page mapped by VA1 needs to be written back to sec-

ondary storage. If reordered, a Dirty bit set by the store could be missed and the page

incorrectly not written back.

Table 4. Sequential consistency for VAMC.

Operation 2

Op

era

tio

n1

Ld Ld-sb St St-sb MRF

Ld X X X X X

Ld-sb X X X X X

St X X X X X

St-sb X X X X X

MRF X X X X X................................................................................................................................*Loads and stores are to synonym sets of virtual addresses. An ‘‘X’’ denotes an enforcedordering.

....................................................................

114 IEEE MICRO

...............................................................................................................................................................................................

TOP PICKS

status>, where the mapping converts the vir-tual page to a physical page. The physicalpage, permissions, and status informationare specified by the page table entry (PTE)defining the translation. The permissionbits include whether the user or kernelowns the page and whether the page is read-able, writeable, or executable. The statusbits denote whether the page has beenaccessed or is dirty. The status bits areatomically updated in the TLB and in thepage table in memory. In an architecturewith hardware-managed TLBs, the hard-ware is responsible for eventually updatingthe status bits. If the TLBs are software-managed, then status bit updates occur inexception handlers.

To create, modify, or delete a translationor to modify a translation’s permission bits,the kernel performs an MRF. An MRF typ-ically has four activities (see Figure 6). Someof the activities in an MRF require the soft-ware or hardware to perform complicatedactions. For example, delivering the TLBinvalidations could require an interprocessorinterrupt or a global TLB invalidation in-struction that relies on hardware for distrib-uting the invalidations.

A provably sufficient address translation modelWe present a model of an address transla-

tion system that, when combined withPAMCSC (see Table 2), is provably sufficientfor providing VAMCSC (see Table 4). Thisaddress translation model, which we callATSC, is similar to current Linux platforms.ATSC is restrictive and conservative, but it’salso realistic.

ATSC: A sequential address translation model.ATSC is a sequential model of an addresstranslation system. It’s a logical abstractionthat encompasses the behaviors of variouspossible physical implementations. Thismodel has three key aspects:

� MRFs logically occur instantaneouslyand are thus totally ordered with respectto regular loads and stores and other ad-dress translation operations. (Recentoperating systems, such as Linux2.4.16-2.6.24, relax this ATSC con-straint by instead postponing memory

accesses that depend on the translationmodified by the MRF.) Linux enforcesthis aspect of the model using locks.

� A load or store logically occurs instanta-neously and simultaneously with itscorresponding translation access(accessing the mapping, permissions,and status) and possible status bitupdates. A core can adhere to this as-pect of the model in many ways, suchas by snooping TLB invalidations be-tween when a load or store executesand when it commits. A snoop hitforces the load or store to be squashedand reexecuted.

� A store atomically updates all the valuesin the synonym set cached by the coreexecuting the store, and a coherence in-validation atomically invalidates all ofthe values in the synonym set cachedby the core receiving the invalidation.To our knowledge, current systems ad-here to this aspect either by using phys-ical caches or by using virtual cacheswith the same index mapping of syn-onym set virtual addresses.

We now show that ATSC is the bridgebetween PAMCSC and VAMCSC.

PAMCSCþATSC!VAMCSC. PAMCSC spec-ifies that all loads and stores using physicaladdresses are totally ordered. ATSC specifiesthat a translation access occurs instantane-ously and simultaneously with the load orstore. Under ATSC, all MRFs are totally or-dered with respect to each other and with re-spect to loads and stores. ATSC also specifiesthat accesses to synonyms are ordered accord-ing to PAMCSC (for example, via the use ofphysical caches). Therefore, all loads andstores using virtual addresses are totally or-dered. Finally, ATSC specifies that status bit

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 115

generic MRF{

acquire page table lock(s);

create/modify the translation;

send TLB invalidations to other cores;

release page table lock(s);

}

Figure 6. Pseudocode for a generic map/remap function (MRF).

....................................................................


updates are performed simultaneously withthe corresponding load or store, and thusstatus bit updates are totally ordered withrespect to all other operations. Thus,PAMCSC plus ATSC results in VAMCSC,where ordering is enforced between alloperations.

A framework for specifying addresstranslation models

ATSC is just one possible model for ad-dress translation and thus one possiblebridge from a PAMC model to a VAMCmodel. We present a framework for specify-ing address translation models. A preciselyspecified address translation model helpsverify the address translation system and,in turn, VAMC. We haven’t yet provedthe sufficiency of address translation mod-els other than ATSC (that is, that theybridge any particular gap between PAMC

and VAMC); we leave such proofs for fu-ture work. Our framework consists of twoinvariants that are enforced by a combina-tion of hardware and privileged software:the page table is correct, and translationsare coherent.

Page table integrity. The page table mustcontain the correct translations. The pagetable is simply a data structure in memorythat we can reason about in two parts.One part is the page table’s root (or lowestlevel table). The page table’s root is at afixed physical address and uses a fixed map-ping from the virtual to physical address.The second part is dynamically mappedand thus relies on address translation.

To more clearly distinguish howhardware and software collaborate in theaddress translation system, we divide pagetable integrity into two subinvariants.

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 116

...............................................................................................................................................................................................

Dynamic verification of VAMC

One of our primary goals in specifying virtual address memory consis-

tency (VAMC) was to be able to dynamically verify it. Prior work devel-

oped schemes for dynamic verification of PAMC,1,2 and we initially

attempted to develop an analogous scheme for VAMC. Due to implemen-

tation costs, we changed our approach. Rather than directly verify

VAMC, we factored it into its constituent parts, PAMC and address

translation, and dynamically verified them instead. Because schemes al-

ready exist for PAMC, we focused on dynamic verification of address

translation.

To dynamically verify ATSC (a sequential model of an address trans-

lation system)—which we call DVATSC—we must dynamically verify

both ATSC invariants: page table integrity and translation mapping

coherence.

� Checking page table integrity. PT-SubInv1 is an invariant that’s

maintained by software. Fundamentally, no hardware solution

can completely check this invariant, because the hardware doesn’t

have semantic knowledge of the software’s goal. One existing so-

lution to this problem is self-checking code;3 we defer further re-

search in this area to future work. To check that PT-SubInv2 is

maintained, we can adopt any previously proposed dynamic veri-

fication scheme for PAMC.

� Checking translation coherence. DVATSC aims to dynamically ver-

ify the three translation coherence subinvariants. Because we’ve

specified these subinvariants in terms of tokens, we can dynam-

ically verify them by adapting the TCSC scheme,4 which was pre-

viously used to dynamically verify token-based cache coherence.

TCSC’s key insight is that cache coherence states can be logically

represented with token counts that can be periodically checked;

this same insight applies to translation coherence. In a correctly

operating ATSC system, the exchanges of logical tokens will

obey ATSC’s three coherence subinvariants. DVATSC therefore

checks these three subinvariants at runtime.

We’ve implemented and experimentally evaluated DVATSC. Error in-

jection results show that DVATSC detects the errors that mimic all 21 pub-

lished address translation bugs. Performance results show that the

performance impact of DVATSC is less than 2 percent.

References

1. K. Chen, S. Malik, and P. Patra, ‘‘Runtime Validation of Mem-

ory Ordering Using Constraint Graph Checking,’’ Proc. 13th

Int’l Symp. High-Performance Computer Architecture, IEEE

Press, 2008, pp. 415-426.

2. A. Meixner and D.J. Sorin, ‘‘Dynamic Verification of Memory

Consistency in Cache-Coherent Multithreaded Computer

Architectures,’’ Proc. Int’l Conf. Dependable Systems and

Networks, IEEE Press, 2006, pp. 73-82.

3. M. Blum and S. Kannan, ‘‘Designing Programs that Check

Their Work,’’ Proc. 21st Ann. ACM Symp. Theory of Comput-

ing, ACM Press, 1989, pp. 86-97.

4. A. Meixner and D.J. Sorin, ‘‘Error Detection via Online

Checking of Cache Coherence with Token Coherence Signa-

tures,’’ Proc. 12th Int’l Symp. High-Performance Computer

Architecture, IEEE Press, 2007, pp. 145-156.

....................................................................

116 IEEE MICRO

...............................................................................................................................................................................................

TOP PICKS

PT-SubInv1 requires that the page tabledata structure correctly defines the transla-tions. This subinvariant is enforced by theprivileged code that maintains the pagetable. PT-SubInv2 requires that thepage table’s root is correct. This subinvar-iant is enforced by hardware (as specifiedby PAMC) because the root has a fixedphysical address.

Translation coherence. Translation coher-ence is similar but not identical to cache co-herence for regular memory. All cachedcopies of a translation (in TLBs) should becoherent with respect to the page table.The notion of TLB coherence isn’t new,11

although it hasn’t yet been definedprecisely.

We choose to specify the translation co-herence invariants in a way that’s similar tohow Martin et al. specified cache coherenceinvariants for Token Coherence.12 (The ab-stract tokens that we consider here are inde-pendent of any tokens used for the purposesof implementing either regular cache coher-ence or translation coherence.) We considereach translation to logically have a fixed num-ber of tokens, T, associated with it. T mustbe at least as great as the number of TLBsin the system. Tokens can reside in TLBsor in main memory. The following threesubinvariants are required:

� Coherence-SubInv1: At anypoint in logical time, exactly T tokensexist for each translation. This conser-vation law doesn’t permit a token tobe created, destroyed, or convertedinto a token for another translation.

� Coherence-SubInv2: A core thataccesses a translation (to perform aload or store) must have at least onetoken for that translation.

� Coherence-SubInv3: A core thatperforms an MRF to a translationmust have all T tokens for that trans-lation before completing the MRF(that is, before releasing the lock)and making it visible. This invariantensures that there’s a single point intime at which the old (pre-modified)translation is no longer visible toany cores.

The first two subinvariants are almostidentical to those of Token Coherence.12

The third subinvariant (which is analogousto Token Coherence’s invariant that a coreneeds all tokens to perform a store) is subtlydifferent from Token Coherence, becausean MRF usually isn’t an atomic write. InToken Coherence, a core must hold alltokens throughout the entire lifetime ofthe store, but an MRF only requires thecore to hold all tokens before releasing thelock.

H aving a thorough, multilevel specifi-cation of consistency enables pro-

grammers, designers, and design verifiers tomore easily reason about the memorysystem’s correctness. Furthermore, it facil-itates the development of comprehensivedynamic verification techniques that can, atruntime, detect errors due to design bugsand physical faults.

This work represents an initial explora-tion of this research area. We foresee furtherresearch into VAMC models and addresstranslation systems, as well into relationshipsbetween them. One future avenue of researchis to explore address translation models thatare more relaxed than ATSC, yet still prov-ably sufficient for bridging gaps betweenspecific PAMC and VAMC models. We an-ticipate that address translation can be mademore scalable if it’s less conservative,but more relaxed designs are viable onlyif designers and verifiers can convincethemselves that they’re correct. Our frame-work for specifying VAMC enables theseexplorations. MICRO

AcknowledgmentsThis work was supported in part by Semi-

conductor Research Corporation contract2009-HJ-1881 and the National ScienceFoundation grant CCR-0444516. For theirhelpful feedback on this work, we thankAnne Bracy, Dave Christie, Landon Cox,Stephan Diestelhorst, Anita Lungu, MiloMartin, and Albert Meixner. We thankTrey Cain and Cathy May for help in under-standing issues specific to the Power ISA. Anearlier version of this paper was published atASPLOS 2010.13

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 117

....................................................................


....................................................................References

1. S.V. Adve and K. Gharachorloo, ‘‘Shared

Memory Consistency Models: A Tutorial,’’

Computer, vol. 29, no. 12, 1996, pp. 66-76.

2. L. Lamport, ‘‘How to Make a Multiprocessor

Computer that Correctly Executes Multi-

process Programs,’’ IEEE Trans. Computers,

vol. 28, no. 9, 1979, pp. 690-691.

3. ‘‘Revision Guide for AMD Family 10h Pro-

cessors,’’ tech. report 41322, Advanced

Micro Devices, 2008.

4. IBM, ‘‘IBM PowerPC 750FX and 750FL

RISC Microprocessor Errata List DD2.X,

version 1.3,’’ Feb. 2006.

5. ‘‘Intel Core2 Extreme Quad-Core Processor

QX6000 Sequence and Intel Core2 Quad

Processor Q6000 Sequence Specification

Update,’’ tech. report 315593-021, Intel,

2008.

6. Advanced Micro Devices, ‘‘Revision Guide

for AMD Athlon64 and AMD Opteron Pro-

cessors,’’ Publication 25759, rev. 3.59, 2006.

7. ‘‘Intel Core Duo Processor and Intel Core

Solo Processor on 65nm Process Specifica-

tion Update,’’ tech. report 309222-016,

Intel, 2007.

8. J. Manson, W. Pugh, and S.V. Adve, ‘‘The

Java Memory Model,’’ Proc. 32nd ACM

Sigplan-Sigact Symp. Principles of Program-

ming, ACM Press, 2005, pp. 378-391.

9. M.D. Hill et al., ‘‘A System-Level Specifica-

tion Framework for I/O Architectures,’’

Proc. 11th Ann. ACM Symp. Parallel Algo-

rithms and Architectures, ACM Press,

1999, pp. 138-147.

10. A. Arvind and J.-W. Maessen, ‘‘Memory

Model ¼ Instruction Reordering þ Store

Atomicity,’’ Proc. 33rd Ann. Int’l Symp.

Computer Architecture, IEEE CS Press,

2006, pp. 29-40.

11. P.J. Teller, ‘‘Translation-Lookaside Buffer

Consistency,’’ Computer, vol. 23, no. 6,

1990, pp. 26-36.

12. M.M.K. Martin, M.D. Hill, and D.A. Wood,

‘‘Token Coherence: Decoupling Perfor-

mance and Correctness,’’ Proc. 30th Ann.

Int’l Symp. Computer Architecture, ACM

Press, 2003, pp. 182-193.

13. B.F. Romanescu, A.R. Lebeck, and D.J.

Sorin, ‘‘Specifying and Dynamically Verify-

ing Address Translation-Aware Memory

Consistency,’’ Proc. 15th Int’l Conf. Archi-

tectural Support for Programming Lan-

guages and Operating Systems, ACM

Press, 2010, pp. 323-334.

Bogdan F. Romanescu is a member of theOnline Services Division at Microsoft. Hisresearch interests include memory consis-tency, memory coherence, and scalablecommunication protocols in large-scalesystems. He has a PhD in electrical andcomputer engineering from Duke University.

Alvin R. Lebeck is a professor in theDepartments of Electrical and ComputerEngineering and of Computer Science atDuke University. His interests includearchitectures for emerging nanotechnologies,multicore processors, memory systems, andenergy-efficient computing. He has a PhD incomputer science from the University ofWisconsin, Madison. He’s a senior memberof IEEE and a member of the ACM.

Daniel J. Sorin is an associate professor inthe Departments of Electrical and ComputerEngineering and of Computer Science atDuke University. His research interestsinclude computer architecture, fault toler-ance, memory systems, and design forverifiability. He has a PhD in electricalengineering from the University of Wiscon-sin, Madison. He’s a senior member of IEEEand the ACM.

Direct questions and comments to DanielJ. Sorin, Duke Univ., Box 90291, Durham,NC 27708-0291; [email protected].

[3B2-14] mmi2011010109.3d 20/1/011 15:12 Page 118

....................................................................

118 IEEE MICRO

...............................................................................................................................................................................................

TOP PICKS

Documents

Address Translation Aware Memory Consistency