Formal Verification and its Impact on the Snooping versus Directory Protocol Debate

Milo M. K. MartinDepartment of Computer and Information Science

University of Pennsylvaniamilom@cis.upenn.edu

Abstract

This invited paper argues that to facilitate formal ver-ification, multiprocessor systems should (1) decoupleenforcing coherence from enforcing a memory consis-tency model and (2) decouple the interconnection net-work from the cache coherence protocol (by not rely-ing on any specific interconnect ordering or synchronic-ity properties). Of the two dominant classes of cachecoherence protocols—directory protocols and snoopingprotocols—these two desirable properties favor use ofdirectory protocols over snooping protocols. Althoughthe conceptual simplicity of snooping protocols is seduc-tive, aggressive implementations of snooping protocolslack these decoupling properties, making them perhapsmore difficult in practice to reason about, verify, and im-plement correctly. Conversely, directory protocols mayseem more complicated, but they are more amenable tothese decoupling properties, which simplify protocol de-sign and verification. Finally, this paper describes therecently-proposed token coherence protocol’s adherenceto these properties and discusses some of its implica-tions for future multiprocessor systems.

1 IntroductionAfter years as an exotic and exclusively high-end

technology, shared-memory multiprocessors are nowsolidly mainstream. Even low-end servers routinelyhave at least two processors, and high-end systems of-ten have dozens of processors. With the performanceand power advantages of multi-core chips, multiproces-sors are expanding beyond servers into desktops, lap-tops, handhelds, game consoles, and embedded devices.If current trends continue, multi-core systems will be-come so ubiquitous that in a few years it may not bepossible to purchase a uniprocessor system.

Cache coherence protocols. These shared-memorymultiprocessor systems use a cache coherence protocolto coordinate the caches and memories as part of provid-ing the processors with a consistent view of the contentsof a single shared address space. The exact definitionof this “consistent view” of memory is defined by thesystem’s memory consistency model [2], which is com-monly specified as part of the instruction set architectureof the system. As part of enforcing a consistency model,

invalidation-based cache coherence protocols conceptu-ally maintain a global invariant: for each block of sharedmemory either (1) zero or more processors are allowedto read the block or (2) exactly one processor is allowedto write and read the block.

Snooping versus directory protocol debate. Al-though multiprocessors are now common, there is nota clear consensus on the design of cache coherenceprotocols. Whereas other areas of computer architec-ture have slowly gravitated to canonical designs (e.g.,processor microarchitecture), multiprocessor system de-signs are still widely varied. In fact, a debate on design-ing such systems has raged for decades. Much of thedebate has centered around two classes of cache coher-ence protocols: snooping protocols and directory proto-cols. Snooping protocols use a totally-ordered intercon-nect (e.g., a bus) to broadcast request to all processors,allowing all system components to transition betweencoherence states in a consistent fashion. In contrast, di-rectory protocols send all requests to a directory at thememory, which forwards requests to other processorsas needed. Directory protocols avoid broadcast and atotally-ordered interconnect in exchange for adding in-direction latency to some misses.

The debate revisited. This invited paper containsthoughts and opinions on this debate based on expe-rience with developing and enhancing protocols in anacademic research environment, limited industrial expe-rience, conversations with system designers, and dab-bling in formal verification of such protocols. Insteadof focusing exclusively on the performance or scala-bility of cache coherence protocols, this paper revisitsthe snooping versus directory protocol debate by explic-itly considering the impact of formal verification on de-sign complexity and design verification of multiproces-sor systems. This paper describes the impact of formalverification on the cache coherence protocol design pro-cess (Section 2), distills two desirable decoupling prop-erties for formal modeling and implementing coherenceprotocols (Section 3), revisits the snooping versus direc-tory debate in this context (Section 4), and reflects onthe creation of token coherence [16, 18, 19], a recently-proposed alternative approach to cache coherence (Sec-tion 5). This paper concludes that formal verification

Proceedings of the 2005 International Conference on Computer Design (ICCD’05) 0-7695-2451-6/05 $20.00 © 2005 IEEE

considerations favor choosing either a directory proto-col or a token coherence protocol over all but the sim-plest implementations of snooping-based cache coher-ence protocols.

2 Impact of Formal Verification

In addition to simulation and other forms of tradi-tional design verification, formal verification plays as in-creasingly important role in the design of today’s digitalsystems. Formal verification includes techniques suchas model checking via state space exploration and auto-mated theorem proving techniques. Using formal veri-fication methods has become almost routine for enhanc-ing confidence (and finding bugs) in cache coherenceprotocols [1, 5, 6, 8, 12, 25, 26].

2.1 Explicit Role of Formal Verification

The explicit role of formal verification is finding bugsand improving confidence in the correctness of a design.These methods can either be applied after the high-leveldesign has been completed or during the high-level de-sign phase.

Post-design verification. If formal verification isthought of as part of traditional design verification ef-forts (e.g., simulation-based techniques), formal verifi-cation will likely be applied later in the design and im-plementation of a system. Although late application offormal verification is still effective at finding bugs, itsuffers from first finding many “false bugs” caused byincomplete documentation, design specification errors,and real bugs that were already fixed by the designers[12]. Bugs that are encountered late may also be moredifficult to fix cleanly or the fix may have a significantimpact on the performance or complexity of the design.

During-design verification. In contrast, if formalverification is thought of as an integral part of the de-sign process, formal verification will likely be appliedearly in the design process and have a more perva-sive impact on the actual design. Formal verification ismost effective when used during the initial design phases[8, 12]. Instead of relying on incomplete specifications,the modeling processes can be part of the specificationprocess (or the model may even be part of the actualspecification). Bugs found early can be fixed more easilyand at the appropriate level of the design. For example,instead of adding a localized timing-sensitive kludge tofix a subtle coherence protocol race, early detection ofsuch a bug could have resulted in a higher-level designchange that would have been simpler, cleaner, and eas-ier. In addition, a set of bugs discovered early may ac-tually cause the designers to rethink or adjust the high-level design.

2.2 Implicit Role of Formal Verification

If verification is an integral part of the high-leveldesign process, formal verification has several implicitbenefits—benefits that arise not from the actual compu-tational verification of the model, but from the creationof the model itself. Creating a model requires one ormore engineers to think systematically about the correct-ness of the design and the accuracy of its specification.1

In addition, the designers themselves may be influencedby the knowledge that whatever the designers propose,someone will need to model it.2 This subtle effect caninfluence the designers to create a more modular designwith clearer abstraction layers and better specificationand documentation. Knowing that a design is going tobe formally verified may encourage designers to actu-ally document their designs and employ “principles ofgood design” that they otherwise might not be disci-plined enough to self-enforce. In essence, placing theextra design constraint of “verifiability” on the design-ers in some cases may produce a better design.

3 Two Desirable Properties of Cache Coher-ence Protocols

This section describes two properties found in somecoherence protocols that give them advantages in termsof tractability of formal verification, design complexity,and ability to be modified for future design iterations.

3.1 Decouple Coherence from Consistency

The definition of correctness of a multiprocessormemory system is its memory consistency model [2].Informally, coherence refers only to the behavior ofa single memory location, whereas the memory con-sistency model encompasses the ordering and interac-tion of memory operations to multiple memory loca-tions. Because individually verifying either a consis-tency model or simple coherence properties is a difficulttask, we advocate clearly decoupling these two aspectsof a multiprocessor memory system to simplify verifica-tion. To enable this decoupling, the coherence protocolshould provide a coherence interface sufficient to pro-vide a serializable view of memory. The processor coreshould interact with that interface to provide whatevermemory consistency model the processor design team

1For example, when talking with a verification engineer about onespecific verification effort, I was told that the only critical bug foundduring the formal verification process was encountered during con-struction of the model; the automated checking of the model did verifythe bug and its fix, but it did not discover any other significant bugs.

2For example, after suggesting a specific cache coherence protocolproposal to one industrial designer, the designer’s face went pale, andhe said something like “the verification team would never let us getaway with that”.

Proceedings of the 2005 International Conference on Computer Design (ICCD’05) 0-7695-2451-6/05 $20.00 © 2005 IEEE

deems necessary (using prefetching and various specu-lative techniques to aggressively implement the consis-tency model [3, 9]).

In such a system, the only role of the coherence pro-tocol is to inform the processor when it can read a blockand when it can write a block. This notion closely corre-sponds with the multiple-reader-single-writer coherenceinvariant described in the introduction. Such a coher-ence protocol (1) provides a simple interface to the pro-cessor and (2) allows the processor to aggressively im-plement the desired consistency model. To clarify thesepoints, consider the implementation of memory barrieroperations (used to establish memory ordering). A sys-tem that follows the above decoupling property can effi-ciently handle memory barriers entirely within the pro-cessor core, freeing the coherence protocol from provid-ing special-purpose operations with complicated seman-tics.

Although some systems have at least partially adopteda decoupled consistency and coherence philosophy (e.g.,[4, 7, 13]), many systems have not (e.g., [10, 27]). Thosesystems intertwine coherence and consistency through-out the system, often relying on a patchwork of specialordering properties of processor queues and a total or-dering of requests via a totally-ordered interconnect tocarefully and delicately orchestrate the desired consis-tency model (discussed further below).

3.2 Decouple Interconnect from ProtocolThe interconnect is the communication mechanism

that reliably delivers messages between the caches andmemories of a multiprocessor system. A cache co-herence protocol might rely on an interconnect thatprovides (1) point-to-point ordering, (2) totally-orderedmessage delivery, (3) no special ordering properties. Wecontend that to ease verification, the coherence protocolshould not depend upon any special ordering propertiesof the interconnect. To support this position, this sectiondiscusses some of the difficulties of designing and ver-ifying coherence protocols that rely on these two typesof interconnect orderings.

Avoid point-to-point ordered interconnects. Of thetwo types of interconnect ordering properties, point-to-point ordering is less problematic than totally-orderedinterconnects. However, point-to-point ordering is un-desirable because (1) it constrains interconnect design(e.g., by precluding or limiting adaptive routing) and (2)it complicates formal verification by encumbering themodel and increasing the state space by reducing sym-metry (as compared with modeling an unordered inter-connect) [12]. Fortunately, many protocols—especiallydirectory protocols—do not rely on any sort of inter-

connect ordering (e.g., [24]). However, some protocolsrequire at least point-to-point ordering (e.g., [28]), andmany protocols require the even more onerous totally-ordered interconnect (as discussed next).

Avoid totally-ordered interconnects. Relying on atotally-ordered interconnect has pervasive design ram-ifications. Depending on such an interconnect is unde-sirable because it intertwines the processing of differ-ent addresses. An interconnect provides a total orderingof messages if all messages are delivered to all destina-tions in some order. A total ordering requires an order-ing among all the messages (even those from differentsources or sent to different destinations). For example,if any processor receives message A before message B,then no processor receives message B before A. Manyprotocols (e.g., most snooping protocols and some di-rectory protocols [10]) require an interconnect that pro-vides a total ordering of requests. Unfortunately, estab-lishing a total ordering of requests can add complexity,increase cost, and increase latency. For example, totally-ordered interconnects commonly use some centralizedroot switch or arbitration mechanism, and such mecha-nisms are not a good match for direct interconnects.

In protocols that do not rely on a total ordering ofrequests—e.g., traditional directory protocols [14, 24,28] and AMD’s Opteron protocol [4]—requests and re-sponses for different blocks have no need to unduly in-fluence each other. Only when a processor decides tocommit a read or write to a block does the processorneed to consider the interactions of reads and writesto other blocks. In contrast, in systems that rely on atotal ordering of requests, requests (and sometimes re-sponses) for different addresses must be kept in orderwith respect to each other throughout the system.3 Thisrequirement forces cache controllers to process all re-quests in order, making banking of coherence controllersto increase bandwidth difficult. To use a uniprocessoranalogy, relying on a total ordering of requests intro-duces “dependences” (both true and false dependencies)between requests for various blocks.

3.3 Discussion and ImplicationsThe decoupling of coherence and consistency and

the decoupling of protocol and interconnect results ina much more modular design. Such a design has sim-pler interfaces between the various system components,which should substantially simplify verification (bothformal verification and traditional verification). For ex-ample, each component can be designed and verified in

3For further discussion of the subtle implications and interactionsof memory consistency models and totally-ordered interconnects, seechapters 2 and 10 of Martin [16].

Proceedings of the 2005 International Conference on Computer Design (ICCD’05) 0-7695-2451-6/05 $20.00 © 2005 IEEE

isolation, and a high-level model can be used to verifymany of their interactions.

In addition, such a design is more amenable to in-cremental improvements for future product generationsbased on the same basic design. Because of the hugeeffort to fully design, verify, and performance debug ahigh-performance design, a design will often be usedfor several product generations. For example, the initial“P6” core design of the Pentium Pro has been improvedand adapted over many years to be used in desktop chips(Pentium II and Pentium III), server chips (Xeon), andmost recently in low-power mobile chips (Pentium M).Similar derivations occur with multiprocessor systems.A multiprocessor that adheres to these two decouplingproperties will be easier to adapt over time, because anindividual component (e.g., the interconnect or cachecontroller) can be improved with fewer global interac-tions than in a less modular design.

As an example of non-localized cascading changes,consider a first-generation design with an interconnectthat does not implement adaptive routing (and thus im-plicitly provided point-to-point interconnect ordering);if the coherence protocol exploits that point-to-point or-dering, the second-generation system cannot implementadaptive routing in the interconnect unless the coherenceprotocol is also redesigned. However, if the system de-signers had decoupled the interconnect ordering fromthe protocol, such a change to the interconnect wouldbe a purely local change.

Although these two decoupling properties seem likelaudable and desirable properties, they also may be con-troversial, as some proposals have explicitly argued theopposite approach [10], and systems such as IBM’sPower4 [27] and AlphaServer GS320 [10] designs haveneither property. Both of these systems use an or-dered interconnect and intermingle coherence and con-sistency. For example, Power4 relies on ordering inthe interconnect and requires broadcasting of certainmemory-ordering barrier operations. In contrast, the Al-pha 21364/GS1280 and AMD Opteron protocols bothappear to exhibit both decoupling properties describedabove (based on the limited published descriptions ofthe protocols [4, 7, 24]).

4 Revisiting Snooping vs. Directory ProtocolsThe choice of coherence protocol is as subtle and con-

troversial today as it has ever been. Instead of focusingprimarily on performance or scalability, this section re-visits the snooping versus directory protocol debate byconsidering the impact of formal verification and the twodesirable decoupling properties described in the last sec-tion. Although any high-performance implementationof a multiprocessor memory system is complicated, we

contend that directory protocols are preferable from acomplexity and formal verification viewpoint, becausedirectory protocols are more amenable to decoupling co-herence from consistency and decoupling the intercon-nect from the coherence protocol.

Snooping protocols. Although snooping is seduc-tively simple from a conceptual viewpoint, as snoop-ing implementations have evolved over time theyhave become anything but simple. Real-world high-performance implementations are not simple becausethey use many advanced techniques such as snoopresponse combining, split-transaction protocols, splitrequest/response interconnects, and multiple totally-ordered switched interconnects. These enhancementstightly couple the timing and ordering of the intercon-nect with the protocol, and they couple coherence andconsistency. In essence, aggressive snooping proto-cols are complex and difficult to verify because they donot exhibit the two desirable decoupling properties de-scribed in the last section.

Directory protocols. In contrast, even the simplestdirectory protocol may seem complicated (e.g., becauseof the number of request and writeback races that canoccur). However, these situations are exactly the sort ofhigh-level protocol issues that formal verification meth-ods can help overcome, especially if applied early inthe design. Directory protocols are more amenable toformal verification techniques because they often ex-hibit these two decoupling properties. These propertiesalso confer such protocols better “complexity scalabil-ity” over time. That is, as a directory protocol imple-mentation becomes more aggressive—faster intercon-nects, more outstanding requests, more protocol statesand optimizations—the changes are more localized.

Why then have directory protocols not been usedmore frequently? One possible explanation is that di-rectory protocols were promoted—both in academic re-search and in industrial designs—primarily for beingscalable (i.e., support hundreds or thousands of proces-sors).4 In fact, the term “scalable cache coherence” issynonymous with a directory protocol. Such emphasison scalability and their first-glance complexity may havegiven directory protocols a reputation as an exotic andexpensive technology. Perhaps this reputation has leddesigners to be more comfortable evolving their exist-ing snooping-based system over the years, especially asmost systems sold—even of “scalable” systems—have amodest number of processors [23]. Transitioning to a di-rectory protocol represents a more radical design change

4See Hill [11] for an early skeptical view of scalability.

Proceedings of the 2005 International Conference on Computer Design (ICCD’05) 0-7695-2451-6/05 $20.00 © 2005 IEEE

than just evolving an existing snooping system.5 In ad-dition, many early directory systems were hierarchicalmultiprocessors built from snooping-based multiproces-sor building blocks (e.g., [14, 15, 28]), resulting in a rep-utation of substantial complexity and large latency over-head from bridging between the two protocols. Finally,performance issues may have played a role (discussedbriefly in the next section).

5 A New Alternative: Token CoherenceAlthough we encourage designers to consider direc-

tory protocols over snooping protocols for complexityand design verification reasons, directory protocols havean important performance disadvantage that must be ad-dressed: directory protocols add indirection latency tocache-to-cache misses. To resolve races, a directory pro-tocol sends all requests to a home node that then for-wards the request (if needed) or responds with the datafrom memory. In contrast, a snooping protocol avoidsindirection by broadcasting all requests to all nodes andrelying on interconnect ordering to help resolve races.Although the additional indirection latency of a direc-tory protocol can be partially mitigated by using a di-rectory cache, an extra interconnect traversal remains onthe critical path of some cache misses.

The recently-proposed token coherence protocol [16,18, 19] can eliminate the constraint of directory indirec-tion without sacrificing either decoupling of the the in-terconnect from the coherence protocol or decoupling ofcoherence from consistency. Token coherence uses to-ken counting to resolve races without requiring a homenode or an ordered interconnect. Token coherence em-braces even further levels of decoupling by separatingthe correctness substrate from the system’s performancepolicy. The correctness substrate is further decoupledinto enforcing safety and avoiding starvation.

Enforcing safety. Token coherence’s correctnesssubstrate provides safety by counting tokens for eachblock of memory in the system. Each block in the sys-tem has a fixed number of tokens (T ). If a processor’scache has all T tokens for the block, it is allowed to readand write the block. If a processor’s cache has at leastone token, it can read the block (but not write it). Ifa processor’s cache holds no tokens, it can neither readnor write the block. These token counting rules directlyensure that while one processor is writing the block, noother processor is reading or writing it. In essence, it di-rectly enforces the multiple-reader-single-writer coher-

5In fact, some of my earlier research took this evolutionary ap-proach to trying to enhance snooping protocols to mitigate their disad-vantages (e.g., [20, 21]). However, I now believe the disadvantages ofdirectory protocols are perhaps more easily mitigated than the disad-vantages of snooping protocols.

ence invariant suitable for allowing the processor to en-force the desired memory consistency model. Such sim-ple rules allow for reasoning about protocol safety in amuch simpler fashion, and by its nature, token coher-ence does not rely upon complicated ordering propertiesof the interconnect or the use of a directory home nodeto resolve races.

Avoiding starvation. Although token counting en-sures safety, it does not ensure that a request is even-tually satisfied. Thus the correctness substrate providespersistent requests to prevent starvation. When a proces-sor detects possible starvation (such as via a time-out), itinitiates a persistent request. The substrate then activatesat most one persistent request per block, using a fair ar-bitration mechanism. Each system node remembers allactivated persistent requests (for example, in a table ateach node) and forwards all tokens for the block—thosetokens currently present and received in the future—tothe request initiator. Finally, when the initiator has suffi-cient tokens, it performs a memory operation (a load orstore instruction) and deactivates its persistent request.

Performance policies. The correctness substrate pro-vides a foundation for implementing many performancepolicies. These performance policies focus on makingthe system fast and bandwidth-efficient, but have no cor-rectness responsibilities, because the substrate is respon-sible for correctness. This decoupling of responsibil-ity between the correctness substrate and performancepolicy enables the development of performance policiesthat capture many of the desirable attributes of snoop-ing and directory protocols. For example, token co-herence performance policies have been developed [16]to approximate an unordered broadcast-based protocol(inspired by snooping protocols), a bandwidth-efficientperformance policy that emulates a directory protocol,and a predictive hybrid protocol that uses destination-setprediction [17].

Ramifications on design verification. As token co-herence’s use of decoupling goes beyond the two de-coupling properties described in Section 3, it perhapshas some additional verification advantages. For exam-ple, Marty et al. [22] used a single-level token coher-ence protocol to approximate the performance charac-teristics of a two-level hierarchical coherence protocol.That is, the protocol is flat for correctness but hierar-chical for performance. Using token coherence in thisfashion allows for the performance benefits of a hierar-chical protocol combined with the ease of verification ofa single-level protocol. Marty et al. [22] also show thatthe difficulty of verifying the token coherence correct-ness substrate is comparable to verifying a single-leveldirectory protocol. In addition, the flexibility provided

Proceedings of the 2005 International Conference on Computer Design (ICCD’05) 0-7695-2451-6/05 $20.00 © 2005 IEEE

by the performance policy should allow a system using atoken coherence protocol to be enhanced over time with-out substantial changes to the correctness substrate.

6 ConclusionsIn this paper, we reflected on the impact of the in-

creasing use of formal design verification methods dur-ing the early stages of multiprocessor system design. Weidentified two desirable decoupling properties for reduc-ing the complexity and enhancing the verifiability of thesystem: decoupling coherence from consistency and de-coupling the interconnect ordering properties from thecache coherence protocol. We used these properties torevisit the snooping versus directory protocol debate,and we argued that these decoupling properties point todirectory protocols as being more attractive that snoop-ing protocols from a verifiability and modifyability pointof view. Finally, we identified token coherence as a pos-sible approach for (1) further simplifying the verifiabil-ity of coherence protocols and (2) overcoming the in-direction performance penalty found in directory proto-cols. As multi-core designs are becoming ubiquitous,we encourage the designers of these systems to look be-yond simple bus-based snooping-based designs and toconsider directory protocols and token coherence as ap-proaches to creating more verifiable and adaptable de-signs.

AcknowledgmentsThe author thanks Rajeev Alur, Sebastian Burckhardt,

Jesse Bingham, Mark Hill, Alan Hu, and David Woodfor helpful discussions leading up to this paper. Thiswork is funded in part by a gift from Intel Corporation.

References[1] D. Abts, D. J. Lilja, and S. Scott. Toward Complexity-

Effective Verification: A Case Study of the CraySV2 Cache Coherence Protocol. In 1st Workshop onComplexity-Effective Design held in conjunction with the27th International Symposium on Computer Architec-ture, June 2000.

[2] S. V. Adve and K. Gharachorloo. Shared Memory Con-sistency Models: A Tutorial. IEEE Computer, 29(12):66–76, Dec. 1996.

[3] S. V. Adve, V. S. Pai, and P. Ranganathan. Recent Ad-vances in Memory Consistency Models for HardwareShared Memory Systems. Proceedings of the IEEE, 87(3):445–455, Mar. 1999.

[4] A. Ahmed, P. Conway, B. Hughes, and F. Weber. AMDOpteron Shared Memory MP Systems. In Proceedingsof the 14th HotChips Symposium, Aug. 2002.

[5] H. Akhiani, D. Doligez, P. Harter, L. Lamport, J. Scheid,M. Tuttle, and Y. Yu. Cache Coherence Verification withTLA+. In FM’99—Formal Methods, Volume II, volume

1709 of Lecture Notes in Computer Science, page 1871.Springer Verlag, 1999.

[6] E. M. Clarke and J. M. Wing. Formal Methods: State ofthe Art and Future Directions. ACM Computing Surveys,28(4):626–643, Dec. 1996.

[7] Z. Cvetanovic. Performance analysis of the Alpha21364-based HP GS1280 multiprocessor. In Proceedingsof the 30th Annual International Symposium on Com-puter Architecture, pages 218–229, June 2003.

[8] D. L. Dill, A. J. Drexler, A. J. Hu, and C. H. Yang. Proto-col Verification as a Hardware Design Aid. In 1992 IEEEInternational Conference on Computer Design: VLSI inComputers and Processors, pages 522–525, 1992.

[9] K. Gharachorloo, A. Gupta, and J. Hennessy. Two Tech-niques to Enhance the Performance of Memory Consis-tency Models. In Proceedings of the International Con-ference on Parallel Processing, volume I, pages 355–364, Aug. 1991.

[10] K. Gharachorloo, M. Sharma, S. Steely, and S. V. Doren.Architecture and Design of AlphaServer GS320. In Pro-ceedings of the Ninth International Conference on Archi-tectural Support for Programming Languages and Oper-ating Systems, pages 13–24, Nov. 2000.

[11] M. D. Hill. What is Scalability? Computer ArchitectureNews, 18(4):18–21, 1990.

[12] A. J. Hu, M. Fujita, and C. Wilson. Formal Verification ofthe HAL S1 System Cache Coherence Protocol. In Pro-ceedings of the International Conference on ComputerDesign, pages 438–444, Oct. 1997.

[13] J. Laudon and D. Lenoski. The SGI Origin: A cc-NUMA Highly Scalable Server. In Proceedings of the24th Annual International Symposium on Computer Ar-chitecture, pages 241–251, June 1997.

[14] D. Lenoski, J. Laudon, K. Gharachorloo, W.-D. Weber,A. Gupta, J. Hennessy, M. Horowitz, and M. Lam. TheStanford DASH Multiprocessor. IEEE Computer, 25(3):63–79, Mar. 1992.

[15] T. D. Lovett and R. M. Clapp. STiNG: A CC-NUMAComputer System for the Commercial Marketplace. InProceedings of the 23th Annual International Symposiumon Computer Architecture, May 1996.

[16] M. M. K. Martin. Token Coherence. PhD thesis, Univer-sity of Wisconsin, 2003.

[17] M. M. K. Martin, P. J. Harper, D. J. Sorin, M. D. Hill,and D. A. Wood. Using Destination-Set Prediction to Im-prove the Latency/Bandwidth Tradeoff in Shared Mem-ory Multiprocessors. In Proceedings of the 30th An-nual International Symposium on Computer Architec-ture, pages 206–217, June 2003.

Proceedings of the 2005 International Conference on Computer Design (ICCD’05) 0-7695-2451-6/05 $20.00 © 2005 IEEE

[18] M. M. K. Martin, M. D. Hill, and D. A. Wood. Token Co-herence: A New Framework for Shared-Memory Multi-processors. IEEE Micro, November-December 2003.

[19] M. M. K. Martin, M. D. Hill, and D. A. Wood. Token Co-herence: Decoupling Performance and Correctness. InProceedings of the 30th Annual International Symposiumon Computer Architecture, pages 182–193, June 2003.

[20] M. M. K. Martin, D. J. Sorin, A. Ailamaki, A. R.Alameldeen, R. M. Dickson, C. J. Mauer, K. E. Moore,M. Plakal, M. D. Hill, and D. A. Wood. TimestampSnooping: An Approach for Extending SMPs. In Pro-ceedings of the Ninth International Conference on Archi-tectural Support for Programming Languages and Oper-ating Systems, pages 25–36, Nov. 2000.

[21] M. M. K. Martin, D. J. Sorin, M. D. Hill, and D. A.Wood. Bandwidth Adaptive Snooping. In Proceedings ofthe Eighth IEEE Symposium on High-Performance Com-puter Architecture, pages 251–262, Feb. 2002.

[22] M. R. Marty, J. D. Bingham, M. D. Hill, A. J. Hu,M. M. K. Martin, and D. A. Wood. Improving Multiple-CMP Systems Using Token Coherence. In Proceedingsof the 11th IEEE Symposium on High-Performance Com-puter Architecture, Feb. 2005.

[23] J. R. Mashey. NUMAflex Modular Design Approach:A Revolution in Evolution. Posted on comp.arch newsgroup, Aug. 2000.

[24] S. S. Mukherjee, P. Bannon, S. Lang, A. Spink, andD. Webb. The Alpha 21364 Network Architecture. InProceedings of the 9th Hot Interconnects Symposium,Aug. 2001.

[25] F. Pong, M. Browne, A. Nowatzyk, and M. Dubois. De-sign Verification of the S3.mp Cache-Coherent Shared-Memory System. IEEE Transactions on Computers, 47(1):135–140, Jan. 1998.

[26] F. Pong and M. Dubois. Verification Techniques forCache Coherence Protocols. ACM Computing Surveys,29(1):82–126, Mar. 1997.

[27] J. M. Tendler, S. Dodson, S. Fields, H. Le, and B. Sin-haroy. POWER4 System Microarchitecture. IBM Jour-nal of Research and Development, 46(1), 2002.

[28] W.-D. Weber, S. Gold, P. Helland, T. Shimizu, T. Wicki,and W. Wilcke. The Mercury Interconnect Architecture:A Cost-Effective Infrastructure for High-PerformanceServers. In Proceedings of the 24th Annual InternationalSymposium on Computer Architecture, June 1997.

Proceedings of the 2005 International Conference on Computer Design (ICCD’05) 0-7695-2451-6/05 $20.00 © 2005 IEEE