25
Peer-to-Peer Netw. Appl. DOI 10.1007/s12083-013-0231-5 A survey of peer-to-peer overlay approaches for networked virtual environments Eliya Buyukkaya · Maha Abdallah · Gwendal Simon Received: 20 January 2013 / Accepted: 30 August 2013 © Springer Science+Business Media New York 2013 Abstract This paper identifies and analyzes the key design approaches that have been proposed over the last decade to construct a peer-to-peer (P2P) networked virtual envi- ronment (NVE) systems. We discuss each approach, their advantages and drawbacks, and examine several represen- tative P2P-based NVE systems based on the presented approaches, thus providing the background and literature review of P2P NVEs. Keywords Networked virtual environments · P2P systems 1 Introduction During the past decade, the academic community has exten- sively studied peer-to-peer (P2P) networked virtual environ- ments (NVEs). This research activity has been driven by the phenomenal success of other P2P applications, typified by file-sharing applications (e.g., Bit-Torrent), and video streaming systems (e.g., PPLive). However P2P architec- tures have not yet found their ways in NVEs, partly because the development of fully-distributed NVEs has turned out to be especially challenging. In this paper, we review over ten years of research related to the design of fully distributed infrastructure-less NVEs. Our goal is to give a thorough E. Buyukkaya () ENSSAT, University Rennes 1, Rennes, France e-mail: [email protected] M. Abdallah LIP6, Pierre and Marie Curie University, Paris, France e-mail: [email protected] G. Simon Institut Telecom, Telecom Bretagne, Plouzan´ e, France e-mail: [email protected] description of the main approaches that have been explored so far, and to emphasize their main advantages and their weaknesses. We think that it is the right time to survey in a fair manner these approaches because technological shifts, in particular the development of cheap computing and stor- age facilities in data-centers, have made P2P NVE systems less attractive. We discuss the perspectives around these works for the next years. We believe that distributed algo- rithms for the management of large-scale NVEs are still needed, in a context that is just slightly different. We discuss these perspectives at the end of the paper in a more personal opinion. 1.1 Fundamentals about NVEs Networked Virtual Environments (NVEs) can be classi- fied into three major categories: multiplayer online games, military simulations, and social virtual worlds. Among the most popular NVEs, multiplayer online games are computerized games in which a high num- ber of players play in the same game environment at the same time. Multiplayer online games can be further classified into First Person Shooter, Real-Time Strat- egy and Role-Playing games. Below are some typical examples: First Person Shooter: Doom [1], Quake Live [2], Marathon [3], Blake Stone: Aliens of Gold [4], Star Wars: Dark Forces [5]. Real-Time Strategy: Homeworld [6], Starcraft II: Wings of Liberty [7], Company of Heroes [8]. Role-Playing: Ultima Online [9], EverQuest [10], Final Fantasy XI [11], EVE online [12], World of Warcraft [13].

A survey of peer-to-peer overlay approaches for networked virtual environments

  • Upload
    gwendal

  • View
    214

  • Download
    0

Embed Size (px)

Citation preview

Page 1: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.DOI 10.1007/s12083-013-0231-5

A survey of peer-to-peer overlay approaches for networkedvirtual environments

Eliya Buyukkaya · Maha Abdallah · Gwendal Simon

Received: 20 January 2013 / Accepted: 30 August 2013© Springer Science+Business Media New York 2013

Abstract This paper identifies and analyzes the key designapproaches that have been proposed over the last decadeto construct a peer-to-peer (P2P) networked virtual envi-ronment (NVE) systems. We discuss each approach, theiradvantages and drawbacks, and examine several represen-tative P2P-based NVE systems based on the presentedapproaches, thus providing the background and literaturereview of P2P NVEs.

Keywords Networked virtual environments · P2P systems

1 Introduction

During the past decade, the academic community has exten-sively studied peer-to-peer (P2P) networked virtual environ-ments (NVEs). This research activity has been driven bythe phenomenal success of other P2P applications, typifiedby file-sharing applications (e.g., Bit-Torrent), and videostreaming systems (e.g., PPLive). However P2P architec-tures have not yet found their ways in NVEs, partly becausethe development of fully-distributed NVEs has turned out tobe especially challenging. In this paper, we review over tenyears of research related to the design of fully distributedinfrastructure-less NVEs. Our goal is to give a thorough

E. Buyukkaya (�)ENSSAT, University Rennes 1, Rennes, Francee-mail: [email protected]

M. AbdallahLIP6, Pierre and Marie Curie University, Paris, Francee-mail: [email protected]

G. SimonInstitut Telecom, Telecom Bretagne, Plouzane, Francee-mail: [email protected]

description of the main approaches that have been exploredso far, and to emphasize their main advantages and theirweaknesses. We think that it is the right time to survey in afair manner these approaches because technological shifts,in particular the development of cheap computing and stor-age facilities in data-centers, have made P2P NVE systemsless attractive. We discuss the perspectives around theseworks for the next years. We believe that distributed algo-rithms for the management of large-scale NVEs are stillneeded, in a context that is just slightly different. We discussthese perspectives at the end of the paper in a more personalopinion.

1.1 Fundamentals about NVEs

Networked Virtual Environments (NVEs) can be classi-fied into three major categories: multiplayer online games,military simulations, and social virtual worlds.

• Among the most popular NVEs, multiplayer onlinegames are computerized games in which a high num-ber of players play in the same game environment atthe same time. Multiplayer online games can be furtherclassified into First Person Shooter, Real-Time Strat-egy and Role-Playing games. Below are some typicalexamples:

– First Person Shooter: Doom [1], Quake Live [2],Marathon [3], Blake Stone: Aliens of Gold [4], StarWars: Dark Forces [5].

– Real-Time Strategy: Homeworld [6], Starcraft II:Wings of Liberty [7], Company of Heroes [8].

– Role-Playing: Ultima Online [9], EverQuest [10],Final Fantasy XI [11], EVE online [12], World ofWarcraft [13].

Page 2: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

• NVEs originated from military simulations designed fortraining purposes. Typical examples are SIMNET [14],DIS [15], HLA [16], FALCON 4.0 [17] and America’sArmy [18].

• Social virtual worlds are online communities livingin computerized virtual environments and aiming atsocializing with other users via mutual interactionswithin the virtual world. Typical examples are Sec-ond Life [19], HiPiHi [20], The Sims Online [21],There [22] and Active Worlds [23].

A typical virtual world is made up of immutable objects(e.g., terrain geometry, textures), mutable objects (e.g.,food, weapons, tools), characters controlled by players (i.e.,avatars) and non-player characters controlled by automatedalgorithms [24]. The goal of a NVE system is to provide toits participants a shared sense of space, time, and presencethrough virtual representations, called avatars. This idea hasderived into a set of non-functional requirements. In partic-ular, users observing the same virtual scene need to havethe same view on data at one time (i.e., consistency). ANVE should respond to users’ events at a fast rate (i.e.,responsiveness), since high latencies degrade user experi-ence; for instance, first person shooter games cannot toleratelatencies above 200 milliseconds [25]. In addition, usersshould be prevented from cheating, and users’ data shouldbe protected from being changed (intentionally or uninten-tionally) by other users (i.e., security). Content within NVEsshould also remain available across different user sessions,so in-world objects’/avatars’ data should be stored on a per-sistent storage device (i.e., persistence). NVEs should beable to survive failures in order to continuously provide theexpected service (i.e., availability). As NVEs are becomingincreasingly popular, they are expected to support a highnumber of simultaneous participants (i.e., scalability), andconsequently deal with potentially highly populated regionsin the virtual world (i.e., load-balancing).

To address these requirements, NVEs have traditionallybeen developed on client/server architectures. The server(or a cluster of servers) maintains the world data, calculatesstate changes and disseminates state updates to the differ-ent users. A user informs the server about the events thathe/she creates and is informed by the server about statechanges of players and objects in the virtual world. Since aplayer is only interested in the portion of the virtual worldthat is located near the player’s avatar, the server usuallyonly sends information about events that occurred within theArea of Interest (AoI).

1.2 Peer-to-peer NVEs

The scientific community has advanced peer-to-peer (P2P)systems as an alternative design choice for building NVEs.

We identify three main motivations behind this choice. Firstis the perspective of cheap scalability. Centralized sys-tems can indeed lead to communication and computationaloverhead on the servers. To support an ever growing userpopulation, the only solution is to increase the number ofservers. To get rid of expensive infrastructure has become akey driver for scientists. The second motivation is related toavoid possible points of failure. Fully distributed architec-tures exhibit extra-ordinary self-healing and self-stabilizingproperties. Since maintaining the redundancy of such large-scale applications is hard, scientists proposed P2P solutions.Finally, the third motivation refers to the libertarian vision.Like in social networks, users are concerned by the factthat a central authority owns the VE, tracks all the activ-ities of users, potentially uses logs for profit, and mightuse censorships on the activities within the virtual world. AP2P infrastructure appears as a way to build an authority-less NVE, where users are free to act and have controlover several important aspects of their participation tothe NVE.

In P2P systems, the overall system load is distributedamong all participating users/peers, which organize them-selves so that they all have an equal role and act as bothclients and servers. Since in a virtual world a player is onlyinterested in its AOI, a key issue in a NVE is to answer thefollowing question: how can each player be informed of thestate changes of other players and objects within its AOIwhile moving around the world? In other words, how can astate update message be sent to relevant players?

To add to complexity, P2P NVEs bring some addi-tional constraints that are inherent to the fact that thesoftware is running on the machines of end-users. Giventhe relatively limited (upload) bandwidth and processingpower of users’ machines, P2P NVE designers tend tolimit the number of connections as well as the numberof exchanged messages between participating peers. More-over, network latency and message losses make maintainingstrict consistency extremely challenging. Finally, direct con-nections between interacting players should be preferredbecause the transmission of a message through severalintermediate peers leads to high message disseminationdelays.

1.3 The idea behind this survey

As illustrated by the number of references in the presentdocument, numerous P2P-based NVE systems have beenproposed. In this paper, we focus on recent efforts to buildP2P overlay networks for the full support of NVEs. Thismeans that, for instance, special mechanisms that deal withphysical network latency are out of the scope of this surveyeven if they refer to both P2P and NVEs. In other words,we do not review the works that improve one specific

Page 3: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

non-functional requirement in coordination with a tradi-tional client-server system. Instead, we are interested in sys-tems that build their own software infrastructure by lever-aging exclusively on the machines of end users. At a timewhere large-scale distributed systems are more than everdeployed, we believe that leveraging on lessons learned andexperiences gained from a decade of intensive efforts andscientific works through a comprehensive state-of-the-art isnecessary. Most of these previous works did not envision tobe used in what is now known as the cloud. Nonetheless,they are of high interest in the perspective of deployment ofNVEs in large-scale distributed infrastructures.

We identify six key approaches to construct the over-lay network: the world partitioning, DHT, multicast, fully-connected, neighbor-list exchange and mutual notificationapproaches. Given the plethora of existing efforts in thesedirections, the present paper is not intended to be exhaus-tive of all existing related works, but rather to study, throughrepresentative examples of each approach, the general orga-nization of the overlay and the message schemes that suchoverlay implies. More precisely, we describe (i) the mech-anisms that allow the overall system to stay connected, (ii)

the main schemes for message exchange, and (iii) the distri-bution of data. We summarize at the end of each Section themain advantages and drawbacks of the approach. We discusstheir comparative interest for the future in the Discussionpart.

The remainder of this paper is organized as follows.Section 2 provides a brief overview of the different P2PNVE approaches. Sections 3, 4, 5, 6, 7 and 8 providea detailed discussion of each approach, its advantagesand drawbacks, and examine several representative P2PNVE systems based on the presented approaches. Finally,Section 9 provides a critical analysis of the presentedapproaches and Section 10 concludes this work.

2 P2P NVE approaches: A brief overview

The different P2P NVE approaches are briefly introduced inthe following.

World partitioning approach partitions the virtual worldinto regions, which group a set of players potentially inter-ested in each other. The manager of a region manages peersand objects inside its region. The main issues are effi-cient sized partitioning of the world and the management ofplayer crowding.

DHT approach uses Distributed hash tables (DHTs), whichprovide peer connectivity through a dynamic logical over-lay, as well as efficient lookup through a hash table-likefunctionality and load balancing of the overall system’s

load. The open issues are the fair distribution of the sys-tem load among peers, and latency due to the number ofhops associated with every message over the DHT keyspace.

Multicast approach is used to disseminate update messagesto relevant peers (i.e., multicast group) in the virtual world.A multicast group is formed for each world region or entity(e.g. avatar, object) to inform group members about corre-sponding state changes. P2P-based NVE systems proposeboth IP multicast and application-layer multicast (ALM).However, IP multicast is not widely deployed due to groupand network management issues. In ALM, the managementof multicast groups is still challenging, due to a high num-ber of group members as well as highly dynamic players inthe virtual world.

Fully-connected approach connects all peers to each otherin the VE while using message filtering techniques to limitmessage transmissions. Players thus frequently send mes-sages to only a small group of players, while infrequently toall other players in the world. Fully-connected systems donot scale as each player needs to communicate with all otherplayers.

Neighbor-list exchange approach connects peers closest toeach other in the virtual world. Directly connected peersregularly inform each other about not only themselves butalso their own neighbors through neighbor-list exchange.However, global network connectivity is not guaranteedand constant neighbor-list exchanges may introduce a highmessage overhead.

Mutual notification approach connects nearby peers inthe VE usually based on well-known geometrical struc-tures (e.g., Voronoi diagram). Global network connectivityand neighbor discovery are realized only when necessarythrough mutual notification among direct neighbors. How-ever, the underlying overlay might induce high maintenancecost as it is subject to high connection change rate due to thecontinuous movement of players.

3 World partitioning approach

The partitioning of the virtual world into regions wasthe first used method to construct a P2P NVE system[26–29]. Each region is assigned to a manager peer whichis a peer with sufficient resources in terms of computingpower, bandwidth and storage capacity, responsible for themanagement of the region. A region groups a small numberof entities potentially in relation to each other (e.g., play-ers potentially interested in each other and objects in the

Page 4: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

region). However, NVEs can be supported by the world par-titioning approach under the assumption that the resourcesof each manager are sufficient to manage its region.

The manager peer manages the data of objects of itsregion. The data of an avatar on the other hand can be keptby either the manager of the region inside which the avatarresides [30] or the peer controlling the avatar [24]. Themanager of a region determines the relations between peersbased on players’ overlapping AOIs. Regular peers residingin a region inform their manager about the events they create(e.g., position changes). The manager in return informs reg-ular peers about state changes due to events occurring in itsregion. Regular peers directly exchange messages to reducethe load on the manager [31], while neighboring managersexchange their regions information in order to provide acontinuous view to their regular peers [31, 32].

3.1 Different shaped partitioning

The virtual world can be partitioned into different shapedand sized regions. Rectangular [24, 33–36], tree-based [32,36–38], hexagonal [31, 39–42], and polygonal (e.g.,Voronoi) [30, 35, 43–45] regions are the most popularpartitioning forms.

MOPAR [31] partitions the virtual world into hexagonalcells (Fig. 1). Each hexagonal cell is assigned to a homenode through a DHT. A home node is used for assigningthe cell to a master node which is responsible for the cellmanagement (i.e., a DHT is used as a rendezvous pointfor master node registration). The first node entering intoa cell becomes the master node of the cell, otherwise itbecomes a slave node. The master of a cell keeps infor-mation about slave nodes residing inside the cell. Slavenodes inform their master about their position changes inthe VE, and are notified of new neighbors by their master.Slaves directly inform each other about their movements todecrease the load on their master node. Masters exchangetheir slave lists to be informed about the slaves resid-ing in the neighboring cells and thus to completely coverthe AOI of their slaves in order to provide them with a

Fig. 1 Hexagonal partitioning - Mopar [31]

continuous view of the virtual world. However, althoughslave list exchange is performed only between masters, itsfrequency is still a concern. When a master leaves its cell,one of the slaves residing in the cell becomes the new mas-ter, to which the cell data is transferred. The continuous datatransfer to newly assigned cell masters can create a signifi-cant overhead in the system. Besides, the hexagonal zoningdoes not allow the repartitioning of a cell into smallerregions when the cell master becomes overloaded due toslave crowd in the cell. Moreover, in case of crowding,excessive bandwidth requirements beyond slaves’ capabili-ties can occur due to direct connectivity between them.

3.2 Static and dynamic partitioning

Besides different shaped zoning, NVEs based on world par-titioning can follow two types of world division strategieswith different partitioning dynamics: static and dynamic. Instatic partitioning, the virtual world is partitioned into fixed-size regions (i.e., prepartitioning of the virtual world intoregions); thus, the regions do not change with the evolutionof the VE [24, 31, 41, 46, 47]. In dynamic partitioning, onthe other hand, regions can evolve over time due to appli-cation needs. A region can be repartitioned into smallerregions each of which is assigned to a new manager [34,36–38, 45, 48–50]. In the following, COVER [37] and Bad-umna [35] are presented as P2P NVE system examplesfollowing different types of world partitioning strategies.

COVER [37] proposes a dynamic world decomposi-tion through a global quadtree to partition the world byrecursively subdividing it into four quadrants (i.e., regions)depending on the number and position of players (Fig. 2).Each region is assigned to a super-node, the closest playerto the geometric center of the region. Each player is directlyconnected to all other players in its AOI. A player is calledas a covered player if its AOI is completely covered by itsneighbors’ AOI, and as an uncovered player if its AOI isnot completely covered. To provide awareness, each playersends its movement update messages to all its AOI neigh-bors, which in turn propagate the messages to their ownAOI neighbors. Players receiving a message can thus deter-mine their AOI neighborhood relationship with the playersending the message. A super-node is responsible for themanagement of the uncovered players in its region. Uncov-ered players inform not only their AOI neighbors but alsotheir super-node about their position changes in the VE. Asuper-node can thus provide awareness to uncovered play-ers located within its region. However, players’ movementcan produce frequent changes in the underlying quadtreestructure, making the management of the quadtree structuredifficult. When crowding occurs in a small area inside aregion, successive divisions are needed to be able to parti-tion the crowded area, which results in several nearly empty

Page 5: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Fig. 2 COVER [37]

regions. In addition, in case of crowding, direct connectiv-ity to all AOI neighbors can result in excessive bandwidthrequirements beyond players’ capabilities.

Badumna [35, 51] provides both static partitioningthrough fixed-size cells and dynamic partitioning throughdynamic bounded regions, which can change size and posi-tion as required for the management of a dense region(Fig. 3). Badumna is based on both world partitioningand DHT approaches. Both cells and bounded regions areassigned to peers through the underlying DHT. Entitiesquery the DHT to find out about other entities in their AOI.A bounded region containing several entities queries theDHT as a single entity (i.e., one query serving for all entitiesin the region), thus reducing the traffic on the DHT. In addi-tion, a gossip protocol is proposed for the management of ahighly dense area (i.e., a large number of players clusteredin a small area) in order not to overload the peer respon-sible of this area. In a highly dense area, players reducethe frequency of their interaction with their manager peer;instead, players rely on their neighbors to inform each otherabout other players in the region based on the gossip proto-col. However, players’ frequent position changes in the VEmake the management of bounded regions difficult. In addi-tion, the search for interest purposes through queries on theDHT introduces unacceptable latency due to the number ofhops required by the underlying DHT structure.

3.3 Discussion

Table 1 presents a list of the different NVE systems pre-sented in this section. The world partitioning approach iswidely used to construct P2P NVE systems in a simple man-ner, where the global world management is divided into themanagement of small regions containing entities possibly inrelation based on interest. However, the partitioning of thevirtual world, which provides a fair distribution of the VEload among managers, remains a challenging issue. Notethat a large number of players can congregate in a regionfor interest purposes and crowding can thus dynamicallyoccur in different regions due to players’ frequent positionchanges in the VE.

To address this issue, in static world partitioning-basedNVEs, communication is generally constructed based on asecondary mechanism in order to reduce the load on man-agers, such as message dissemination in a region throughload balancing trees in case of crowding (e.g., Distributedevent delivery [33]), application layer multicast based onthe underlying DHT structure (SimMud [24]), direct com-munication between regular peers (e.g., MOPAR [31]), etc.Dynamic partitioning is proposed as an alternative solu-tion, in which overloaded regions due to high density arerepartitioned into smaller regions managed by new man-ager peers. However, region reassignments and management

Fig. 3 Badumna [35]

Page 6: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Table 1 The presented world partitioning based NVE systems

NVE systems Features

MOPAR [31] Static polygonal

Player list exchange between region managers

Direct message exchange between regular

peers in a region

N-Trees [32] Dynamic tree-based

Interregional communication supported by

managers

Direct message exchange between regular

peers in a region

COVER [37] Dynamic tree-based

Management of the uncovered regular peers

by managers

Direct message exchange between regular peers

based on AOI

Badumna [35] Static rectangular and dynamic polygonal

DHT support

Latency due to querying the DHT

of dynamic entities can create a significant overhead onthe system. A dynamic partitioning approach capable ofproviding static assignment of entities to managers is thusthe most suitable world partitioning approach to the NVErequirements, which can also be supported by a secondarycommunication mechanism as in static world partitioning.

4 DHT approach

The use of distributed hash tables (DHTs) [52–54] is anothermethod commonly used to support P2P NVEs. DHTs pro-vide an efficient way of indexing data items by fairly bal-ancing the system load among all participating peers. DHTsystems implement a hash function that maps any given itemto a specific peer in the system. Items and peers are mappedto a key space, which is partitioned among all peers. Eachpeer is responsible of a specific key space partition. DHTsprovide dynamic maintenance of peers, efficient lookup andload balancing of both resources and query messages.

P2P NVE systems partitioning the virtual world intoregions can use a DHT to assign each region to a managerpeer [24, 31, 35, 36, 38, 44, 46, 49, 55]. On the other hand,Colyseus [56] and Loader [57] use a DHT for distributingentities, instead of regions, to peers.

To ensure a fair distribution for load balancing purposes,DHT systems use a uniform hash function that randomlymaps items and peers to a common key space [53, 54]. How-ever, hashing destroys both the physical and logical localityrelation among peers. Pastry [52] takes into account physi-cal network locality, which minimizes the distance that the

messages have to travel. However, peers residing inside thesame region in the virtual world are still placed far-apart inthe DHT key space. Thus, peers probably interacting witheach other are connected via multiple hops over the DHT,which may result in high routing latency. To address thisissue, NL-DHT [58], Loader [57], Walkad [38] take intoconsideration the logical locality (i.e., distance in the vir-tual world) by mapping nearby peers in the virtual world tonearby keys on the DHT space. In addition, Colyseus [56],OPeN [55], Walkad [38] propose range queries on the DHTto find the set of entities located in peers’ area of interest.

To support communication between peers inside aregion, SimMud [24], PastryMMOG [40], Loader [57] andPBT [36] use Scribe [59], the application layer multicastbuilt on the underlying DHT overlay, Pastry [52]. However,to avoid unnecessary network delay due to possible numberof hops on the DHT, [31, 32, 34–36, 46, 49, 56] use DHTsonly as a rendezvous point for peers to discover relevantentities, and propose a communication model independentfrom the DHT routing infrastructure (e.g., direct, multicast,etc.) for update message dissemination.

4.1 Representative systems

This section examines several P2P NVE systems based ona DHT, namely Colyseus [56], SimMud [24] and PBT [36].Colyseus assigns entities to peers, while SimMud and PBTassign regions. Update messages are disseminated to rele-vant peers through a publish-subscribe fashion in Colyseus,and through Scribe [59] in SimMud and PBT.

4.1.1 Colyseus

Colyseus [56] proposes a NVE system based on Mer-cury [60], which is used as both a traditional randomizedDHT and a range-queriable DHT to distribute objects topeers and to discover objects within the AOI of a peer. Eachobject has one primary copy assigned to a peer through theDHT, and may have several replicas on different peers in thesystem to enable quick local access to the object. The pri-mary owner of an object is responsible for the serializationand synchronization of the updates to the object performedon any peer in the system. Peers interested in an object sub-scribe to the object owner through the DHT. The ownerof an object periodically publishes the object data to thesubscribers.

The components of Colyseus nodes are presented inFig. 4a, while an example of publication and subscrip-tion in Mercury is given in Fig. 4b. Hx and Hy are therange-queriable DHT hubs corresponding to the X and Ycoordinates, respectively, of a two dimensional world of320 × 320 units. The publication is sent to both Hx and Hy ,where it is stored at peers b and e respectively. However, a

Page 7: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Fig. 4 Colyseus’ system design

(a) Colyseus nodes [56]

x <= 150

int

int

int

int x >= 50

y <= 250

y >= 150

[0,80)

[240,320)

a

c

d

b[80,160)

[160,240)

int x 100int y 200

(query)

(data−item)

[200, 320)

[100,200)

g

e

f

H

H

x

y[0,100)

(b) Publication and subscription in Mer-cury [60]

subscription is passed to exactly one of the hubs correspond-ing to the required attributes. The subscription thus entersHx at peer d and is processed at peers b and c.

Through interest prediction, a peer prefetches the replicaof the objects likely to be needed in the near future (e.g.,objects entering in the player AOI due to the player move-ment in the world), and subscribes to the owner of theseobjects through the DHT. However, a short-lived object(e.g., missile) might not be consistently prefetched. Toaddress this issue, Colyseus proposes a proactive replica-tion algorithm which attaches a short-lived object to a peerbefore the object is requested by the peer based on the factthat most short-lived objects originate at locations very closeto their creator; thus, peers interested in the creator willprobably be interested in the newly created objects.

Colyseus performs better scalability and load balancingwith a range-queriable DHT compared to a randomizedDHT. However, in a dynamic world based on a range-queriable DHT, the continuous movement of objects intro-duces a high message overhead due to the frequent DHTupdate requirements. In addition, since the neighborhoodrelations between peers are built independently from thelogical locality relations among them in the VE, peersresiding inside the same area in the VE are connected viamultiple hops over the DHT, which may result in highlatency. Finally, in a world with a large number of peers,looking up objects with O(log N) search hops may createunacceptable object discovery latency.

4.1.2 SimMud

SimMud [24] is a distributed NVE system following thestatic world partitioning, DHT and multicast approaches.SimMud’s architecture is based on Pastry [52] and its cor-responding application layer multicast, Scribe [59]. Thevirtual world is partitioned into fixed-size regions contain-ing players and objects (Fig. 5). Using Pastry, each regionis randomly mapped to a coordinator peer that manages allobjects in the region and serves as the root of the multicasttree to which all players residing in the region subscribe.

State changes occurring in a region are periodically multi-cast by the coordinator to all region members. Coordinatorsconnect to each other through Pastry to enable player migra-tion between regions. Due to disjoint region partitioning,SimMud does not provide players a continuous view ofthe virtual world. Since a region’s area may exceed play-ers’ AOI, players receive irrelevant messages beyond theirAOI. If many players crowd in a region, the coordinator canquickly become a bottleneck and suffers from performanceand robustness issues. In most cases, messages are deliveredin less than 6 hops on Pastry’s address space. However, dueto Scribe multicast routing, the number of hops might gobeyond 50 hops, resulting in a delay up to several seconds,since a multicast tree is not constructed based on the AOIconcept, and so may include peers not residing inside theregion.

4.1.3 PBT

PBT [36] is a P2P NVE system following both the static anddynamic world partitioning, DHT and multicast approaches.The whole world is first partitioned into static regions, eachof which is assigned to a region master through the under-lying DHT (Fig. 6). Players interested in a region subscribeto the region through its region master, to be informedabout state changes occurring inside the region. Update

Food

Player

Multicast

Multicast

Direct connection

Region 1

Region 2

Region 3

Fig. 5 SimMud [24]

Page 8: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Fig. 6 PBT’s systemarchitecture [36]

dissemination inside a region is maintained based onScribe [59]. In addition, in order to manage a static regionwith a high number of entities or subscribers, a Predic-tion Binary Tree (PBT) is built on top of the region, whichbecomes the root of the PBT. When necessary, a region isdynamically partitioned into subregions, which correspondto the leaves of the PBT managed by the region master.Each subregion is assigned to a subregion worker throughthe DHT. A new multicast group is constructed for eachsubregion. A subregion worker manages the entities insidethe subregion and multicasts update messages to subscriberpeers. Compared to SimMud [24], PBT additionally pro-vides continuous view of the virtual world and dynamicpartitioning of crowded regions. However, players’ frequentposition change makes it extremely difficult to efficientlymanage a PBT in order to provide an adequate managementof world regions. In addition, PBT is subject to messagetransmission latency, as SimMud, due to the underlyingDHT structure and application layer multicast.

4.2 Discussion

The DHT-based NVE systems presented in this section arelisted in Table 2. A DHT assigns items to peers in a simpleway; thus, VE items can efficiently be distributed amongpeers via a DHT. However, the relation between items andpeers can be destroyed by the hash function due to randommapping of items and peers over the DHT key space. On theother hand, DHT-based NVE systems addressing this issueby mapping items on the DHT key space based on the logi-cal locality can suffer from the high dynamism of the VE interms of position changes in the virtual world. In addition,frequent entity discovery and message dissemination based

on a DHT can cause unacceptable delay for NVE applica-tions. Therefore, a DHT can be used for static assignment ofentities to peers while taking into account the logical rela-tion in the virtual world; however message disseminationbased on an approach other than a DHT might produce moreefficient results in the context of NVEs.

5 Multicast approach

In a VE, when the state of an entity is modified due to anevent, related update messages are disseminated to play-ers interested in the entity [61]. Multicast is a method usedto send an update message to relevant players, namely, amulticast group. A world region or an entity is assigneda multicast group to inform group members about corre-sponding state changes. Players subscribe to the multicast

Table 2 A list of the representative DHT based NVE systems

NVE systems Features

Colyseus [56] Entities assigned to peers

No logical locality relations between peers

on the DHT

Both randomized and range-queriable DHT support

SimMud [24] Regions assigned to peers

Static world partitioning support

Multicast support

PBT [36] Regions assigned to peers

Both static and dynamic world partitioning support

Multicast support

Page 9: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

group of regions or entities intersecting with their AOI toreceive relevant update messages. P2P-based NVE systemsexploit both IP multicast and application layer multicast(ALM) for update message dissemination. In the following,we examine different types of multicast schemes proposedby NVE systems.

5.1 IP multicast

Both region and entity-based IP multicasts are proposed byNVE systems to support message dissemination in VEs.In region-based IP multicast, the virtual world is parti-tioned into regions, each of which is assigned a multicastgroup through a multicast address (e.g., NPSNET [26],SPLINE [28], SCORE [62], DIVE [63]). When an eventtakes place inside a region, the related message is multi-cast to all players residing in the region. On the other hand,entity-based IP multicast focuses on the entity, instead of theregion, and assigns each entity (e.g., avatar) a group to mul-ticast its state information to other interested players (e.g.,VELVET [64], Massively Multiplayer Audio Communica-tion [65]).

5.2 Application layer multicast

Application layer multicast (ALM) has been proposed toprovide multicasting functionality as an application serviceinstead of a network service, based on overlay networks.In ALM, a multicast tree is constructed through unicastlinks between overlay network peers forming the multi-cast group [66]. NVE systems propose both physical andlogical locality aware ALM algorithms to support messagedissemination.

5.2.1 Physical locality aware ALM

ALM algorithms typically construct their multicast treebased on the characteristics of unicast links between peers(e.g., end-to-end latency) in order to minimize the additionalrouting overhead compared to IP multicast [67]. Physicallocality aware ALM thus realizes update message dissem-ination based on nearby peers in the physical network interms of latency. NVE systems based on physical localityaware ALM are detailed in the following.

Region-based physical locality aware ALM

This approach has been used in [24, 32, 33, 36, 39, 40,42, 68–70]. These systems follow the region-based multi-cast strategy and propose an ALM scheme to disseminateupdate messages to players inside a region in the virtualworld. SimMud [24] and PastryMMOG [40] are based onPastry [52] and use its corresponding ALM, Scribe [59].

The virtual world is partitioned into regions, each assigneda multicast group with a unique ID. Regions are distributedto peers through Pastry. The peer with an ID closest to aregion ID serves as the manager of the region and the rootof its multicast tree. The manager of a region manages theresiding entities and multicasts messages related to eventsoccurring in the region, to the players residing inside theregion.

Various systems [33, 39, 42, 68–70] have proposed adifferent region-based ALM algorithm, which aims to pro-vide an efficient group management, update transmissiondelay and load balancing. MM-VISA [42, 71, 72] orga-nizes and manages entities by partitioning the virtual worldinto hexagonal regions, each of which is assigned to acoordinator, responsible for multicast-based communica-tion between players inside a region. Coordinators are con-nected to each other to support interregional communicationdue to AOI overlaps of players with different neighborregions. To provide a better multicast group management,entities are clustered into classes according to their move-ment behavior in the virtual world (e.g., stationary, slow,normal and fast moving objects). Multicast trees in a regionare constructed based on one cluster; in other words, themembers of a multicast group belong to the same cluster ofthe region. While constructing a multicast tree, the cumu-lative end-to-end delay between a coordinator and peers istaken into consideration, which results in multicast treeswith limited height. When all tree paths in a region reach themaximum height limit, a new multicast channel is openedfor newly joined/arrived players to the region. Figure 7represents the ALM scheme and message propagation inMM-VISA architecture, where C stands for coordinator,and O-1, O-2, O-3, and O-4, for classes having differentbehaviors.

Entity-based physical locality aware ALM

In entity-based ALM, each entity’s update messages are dis-seminated based on ALM multicast to players interested inthe entity. P2P propagation [73] is a client/server systemusing P2P multicast for message dissemination to reducethe server load. P2P propagation follows the entity-basedmulticast strategy, while proposing its own ALM algorithm.Each entity is assigned a multicast group. The set of peersinterested in an entity is determined by the server. For eachmulticast group, the peer having the minimum delay to allother peers inside the group is selected as relay to forward amessage destined to the group, from the server to the groupmembers. The update message of an entity thus goes fromthe server through the relay of the multicast group to reachthe members (Fig. 8). However, when crowding occurs, arelay can be overloaded due to message delivery to all groupmembers.

Page 10: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Fig. 7 ALM scheme andmessage propagation inMM-VISA [42]

C

O-4

O-2O-1

O-3

C

O-4

O-2O-1

O-3

Local multicast messageInterested message

5.2.2 Logical locality aware ALM

Players’ interest in the virtual world can follow a differentorder than the peers’ layout on the latency-based multi-cast tree. The message transmission delay between nearbyplayers in the virtual world can be more significant thanthe delay between distant players, since distant playershave more tolerance for weak synchronization than nearbyplayers. Logical locality aware ALM is thus proposed toprioritize update message dissemination to players that arenear the event. NVE systems based on logical locality awareALM are detailed in the following.

Region-based logical locality aware ALM

To disseminate update messages first to the most interestedplayers (i.e., nearby players to the event), N-Trees [32] andPIMTRE [74] construct a multicast tree based on the inter-est relation between peers in the multicast group. A peerresiding in a region transmits update messages due to anevent, first to nearby peers residing in the region, then to dis-tant peers residing in the neighbor regions within the event’sscope. Therefore, nearby peers in the world can receivemessages faster.

N-Trees [32] divide the virtual world into nested rectan-gular regions based on a quadtree (i.e., 2-Tree) in order to

Server

Relay Node

Child Node

Fig. 8 P2P propagation [73]

organize players by their interest (Fig. 9). A node in an N-Tree is either a leaf node (i.e., a node with no children) or anode with 2N children. Each node in the N-tree is managedby a selected peer, called leader. Each leaf node in the N-Tree corresponds to a world region. Each successive levelbelow the root node in the N-tree represents a further sub-division of the virtual world by 2N . Players residing insidea region form a group and join the corresponding leaf node.When a player generates an event in a region, the player dis-seminates the event message to other players, including theleader, residing in the same leaf node corresponding to theregion. If the event scope exceeds the region boundaries, themessage is then propagated to the regions (i.e., nodes) inter-secting with the event scope through their leaders. Whenthe number of players in a region, namely the population ofa leaf, exceeds a given threshold, the leaf in the N-Tree issubdivided into N children and the players are placed intothe appropriate leaves. The subdivision of a crowded regioninto subregions decreases the bandwidth requirements ofa player since the number of players to which a playershould send its update messages is reduced. However, lead-ers managing the corresponding leaves and nodes may beoverloaded due to interregional message transmission. Inaddition, players’ movement can produce frequent changesin the underlying N-tree structure, making the managementof the N-tree structure difficult.

Fig. 9 N-Trees [32]

Page 11: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Entity-based logical locality aware ALM

Apolo [75], AOICast [76] and pSense [77] follow a similarapproach to N-Trees and PIMTRE, while using the entity-based multicast strategy. Players multicast their update mes-sages to other players inside their AOI based on the localityrelation between players in the virtual world.

Apolo [75] is the first P2P NVE system achieving mes-sage through AOI-based multicast trees. Apolo connectseach peer to its closest peer in each of the four quadrantsin the virtual world (Fig. 10). A peer p informs other peersinside its AOI about its state change through a multicast treebuilt on top of the overlay (Fig. 10b). The source peer gener-ating the event, namely p, becomes the root of the multicasttree. For a peer q in p’s AOI, the closest peer to p amongq’s closest quadrant neighbors becomes q’s tree parent. Theevent update message is first received by peers closest tothe source peer p, and then relayed to other farther peers.However, the connectivity of each peer to exactly four peerscan lead to multicast trees with long paths, thus increasingthe number of hops (i.e., latency) needed for sending updatemessages to AOI neighbors.

5.3 Discussion

A list of the multicast-based NVE systems given in this sec-tion are listed in Table 3. Multicast provides an efficientway of communication between peers by distributing mes-sage dissemination load among all relevant peers. However,entity-based multicast might create different trees with thesame multicast group members (e.g., multicast tree struc-tures for entities that are nearby in the virtual world), thusoverloading peers participating to multicast trees. On theother hand, in region-based multicast, the region size hasan important role in the efficiency of message delivery. Incase of large regions, since the region may exceed play-ers’ AOI, players receive irrelevant messages beyond theirAOI. With small regions, since players’ AOI may cover sev-eral regions, the same players become participants of severalmulticast trees, thus increasing computation overhead on

peers managing the multicast trees (i.e., tree roots), andplayers participating in the multicast.

IP multicast offers efficient message delivery inside agroup, however it is not widely deployed due to issuesrelated to group management, multicast address allocationand network management [78, 79]. On the other hand,in ALM, peers’ upload bandwidth is a critical resourcethat should be optimized. The depth of the ALM tree isincreased as group size increases, leading to long latencywhich may not be appropriate for NVE requirements [80].In addition, message dissemination may be delayed in ALMalgorithms that do not construct multicast trees based onthe AOI concept; for instance, Scribe [59] can create treeswith long paths which include peers transmitting event mes-sages while not residing in event scopes. As such, theproperties of both the underlying physical network as wellas the logical network built on top of the physical net-work should be considered to provide an efficient multicastalgorithm.

6 Fully-connected approach

In the fully-connected approach, all players in the virtualworld are directly connected to each other. Since everyoneis connected to every-one else, message filtering techniquesare used to limit message transmissions between certainpeers to make the system feasible. Players frequently sendmessages to only a small group of players, while infre-quently to all other players in the world. P2P NVE systemsbased on this approach focuses mainly on the managementof avatars where avatars’ data are stored on the correspond-ing user machines. A world object, if any, can be treated asan avatar and its data are thus managed on the machine ofthe user creating the object.

6.1 Representative systems

Two fully-connected P2P NVE systems, Frontier set [81]and Donnybrook [82] are examined in the following.

Fig. 10 Entity-based logicallocality aware ALM - Apolo [75]

(a) Topology (b) Multicast tree

Page 12: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Table 3 A list of the multicast-based NVE systems

Multicast NVE systems

IP multicast Region based NPSNET [26], SPLINE [28],

SCORE [62], DIVE [63]

Entity based VELVET [64], Massively multiplayer

audio communication [65]

ALM Physical locality aware Region based SimMud [24], Event delivery [33], AVDNet [39],

PastryMMOG [40], MM-VISA [42], Federated P2P [68],

Update dissemination [69]

Entity based P2P propagation [73]

Logical locality aware Region based N-Trees [32], PIMTRE [74]

Entity based Loader [57], Apolo [75], AOICast [76], pSense [77]

Frontier set

Frontier set [81, 83] is inspired from update-free region(UFR) in the geometric message filtering protocols [84, 85]and potentially visible set (PVS) [86]. The UFR concept isused to determine the message exchange protocol betweenplayers, while the PVS concept is used to group playersinterested in each other and thus frequently exchangingmessages.

In the geometric message filtering protocols [84, 85],each player of any pair of players defines its respectiveupdate-free region (UFR) inside which the player does notinform the other about its position change. When a playerleaves its UFR, the pair of players become relevant to eachother. From that time on, update messages are thus mutuallyand regularly transmitted between the two players. Whenthey become irrelevant again, new UFRs are computed. Theplayers do not send any update message until they becomerelevant again.

Potentially visible set (PVS) [86] is based on the fact thatfrom most viewpoints in the world, only a small portion isvisible due to occlusion by opaque surfaces (e.g., walls).The world is partitioned into rectangular cells, the bound-aries of which coincide with opaque surfaces (Fig. 11a).Non-opaque portals are identified on cell boundaries, and

used to form an adjacency graph connecting the world cells.Each cell can thus determine other visible cells from thiscell, based on existing unblocked sightlines between cells.In Fig. 11b, cells B, C, and E are visible to cell A since thereexists a sightline from A to each of them.

Frontier set [81, 83] defines frontier to determine the vis-ibility between cell pairs based on the PVS concept andthe message exchange frequency between players resid-ing in different cells based on UFR concept. Given twocells A and B, a frontier consists of two cell sets, FAB

and FBA, such that no cell in FAB is visible to a cell inFBA and vice-versa. There exists no frontier between twocells visible to each other. The players residing inside themutually visible cells exchange messages. No message isexchanged between players residing inside the cells invis-ible to each other. Therefore, players residing inside cellswithout any frontiers are visible to each other and thus fre-quently exchange messages until they become invisible dueto their movement. Players invisible to each other do notexchange messages until they cross any frontier and thusbecome visible again. Fig. 12 shows the frontier of thecell pairs corresponding to players’ positions in the world.No message is exchanged between players in Figs. 12a, b,and d. However, players exchange messages while residinginside visible cells (Fig. 12c).

Fig. 11 Potential visibility set

(a) Portals and adjacency graph [86]

A

G H I

(b) Visible cells from cell A [83]

D E F

B C

Page 13: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

(a) (b) (c) (d)

Fig. 12 Frontier sets [83]. Points represent players. a FAI ={A,B,C, F }, FIA = {G,H, I}. b FEH = {A,B,C,E,F }, FHE ={H, I}. c No frontier set since cell D can see cell H. d FDI =

{A,B,C,D,E,F }, FID = {I}. No message exchange betweenplayers in a, b, and d. Mutual message exchange in c

Although only players visible to each other exchangemessages, computing requirements can exceed peers’ capa-bilities in a VE with a high number of players frequentlychanging position, since each player should check the fron-tier set with each other player in the VE, and calculate newfrontier sets if necessary.

Donnybrook

Donnybrook [82] is a fully-connected NVE system basedon the fact that a person can only focus on a constant num-ber of entities at any moment [87, 88]. Donnybrook thuslimits the number of players in which a player is most inter-ested. Players frequently send their state update messagesto the players which are interested in them. Besides, playerssend infrequent state information to other players (i.e., play-ers with less interest in them). A player p is thus frequentlyinformed about the state of players in which p is inter-ested, and infrequently informed about the state of playersin which p is less/not interested. To extrapolate the behaviorof a remote player q, from which p receives infrequent infor-mation, an intelligent bot runs on p’s local machine. Thisbot approximates q’s behavior through q’s infrequently sentinformation. In case of a difference between q’s approxi-mated and actual state, the bot approaches smoothly to theactual state. Since p is not interested in q, p is unlikely tonotice the difference.

6.2 Discussion

The presented NVE systems based on the fully-connectedapproach are given in Table 4. In the fully-connectedapproach, all players in the virtual world are directly con-nected to each other. The fully-connected approach thusdoes not scale well with the number of players in the VEdue to bandwidth and computing overhead on each peer.In addition, if a large number of peers are interested in(and thus want to directly interact with) each other, thenecessary message exchanges can quickly exceed peers’capabilities.

7 Neighbor-list exchange approach

In P2P systems, using neighbors of neighbors knowledgeis a well-known strategy to acquire required informationand to make a better routing decision [89]. In neighbor-listexchange-based P2P NVEs, peers exchange their knowl-edge of the virtual world in order to be better informed ofthe changes occurring in their surroundings. P2P NVE sys-tems following this approach only consider avatars, the dataof which are managed on corresponding user machines.

7.1 Representative systems

This section examines two P2P NVE systems based on theneighbor-list exchange approach, P2P message exchangescheme [90] and Message interchange protocol [91].

7.1.1 P2P message exchange scheme

P2P message exchange[90] is a fully-distributed system inwhich the VE is supported through regular neighbor-listexchange between peers. Each peer maintains a fixed num-ber of direct connections to the closest neighbors in thevirtual world. A peer p’s direct neighbors are called activeentities (AE). p is regularly informed by its AE about notonly themselves but also their own neighbors. The neigh-bors of p’s AE are called latent entities (LE). p regularlyupdates its AE list based on the information received fromits AE. Figure 13 shows the neighborhood relationship and

Table 4 A list of the representative fully-connected NVE systems

NVE systems Features

Frontier sets [83] Based on update-free region [84] and

potentially visible set [86]

Bottleneck on highly visible players

Donnybrook [82] Based on the fact of the limited number

of focus entities [87, 88]

Bottleneck on popular players

Page 14: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Fig. 13 P2P message exchange scheme [90]

message flow for a peer O, in which O’s active entities areA1, A2, A3, A4, and latent entities are B1, B2, . . . ,B8.

In a world with uncrowded regions, connections to onlya fixed number of nearest peers can divide the networkinto isolated subgroups. [92] is an extension of [90] thatproposes a probabilistic solution to the network partition-ing problem. To ensure global network connectivity throughinformation exchange between isolated subgroups, peersconnect to randomly selected peers at arbitrary time inter-vals through a bootstrap server and exchange informationwith those peers.

To provide an efficient solution to the network parti-tioning problem even in the presence of a high numberof entities, [93] replaces the random message exchangescheme proposed in [92] with a DHT-based scheme toreorganize isolated subgroups through message exchangebetween them. The virtual world is partitioned into regions,each of which is assigned to a manager peer through a DHTbuilt separately from the P2P message exchange protocol.The manager peer of a region keeps information about peersinside its region. Peers thus inform the manager of regionsinside which they reside about their position changes. Whena peer p belonging to an isolated subgroup inquires the man-ager of a region, the manager provides p with the list ofpeers residing inside the region. Then, p asks all peers in thelist their current location in the world and updates its AE listbased on distance. The network is thus reorganized and sub-groups are kept connected. However, this solution requiresthe management of a DHT structure based on peers’ posi-tions in the virtual world in addition to the P2P messageexchange protocol, and thus increases the load on peers.Furthermore, as in exchanged neighbor-list information, thepeer list information provided by the manager of a region toan inquiring peer can contain redundant information, espe-cially in case of player crowding in the region, which cancreate excessive message exchanges.

7.1.2 Message interchange protocol

Message interchange protocol [91] connects each peer to atleast four peers, the closest in each of the four quadrants inthe virtual world (Fig. 14) in a similar way to Apolo [75](cf. Section 5.2.2). However, in Apolo, each peer connectsto exactly four peers (i.e., the closest peers in each of thefour quadrants), which can introduce additional hops to senda message to an AOI neighbor, thus increasing latency.

In Message interchange protocol, a peer p keeps infor-mation about other peers in three tables: Observable Table(OT), Global Table (GT), and Group Leader Table (GLT).The OT contains information about peers interested in p andto which p regularly sends its update messages. The GTcontains information about other peers residing in p’s sur-roundings, however not directly interested in p; this tableis used for accurate AOI neighbor discovery. Peers inter-change their GT within a time period depending on thedistance between them in the virtual world. Nearby peersinterchange their GT more frequently than faraway peers.Besides, the GLT keeps track of the closest peers in each ofthe four quadrants to ensure global connectivity. Peers peri-odically updates their GLT and peers having empty OT andGT interchange their GLT to keep the network connected.Message interchange protocol is, therefore, basically basedon the exchange of peers’ OT, GT and GLT with their neigh-bors, while the OT, GT and GLT are used for state updatedissemination, AOI neighbor discovery and global networkconnectivity, respectively.

Compared to P2P message exchange scheme, Messageinterchange protocol provides better AOI and global net-work connectivity managements. However, similarly to P2Pmessage exchange, exchanged lists may contain redundantinformation. Player crowding causes a high number of peers(with long lists) exchanging their list information with eachother, the bandwidth requirements of which may exceedpeers’ capabilities.

Fig. 14 Message interchange protocol [91]. e connects to at least theclosest peers in each of the four quadrants

Page 15: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

7.2 Discussion

Table 5 shows the neighbor-list exchange based NVE sys-tems presented in this section. In the neighbor-list exchangeapproach, each peer has limited number of connectionsindependently from the total number of users in the VE.The direct connectivity between peers in relation minimizesthe delay of message propagation in the VE. However, aspeer connectivity supported by only neighbor-list exchangeis not sufficient to ensure global network connectivity,an auxiliary mechanism is thus needed to avoid networkpartitioning. In addition, constant neighbor-list exchanges,which may contain redundant information for destinationpeers, introduce a high message overhead.

8 Mutual notification approach

In P2P NVE systems based on the mutual notificationapproach, peers maintain direct connections to nearby peerssurrounding them in the VE, usually based on well-knowngeometrical structures such as Voronoi in [94], Yao graphin [77] or quadtree in [95]. In the event of peers’ positionchange in the VE, global network connectivity and neigh-bor discovery are achieved through mutual cooperation andnotification among direct neighbors.

8.1 Representative systems

This section examines several P2P NVE systems basedon the mutual notification approach, namely, pSense [77],Solipsis‘03 [96], VON [94], Relaxed Triangulation [97] andRed-Black Delaunay [98].

8.1.1 pSense

pSense [77] provides an AOI-based peer connectivity.Each peer keeps information about other peers in its AOI(Fig. 15). To ensure global network connectivity and neigh-bor discovery, each peer also connects to sensor peers, the

Table 5 The presented neighbor-list exchange based NVE systems

NVE systems Features

P2P message exchange [90] Global network connectivity

through DHT

Irrelevant and redundant data

transmission

Message interchange Global network connectivity through

protocol [91] closest quadrant peers

Irrelevant and redundant data

transmission

closest peers outside its AOI in each of the six sectors ofequal angle, similarly to Yao graphs [99]. When a peer pmoves in the world, p informs its AOI and sensor neighborsabout its position change. If a sensor peer detects a peer qentering in p’s AOI, it informs p about q which becomes p’snew AOI neighbor. If the total number of neighbors exceedsp’s upload bandwidth capacity, p sends its update messageto only a randomly selected subset of its neighbors. Theneighbors receiving p’s update message then forward themessage to their own neighbors residing in p’s AOI. How-ever, in case of player crowding, many duplicates can beproduced during message forwarding. Also, peers receivinga message may not have enough bandwidth to forward themessage to all their neighbors, leading to a second roundof randomly selected (instead of all) neighbors, and thusincreasing the delay with which some peers will receivemessages.

8.1.2 Solipsis ’03

Solipsis‘03 [96, 101] is a P2P NVE system that proposesa new connectivity structure based on the angular rela-tion between peers’ positions in the virtual world. Thevirtual world is designed as a two dimensional torus toprovide an unbounded surface. Each peer maintains directconnections with the peers residing in its AOI. Neighbordiscovery is achieved through the notification of neighbors.The neighbors of a peer should always form a convex hull(Fig. 16), which guarantees global network connectivity.If a peer detects a convex hull violation through recursivecollaboration with its direct neighbors, the peer establishesnew connections until the convex hull condition is satisfied(Fig. 16). In the event of position change, new neighborsare discovered through mutual cooperation and notifica-tion among direct neighbors. When a peer wants to moveto a specific position in the virtual world upon login orby teleportation, the peer’s request is transmitted from therequesting peer to the peer closest to the specified position(Fig. 16).

However, Solipsis‘03 might lead to inconsistencieswhere some neighboring peers go undetected [100].Figure 17 presents such a situation where e and ef are notconnected although ef is residing inside e’s AOI.

8.1.3 VON

VON [94, 102] proposes a distributed NVE system similarto Solipsis‘03 [96]. VON mainly replaces the geometricalstructure providing peer connectivity in Solipsis‘03 with aVoronoi diagram [103] to resolve the undetected neighborproblem in Solipsis. The virtual world is partitioned usingVoronoi based on peers’ positions in the virtual world. Eachpeer maintains a Voronoi diagram of its AOI neighbors,

Page 16: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Fig. 15 pSense [77]

N

N

N

N

N

N

N

N

S

S

S

S

S

S

N

S

Local Node

Near Node

Sensor Node

Unknown Node

S

S

Vision Range

(a) A peer’s view

S

S

S

S

S

S

S

Uneligible Node

S Sensor Node

Local Node

SVision Range

(b) Sensor node selection

and keeps a direct connection to all of them. Figure 18presents the Voronoi-based partitioning of the virtual worldaccording to the participants’ position and the neighborhoodrelations between them. Each peer keeps connections withits enclosing neighbors whose Voronoi regions have a com-mon edge with its own Voronoi region (even if they resideoutside its AOI). The outermost peers inside the AOI of apeer are called boundary neighbors, and are responsible forneighbor discovery. VON thus optimizes the mutual notifi-cation between peers compared to Solipsis‘03 by reducingthe number of peers participating in the neighbor discoveryfrom all AOI neighbors to only boundary neighbors. VONprovides limited number of connections between peersindependently from the total number of users in the VE,and guarantees global network connectivity since peers arealways connected to their enclosing neighbors even thoughthese neighbors reside outside peers’ AOI.

In the event of player crowding, however, mechanismsthat connect a peer to all its AOI neighbors might leadto situations where almost all peers become connected toeach other due to overlapping AOIs. To deal with this issue,

AOICast [76], built on VON, proposes to construct a span-ning tree over all AOI neighbors of a peer, thus reducingbandwidth usage at the cost of a higher latency.

VON however considers peer movements under theassumption that when a peer moves in the virtual world, allother peers surrounding the moving peer remain stationary.This assumption is needed so that the surrounding peers canbe used as reliable landmarks to update the underlying over-lay structure according to the peer’s movement. In addition,the layout of peers in the VE might overload some peersby forcing them to make direct connections beyond theircapacity. For instance, when in the VE a high number ofpeers are lying on a circle inside the center of which a peerp resides, then the peers on the circle become p’s enclos-ing neighbors and p should manage the direct connectionsto all its enclosing neighbors which may exceed p’s band-width capabilities [104] (Fig. 19). Also, in a highly dynamicenvironment, particularly in the context of P2P NVEs, theunderlying Voronoi structure produces high maintenancecost as it is subject to high connection change rate due tocontinuous players’ movement. As players tend to move in

Fig. 16 Solipsis‘03 [96]

(a) Convex hull (b) Convex hull construction

me

en

me’

e’n

e

e

0

1e

(c) Entrance of e

Page 17: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

e

e

Δ1

2Δf

Fig. 17 Solipsis‘03 - Undetected neighbor [100]

the VE, the network topology should continuously adapt toplayers’ movements such that the overlay structure is keptvalid. Maintenance cost can become particularly high whencrowding occurs, and a highly dynamic interaction with thesurrounding takes place.

8.1.4 Relaxed triangulation

To deal with high maintenance cost related to the under-lying overlay in the mutual notification approach, RelaxedTriangulation (RT) [97] proposes a new triangulation algo-rithm to connect peers based on their positions in the virtualworld. Relaxed Triangulation (RT) for a set of vertices Vin a 2-D plane is defined as a triangulation T(V) such thatno vertex of V is inside any triangle in T(V). In order tominimize the maintenance cost resulting from high connec-tion change rate due to user movements, RT maximizes theflip-free area of a node, namely the area inside which thenode’s movement does not trigger any edge flip operation.To this end, RT defines the flip-free area of a node as theregion composed of all the triangles of which the node isa vertex in RT. A flip operation thus occurs only when thenode passes across the base line of one of its triangles. The

Fig. 18 Neighbors in VON [94]

Fig. 19 VON - Peer connectivity in the worst case [104]

gray area in Figs. 20a and b shows point 0’s flip-free area inDelaunay and in RT, respectively. RT provides overlay net-work connectivity to support P2P NVEs while decreasingmaintenance overhead by reducing the number of connec-tion changes due to users’ movement. However, similar toVON, RT considers peer movements under the assumptionthat when a peer moves in the virtual world, all other peerssurrounding the moving peer remain stationary.

8.1.5 Red-Black Delaunay

Red-Black Delaunay [98] proposes a distributed NVE sys-tem based on a Delaunay graph [105] of peers’ positions inthe virtual world. Red-Black Delaunay constructs the sameoverlay structure as VON [94]. Red-Black Delaunay focuseson the construction of a valid Delaunay graph consideringseveral concurrent peers’ movements and clearly defines thegroups of moving peers and of the stationary peers used toupdate the overlay structure. Peers in the virtual world arerandomly grouped into two sets, called red and black. Foreach set, a separate Delaunay subgraph is constructed basedon the position of its peers in the virtual word (Fig. 21).A peer is directly connected not only to its same coloredDelaunay neighbors, but also to a group of different col-ored peers, called parents, which are responsible of updatingthe neighbor connections of the peer during its movement.A peer p in a subgraph becomes the parent of a peer q

(a) Delaunay Triangulation (b) Relaxed Triangulation

Fig. 20 Node 0’s flip-free area [97]

Page 18: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Fig. 21 Red-Black Delaunay [98]

in the other subgraph, if p’s Voronoi region in p’s corre-sponding subgraph intersects with q’s Voronoi region in q’scorresponding subgraph. If the peers in a subgraph movein the virtual world in times T , 2T , 3T , . . ., then the peersin the other subgraph move in times T/2, 3T/2, 5T/2, . . ..Therefore, alternately, when the peers in a subgraph changeposition in the virtual world, the peers in the other subgraphremain stationary and update the subgraph of the movingpeers by advising them about connection changes in theirsubgraph. The overall Delaunay graph including all peers inthe virtual word is then reconstructed based on the red andblack subgraphs.

Red-Black Delaunay constructs a valid Delaunay struc-ture that deals with peer movements based on the sequen-tial synchronized movement of peers in a subgraph (withrespect to peers in the other subgraph) and update ofthe moving subgraph by the stationary peers in the othersubgraph. The feasibility of the proposed scheme strictlydepends on the sequential movement and update phasesof each subgraph. In addition, in the event of crowdingof the same-colored peers, the parent peers are likely tobecome overloaded due to the update requirements of theirsubgraph.

8.2 Discussion

The mutual notification-based NVE systems presented inthis section are given in Table 6. Mutual notification canbe considered as a more specific approach of neighbor-list exchange. In the neighbor-list exchange approach,peers exchange their knowledge of the VE through regu-lar neighbor-list information exchanges regardless of thenecessity of this information on the receiver side. On theother hand, in mutual notification, peers cooperate with eachother on a specific task (e.g., neighbor discovery, globalnetwork connectivity) only when necessary. Since neighbor-list information might contain irrelevant and redundant data,mutual notification provides a more bandwidth efficientsolution. High scalability is achieved since peers maintaina limited number of connections with other peers based onthe underlying geometrical structure independently from thetotal number of peers in the VE. The direct connectivity

Table 6 The representative mutual notification based NVE systems

NVE systems Features

pSense [77] Based on Yao graph

Latency in message transmission

in case of crowding

Solipsis‘03 [96] Based on the angular relation between

peers’ positions

Undetected neighbor problem

VON [94] Based on Voronoi diagram

High overlay maintenance cost

One peer movement at a time

Relaxed Triangulation [97] Based on triangulation

Dealing with the maintenance cost

due to dynamism

Maximizing the flip-free area of a peer

One peer movement at a time

Red-Black Delaunay [98] Based on Delaunay

Focus on several concurrent peers’

movements

Based on sequential synchronized

movement of peer groups

High overlay maintenance cost

between peers in relation minimizes the delay of messagepropagation in the VE. However, the underlying structuremight cause high maintenance cost as it is subject to highconnection change rate due to the continuous movementof players. The management of several concurrent peers’movements remains an open issue. Moreover, the layout ofpeers in the VE might overload some peers by forcing themto make direct connections beyond their capacity.

9 Discussion and future perspectives

Table 7 summarizes the different approaches used to con-struct some of the P2P NVE systems presented in this paper,while Table 8 summarizes the characteristics of the differ-ent approaches discussed in this paper for the constructionof P2P NVE systems. Overall, the world partitioning andfully-connected approaches may suffer from bandwidthconsumption issues, while the DHT approach may intro-duce high latency, and the neighbor-list exchange approachdoes not guarantee global network connectivity. The mutualnotification approach supported by the multicast approachfor message diffusion and the world partitioning approachfor the management of less dynamic entities (in terms ofposition changes) can provide global network connectiv-ity, low latency due to direct connectivity between peersin relation and distributed bandwidth load among peers.

Page 19: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Table 7 P2P NVE systems’ characteristics

NVE systems World par. DHT Multicast Fully-connec. Neigh.-list exchan. Mutual notifi.

SimMud ‘04 [24] X X X

Event delivery ‘05 [33] X X

AVDNet ‘05 [39] X X

Federated P2P ‘04 [68] X X

Zoned federation ‘04 [46] X X

Badumna ‘10 [35] X X X

PBT ‘09 [36] X X X

N-Trees ‘05 [32] X X X

PIMTRE ‘08 [74] X X

COVER ‘06 [37] X

MOPAR ‘05 [31] X X

PastryMmog ‘06 [40] X X X

MM-VISA ‘09 [42] X X

VSM ‘08 [30] X

Solipsis ‘08 [44] X X

Colyseus ‘06 [56] X

Loader ‘08 [57] X X

Update dissemination ‘05 [69] X X

Apolo ‘05 [75] X

Frontier sets ‘06 [81] X

Donnybrook ‘08 [82] X

P2P message exch. ‘02 [90] X

Message intercha. protocol ‘05 [91] X

AOICast ‘08 [76] X X

pSense ‘08 [77] X X

Solipsis ‘03 [96] X

VON ‘06 [94] X

Relaxed triangulation ‘09 [97] X

QuON ‘09 [95] X

However, given the high dynamism of VEs in terms ofposition changes, rigid overlay structures (typified by theVoronoi diagram) introduce a high overlay maintenancecost.

From the above, and despite their several advantages,P2P designs for NVEs still suffer from several weak-nesses that are hindering their adoption by the NVE market.However, we believe that the main reason behind the non-adoption of P2P technologies for the support of NVEs isthe difficulty in providing a high level of availability andtrust. In an open system where nodes join and leave at will,dealing with failures, disconnections, and churn requiresa great degree of redundancy (and its associated consis-tency maintenance cost) to increase service availability.The trust issue is even more challenging in open envi-ronments. In client-server architectures, every user trustsat least the server. However, such trust may not exist ina P2P setting, making security issues very difficult to

solve without centralized control. Although several workshave been lately conducted to provide solutions capableof establishing a higher level of availability and trust inP2P NVEs [106–110], these efforts are still at their earlystages.

However, beyond the very specific characteristics, advan-tages, and drawbacks of P2P NVE approaches, it is prudentto take a step back and observe from a global perspective theplace that a decade of intensive scientific research on thistopic can have in today’s technological landscape. Givencurrent skyrocketing interest in cloud computing as a meansof dynamic provisioning of (virtually infinite) resources,one might wonder whether the P2P technology still has arole to play in today’s computing setting. In other words, thefollowing important question arises: Is there one distributedcomputing paradigm that will define future NVE architec-tures, or will there be a place for multiple paradigms,including P2P?

Page 20: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Tabl

e8

P2P

NV

Eap

proa

ches

’ch

arac

teri

stic

s

App

roac

hes

Adv

anta

ges

Dra

wba

cks

Inte

rest

ing

for

Wor

ldpa

rtit

ioni

ng•G

roup

ing

enti

ties

inte

rest

edin

each

othe

r•E

ffic

ient

size

dpa

rtit

ioni

ngis

sue

•Sta

tic

assi

gnm

ento

fen

titi

esto

man

ager

s

into

regi

ons

•Log

ical

loca

lity

pres

erve

dbe

twee

nen

titi

es•L

oad

bala

ncin

gon

man

ager

sth

roug

hau

xili

ary

mec

hani

sms

•Man

agem

ent

issu

eof

dyna

mic

enti

ties

DH

T•S

impl

eit

em-p

eer

assi

gnm

ent

•Hig

hla

tenc

ydu

eto

hops

onth

eD

HT

key

spac

e•S

tati

cas

sign

men

tof

enti

ties

tom

anag

ers

if

logi

call

ocal

ity

ispr

eser

ved

•Fai

rlo

addi

stri

buti

on•L

ogic

alre

lati

ons

betw

een

enti

ties

may

bede

stro

yed

•Sim

ple

rend

ez-v

ous

poin

tman

agem

ent

Mul

tica

st•F

air

dist

ribu

tion

ofm

essa

gedi

ffus

ion

load

•Man

agem

ent

issu

eof

dyna

mic

grou

pm

embe

rs•M

essa

gedi

ssem

inat

ion

inca

seof

crow

ding

•Hig

hla

tenc

ydu

eto

hops

betw

een

grou

pm

embe

rs

Full

y-co

nnec

ted

•Low

late

ncy

due

todi

rect

conn

ectiv

ity

betw

een

peer

s•O

verh

ead

due

toco

nnec

tivit

yto

allp

eers

•Man

agem

ent

ofpl

ayer

sin

tere

sted

inth

een

tire

VE

•Man

agem

ent

issu

eof

popu

lar

play

ers

Nei

ghbo

r-li

stex

chan

ge•L

imit

ednu

mbe

rof

conn

ecti

ons

betw

een

peer

s•G

loba

lnet

wor

kco

nnec

tivit

yth

roug

h

auxi

liar

ym

echa

nism

s

•Low

late

ncy

due

todi

rect

conn

ectiv

ity

•Hig

hov

erhe

addu

eto

irre

leva

ntan

d

betw

een

peer

sin

rela

tion

redu

ndan

tda

tatr

ansm

issi

on

Mut

ualn

otif

icat

ion

•Lim

ited

num

ber

ofco

nnec

tion

sbe

twee

npe

ers

•Man

agem

ent

issu

eof

conc

urre

nt•A

vata

rm

anag

emen

t

peer

mov

emen

ts

•Low

late

ncy

due

todi

rect

conn

ectiv

ity

•Hig

hov

erla

ym

aint

enan

ceco

st

betw

een

peer

sin

rela

tion

•Ban

dwid

thef

fici

entn

eigh

bor

disc

over

y

(com

pare

dto

neig

hbor

-lis

texc

hang

e)

Page 21: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

We strongly believe that no one distributed computingapproach will be able to meet the requirements of NVEapplications. Just like its client-server and P2P counterparts,cloud computing presents several limitations in the contextof NVEs with strong real-time interactivity requirements.Currently, two major cloud gaming models are emerging,with each having its advantages and drawbacks. In the firstmodel, the entire game software is hosted in the cloud,where heavy processing and graphics rendering are exe-cuted and sent back as video streams to users accessing thegame through thin clients [111, 112]. This gaming modelopens NVEs to a much wider user population by makingthe capabilities of users’ machines unimportant. However,streaming high quality interactive video to clients, espe-cially at a massive scale, is confronted with bandwidthlimitations and cost as major bottlenecks. Also, user qualityof experience relies solely on Internet connection quality,including upload/download speeds, connection stability, aswell as latency. Also, traditional delay compensation tech-niques, such as dead reckoning, used on the client side areno longer possible when the complete game software is runremotely in the cloud. This is a critical issue in an Inter-net environment where latencies are not predictable. Thesecond cloud gaming model avoids the above-mentionedissues by hosting in the cloud only the game server softwarerunning the game logic and handling players’ interactions,while keeping graphics rendering on clients’ machines.This, however, makes high-end computers or particularhardware necessary to process the client software for thegame.

Beyond the advantages of one over the other, both cloudgaming models face the following major challenge: howto best exploit the elastic character of cloud computing toanswer the strong scalability requirements of NVEs.

Current solutions deal with the scalability issue throughrigid parallelization techniques of players’ interactions. Themain technique is sharding, which involves creating sev-eral completely separate instances, or shards, of the virtualworld, and assigning each player to a single shard basedon some criteria, such as players’ geographical regions orskill levels. However, this goes against the idea of a “sin-gle unified virtual world” that all users connect to, with thepotential of interacting with millions of other users world-wide instead of being limited to users of the same worldinstance.

In light of the above discussion, we strongly believethat alternative parallelization techniques of players’ inter-actions are required to best exploit the elastic capabili-ties that cloud infrastructures can offer. In particular, thegame server software should be engineered in a way thatallows the server to elastically stretch and shrink its useof cloud resources. In this respect, P2P technologies areextremely relevant in their extraordinary capacities for

self-organization and self-healing in dynamic environmentswhere peers (and their associated resources) continuouslyjoin and leave the network. We note in particular that:

– The world partitioning approach has a huge potentialfor implementation in data-centers. Many of the algo-rithms that have been described in Section 3 couldsignificantly enhance the rigid sharding process whichis implemented today. Flexible area management andload-balancing among area managers are among themost attractive algorithms.

– The DHT-based approach has high potential given thatits main drawback in the context of P2P systems –highlatency– is no longer an issue in data-centers. Similarlyto Facebook’s Cassandra system that is based on a cus-tomized DHT, we envision promising research activitiesin the customization of the previous works described inSection 4 in the context of data-centers.

– The mutual notification and neighbor-list exchangeapproaches are in our opinion less attractive. Though,an architecture where each avatar runs on a virtualmachine in a data-center makes sense. Most of the pre-vious works described in Section 8 focus on hypothesesthat are not accurate in data-centers, namely the absenceof central authority and peer churn, and are thus notdirectly applicable to data-centers.

Another argument in favor of a solution based on bothcloud and P2P computing relates to the game streamingmodel (i.e., the first cloud gaming model discussed above),where latency and bandwidth consumption are the key bot-tlenecks. Given that P2P solutions have shown to be veryeffective for content streaming, we believe that an inter-esting solution would be to augment the cloud resourceswith a P2P streaming solution where the streaming loadcan be offloaded to the P2P network, thus leveraging peers’bandwidth and reducing latency.

Finally, we would like to emphasize that the libertar-ian motivations for building authority-less NVEs is stillviable. Even if this libertarian motivation is less attractivefrom an economical perspective, we believe that there is aneed to perform research with this objective in mind. Therecent proliferation of projects related to peer-to-peer socialnetworks (for example [113]), and the fear that current dom-inant web giants are provoking, emphasizes the critical needfor research efforts on this area.

10 Conclusion

This paper analyzes the key design approaches to con-struct a P2P NVE system. We discuss each approach, itsadvantages and drawbacks, and examine several representa-tive P2P NVE systems based on the presented approaches.

Page 22: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Despite all the nice characteristics of the P2P approach,we believe that a solution that provides all the necessaryrequirements of NVE systems cannot be achieved follow-ing a pure P2P design choice. However, we argue that futureNVE architectures will build on the strengths of severaldistributed computing paradigms to overcome the shortcom-ings of adopting only one. In particular, a very promisingresearch direction would be to look into hybrid solutionswhere the self-organizing and self-healing nature of P2Papproaches can be exploited on top of cloud resources tobest exploit the elastic capabilities of cloud computing.

References

1. Doom. http://en.wikipedia.org/wiki/Doom (video game)2. Quake Live. http://www.quakelive.com/#!home3. Marathon. http://en.wikipedia.org/wiki/Marathon (video game)4. Blake Stone: aliens of gold. http://en.wikipedia.org/wiki/Blake

Stone: Aliens of Gold5. Star wars: dark forces. http://en.wikipedia.org/wiki/Star Wars:

Dark Forces6. Homeworld. http://en.wikipedia.org/wiki/Homeworld7. Starcraft II: wings of liberty. http://us.battle.net/sc2/en/8. Company of heroes. http://www.companyofheroes.com/9. Ultima online. http://www.uo.com/ageofshadows/viscent.html

10. EverQuest. http://everquest.station.sony.com/11. Final fantasy XI. http://www.playonline.com/ff11eu/index.shtml12. EVE online. http://www.eveonline.com/13. World of warcraft. http://us.battle.net/wow/en/14. Calvin J, Dickens A, Gaines B, Metzger P, Miller D, Owen

D (1993) The simnet virtual world architecture. In: Annualinternational symposium virtual reality IEEE, pp 450–455

15. Pullen JM, Wood DC (1995) Networking technology and dis.IEEE 83:1156–1167

16. Institute of Electrical and Electronic Engineers (2000) IEEE stan-dard for modeling and simulation (m&s) high level architecture(hla)—framework and rules. IEEE Std. 1516-2000, pp i–22

17. FALCON 4.0. http://en.wikipedia.org/wiki/Falcon 4.018. America’s army. http://www.americasarmy.com/19. Second life official site. http://secondlife.com/20. HiPiHi. http://www.hipihi.com/21. The sims online. http://thesims.com/en us/home22. There. http://www.there.com/23. Active worlds. http://www.activeworlds.com/24. Knutsson B, Lu H, Xu W, Hopkins B (2004) Peer-to-peer support

for massively multiplayer games. In: Proceeding the 23rd con-ference of the IEEE communications society, INFOCOM ’04, p107

25. Beigbeder T, Coughlan R, Lusher C, Plunkett J, Agu E, ClaypoolM (2004) The effects of loss and latency on user performance inunreal tournament 2003®. In: Proceeding 3rd ACM SIGCOMMworkshop on network and system support for games, NetGames’04. ACM, New York, pp 144–151

26. Macedonia MR, Zyda MJ, Pratt DR, Brutzman DP, Barham PT(1995) Exploiting reality with multicast groups. IEEE ComputGraph Appl 15(5):38–45

27. Funkhouser TA (1995) RING: a client-server system for multi-user virtual environments. In: Proceeding the 1995 symposiumon interactive 3D graphics, SI3D ’95. ACM, New York, pp 85–ff

28. Barrus JW, Waters RC, Anderson DB (1996) Locales: supportinglarge multiuser virtual environments. IEEE Comput Graph Appl16(6):50–57

29. Abrams H, Watsen K, Zyda M (1998) Three-tiered interest man-agement for large-scale virtual environments. In: Proceeding theACM symposium on virtual reality software and technology,VRST ’98. ACM, New York, pp 125–129

30. Hu S-Y, Chang S-C, Jiang J-R (2008) Voronoi state manage-ment for peer-to-peer massively multiplayer online games. In:5th IEEE consumer communications and networking conference,CCNC ’08, pp 1134–1138

31. Yu AP, Vuong ST (2005) MOPAR: a mobile peer-to-peer overlayarchitecture for interest management of massively multiplayeronline games. In: Proceeding the international workshop on net-work and operating systems support for digital audio and video,NOSSDAV ’05. ACM, New York, pp 99–104

32. GauthierDickey C, Lo V, Zappala D (2005) Using N-Trees forscalable event ordering in peer-to-peer games. In: Proceeding theinternational workshop on network and operating systems sup-port for digital audio and video, NOSSDAV ’05. ACM, NewYork, pp 87–92

33. Yamamoto S, Murata Y, Yasumoto K, Ito M (2005) A distributedevent delivery method with load balancing for MMORPG. In:Proceeding 4th ACM SIGCOMM workshop on network and sys-tem support for games, NetGames ’05. ACM, New York, pp 1–8

34. Rieche S, Wehrle K, Fouquet M, Niedermayer H, Petrak L, CarleG (2007) Peer-to-peer-based infrastructure support for massivelymultiplayer online games. In: 4th annual IEEE consumer com-munications and networking conference, CCNC ’07

35. Kulkarni S, Douglas S, Churchill D (2010) Badumna: a decen-tralised network engine for virtual environments. Comput Netw54(12):1953–1967. P2P technologies for emerging wide-areacollaborative services and applications

36. De Chiara R, Erra U, Scarano V (2009) Some considerationson the design of a P2P infrastructure for massive simulations.In: International conference on ultra modern telecommunication,ICUMT ’09

37. Morillo P, Moncho W, Orduna JM, Duato J (2006) Providingfull awareness to distributed virtual environments based on peer-to-peer architectures. In: Proceeding the 24th computer graph-ics international conference, lecture notes in computer science,CGI ’06, vol 4035. Springer, Berlin, pp 116–127

38. Varvello M, Diot C, Biersack E (2009) A walkable Kademlianetwork for virtual worlds. In: International workshop on peer-to-peer systems, IPTPS ’09

39. Correia RCM, Pellegrino SRM (2005) Scalable distributed vir-tual environments using multicast application level. In: Proceed-ing the ninth international conference on computer supportedcooperative work in design, vol 1, pp 140–145

40. Hampel T, Bopp T, Hinn R (2006) A peer-to-peer architecturefor massive multiplayer online games. In: Proceeding 5th ACMSIGCOMM workshop on network and system support for games,NetGames ’06. ACM, New York, pp 48

41. Chen A, Muntz RR (2006) Peer clustering: a hybrid approach todistributed virtual environments. In: Proceeding 5th ACM SIG-COMM workshop on network and system support for games,NetGames ’06. ACM, New York, p 11

42. Ahmed DT, Shirmohammadi S, Oliveira JC (2009) A hybrid p2pcommunications architecture for zonal mmogs. Multimed ToolsAppl 45(1–3):313–345

43. Tumbde A, Venugopalan S (2004) A voronoi partitioningapproach to support massively multiplayer online games. Tech-nical Report CS 740 Project. University of Wisconsin, Madison

44. Frey D, Royan J, Piegay R, Kermarrec A-M, AnceaumeE, Le Fessant F (2008) Solipsis: a decentralized architec-ture for virtual environments. In: Proceeding the 1st interna-tional workshop on massively multiuser virtual environments,MMVE ’08

Page 23: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

45. Denault A, Canas C, Kienzle J, Kemme B (2011) Triangle-basedobstacle-aware load balancing for massively multiplayer games.In: Proceedings of the 10th annual workshop on network and sys-tems support for games, NetGames ’11. IEEE Press, Piscataway,pp 4:1–4:6

46. Iimura T, Hazeyama H, Kadobayashi Y (2004) Zoned federationof game servers: a peer-to-peer approach to scalable multi-playeronline games. In: Proceeding ACM SIGCOMM workshop onnetwork and system support for games, NetGames ’04. ACM,New York, pp 116–120

47. Chan L, Yong J, Bai J, Leong B, Tan R (2007) Hydra: amassively-multiplayer peer-to-peer architecture for the gamedeveloper. In: Proceeding the 6th ACM SIGCOMM workshopon network and system support for games, NetGames ’07. ACM,New York, pp 37–42

48. Kim KC, Yeom I, Lee J (2004) HYMS: a hybrid MMOG serverarchitecture. IEICE Trans Inf Syst E87-D(12):2706–2713

49. Fan L, Taylor H, Trinder P (2007) Mediator: a design frame-work for P2P MMOGs. In: Proceeding the 6th ACM SIGCOMMworkshop on network and system support for games, NetGames’07. ACM, New York, pp 43–48

50. Botev J, Hohfeld A, Schloss H, Scholtes I, Sturm P, Esch M(2008) The hyperVerse—concepts for a federated and torrent-based ‘3D Web’. Int J Adv Media Commun 2(4):331–350

51. Kulkarni S (2009) Badumna network suite: a decentralized net-work engine for massively multiplayer online applications. In:9th international conference on peer-to-peer computing, IEEEP2P ’09

52. Rowstron A, Druschel P (2001) Pastry: scalable, distributedobject location and routing for large-scale peer-to-peer systems.In: Proceeding IFIP/ACM international conference on distributedsystems platforms. Middleware, pp 329–350:329–350

53. Ratnasamy S, Francis P, Handley M, Karp R, Schenker S (2001)A scalable content-addressable network. In: Proceeding the 2001conference on applications, technologies, architectures, and pro-tocols for computer communications, SIGCOMM ’01. ACM,New York, pp 161–172

54. Stoica I, Morris R, Karger D, Kaashoek MF, Balakrishnan H(2001) Chord: a scalable peer-to-peer lookup service for internetapplications. In: Proceeding the 2001 conference on applica-tions, technologies, architectures, and protocols for computercommunications, SIGCOMM ’01. ACM, New York, pp 149–160

55. Tanin E, Harwood A, Samet H, Nayar D, Nutanong S (2006)Building and querying a P2P virtual world. Geoinformatica10(1):91–116

56. Bharambe A, Pang J, Seshan S (2006) Colyseus: a distributedarchitecture for online multiplayer games. In: Proceedings the3rd conference on networked systems design & implementation,NSDI ’06. USENIX Association, Berkeley, pp 12–12

57. Hariri B, Shirmohammadi S, Pakravan MR (2008) LOADER:a location-aware distributed virtual environment architecture.In: IEEE conference on virtual environments, human-computerinterfaces and measurement systems, VECIMS ’08, pp 97–101

58. Ratti S, Hariri B, Shirmohammadi S (2008) NL-DHT: a non-uniform locality sensitive DHT architecture for massively multi-user virtual environment applications. In: Proceeding the 200814th IEEE international conference on parallel and distributedsystems, ICPADS ’08. IEEE Computer Society, Washington,pp 793–798

59. Castro M, Druschel P, Kermarrec AM, Rowstron AIT (2002)SCRIBE: a large-scale and decentralized application-level mul-ticast infrastructure. IEEE J Sel Areas Commun 20(8):1489–1499

60. Bharambe AR, Agrawal M, Seshan S (2004) Mercury: support-ing scalable multi-attribute range queries. SIGCOMM ComputCommun Rev 34(4):353–366

61. Delaney D, Ward T, McLoone S (2006) On consistency and net-work latency in distributed interactive applications: a survey–partI. Presence: Teleoper Virtual Environ 15(2):218–234

62. Lety E, Turletti T, Baccelli F (2004) SCORE: a scalablecommunication protocol for large-scale virtual environments.IEEE/ACM Trans Networking 12(2):247–260

63. Frecon E (2004) DIVE: communication architecture and pro-gramming model. IEEE Commun Mag 42(4):34–40

64. de Oliveira JC, Georganas ND (2002) VELVET: an adaptivehybrid architecture for very large virtual environments. In: IEEEinternational conference on communications, ICC ’02, vol 4,pp 2491–2495

65. Nguyen CD, Safaei F, Boustead P (2004) Performance evalua-tion of a proxy system for providing immersive audio communi-cation to massively multi-player games. In: IEEE global telecom-munications conference workshops, GlobeCom workshops 2004,pp 192–199

66. Chu Y-h, Rao SG, Seshan S, Zhang H (2002) A case for endsystem multicast. IEEE J Sel Areas Commun 20(8):1456–1471

67. Vogel J, Widmer J, Farin D, Mauve M, Effelsberg W (2003)Priority-based distribution trees for application-level multicast.In: Proceeding the 2nd workshop on network and system supportfor games, NetGames ‘03. ACM, New York, pp 148–157

68. Rooney S, Bauer D, Deydier R (2004) A federated peer-to-peernetwork game architecture. IEEE Commun Mag 42(5):114–122

69. Chen F, Kalogeraki V (2005) Adaptive real-time update dis-semination in distributed virtual simulation environments. In:Proceeding the eighth IEEE international symposium on object-oriented real-time distributed computing, ISORC ‘05. IEEEComputer Society, Washington, pp 233–236

70. Singhal SK (2007) Scalable networked virtual environmentsusing unstructured overlays. In: Proceeding the 13th internationalconference on parallel and distributed systems, ICPADS ’07.IEEE Computer Society, Washington, pp 1–8

71. Ahmed DT, Shirmohammadi S, de Oliveira JC (2006) A novelmethod for supporting massively multi-user virtual environ-ments. In: IEEE international workshop on haptic audio visualenvironments and their applications, HAVE ’06, pp 72–77

72. Ahmed DT, Shirmohammadi S (2010) A fault tolerance proce-dure for p2p online games. In: 10th international conference,information sciences signal processing and their applications(ISSPA), 2010, pp 614–617

73. Ito S, Saito H, Sogawa H, Tobe Y (2006) A propagation of virtualspace information using a peer-to-peer architecture for massivelymultiplayer online games. In: 26th IEEE international conferenceon distributed computing systems workshops, ICDCS workshops2006, p 44

74. Xiang-bin S, Yue W, Qiang L, Ling D, Fang L (2008) An inter-est management mechanism based on N-Tree. In: Proceeding the2008 9th ACIS international conference on software engineer-ing, artificial intelligence, networking, and parallel/distributedcomputing, SNPD ’08. IEEE Computer Society, Washington,pp 917–922

75. Lee J, Lee H, Ihm S, Gim T, Junehwa S (2005) APOLO:ad-hoc peer-to-peer overlay network for massively multi-playeronline games. Technical report cs/tr-2005-248. Korea AdvancedInstitute Of Science And Technology, (KAIST)

76. Jiang J-R, Huang Y-L, Hu S-Y (2008) Scalable AOI-cast forpeer-to-peer networked virtual environments. In: Proceeding the28th international conference on distributed computing systemsworkshops, ICDCSW ’08. IEEE Computer Society, Washington,pp 447–452

77. Schmieg A, Stieler M, Jeckel S, Kabus P, Kemme B, BuchmannA (2008) pSense - maintaining a dynamic localized peer-to-peerstructure for position based multicast in games. In: Proceeding

Page 24: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

the 2008 eighth international conference on peer-to-peer comput-ing, P2P ’08. IEEE Computer Society, Washington, pp 247–256

78. Diot C, Levine BN, Lyles B, Kassem H, Balensiefen D (2000)Deployment issues for the IP multicast service and architecture.IEEE Netw 14(1):78–88

79. Hosseini M, Ahmed DT, Shirmohammadi S, Georganas ND(2007) A survey of application-layer multicast protocols. IEEECommun Surv Tutor 9(3):58–74

80. Lao L, Cui J-H, Gerla M, Chen S (2007) A scalable overlaymulticast architecture for large-scale applications. IEEE TransParallel Distrib Syst 18:449–459

81. Steed A, Angus C (2006) Enabling scalability by partitioning vir-tual environments using frontier sets. Presence: Teleoper VirtualEnviron 15(1):77–92

82. Bharambe A, Douceur JR, Lorch JR, Moscibroda T, Pang J,Seshan S, Zhuang X (2008) Donnybrook: enabling large-scale,high-speed,peer-to-peer games. In: Proceeding the ACM SIG-COMM 2008 conference on data communication, SIGCOMM’08. ACM, New York, pp 389–400

83. Steed A, Angus C (2005) Supporting scalable peer to peer vir-tual environments using frontier sets. In: Proceeding the 2005IEEE conference 2005 on virtual reality, VR ’05. IEEE ComputerSociety, pp 27–34

84. Makbily Y, Gotsman C, Bar-Yehuda R (1999) Geometric algo-rithms for message filtering in decentralized virtual environ-ments. In: Proceeding the 1999 symposium on Interactive 3Dgraphics, I3D ’99. ACM, New York, pp 39–46

85. Goldin A, Gotsman C (2004) Geometric message-filtering proto-cols for distributed multiagent environments. Presence: TeleoperVirtual Environ. 13(3):279–295

86. Teller SJ, Sequin CH (1991) Visibility preprocessing for interac-tive walkthroughs. SIGGRAPH Comput Graph 25(4):61–70

87. Robson JG, Graham N (1981) Probability summation andregional variation in contrast sensitivity across the visual field.Vis Res 21(3):409–418

88. Cowan N (2001) The magical number 4 in short-term memory:a reconsideration of mental storage capacity. Behav Brain Sci24(01):87–114

89. Manku GS, Naor M, Wieder U (2004) Know thy neighbor’sneighbor: the power of lookahead in randomized P2P networks.In: Proceeding the thirty-sixth annual ACM symposium on the-ory of computing, STOC ’04. ACM, New York, pp 54–63

90. Kawahara Y, Morikawa H, Aoyama T (2002) A peer-to-peermessage exchange scheme for large scale networked virtualenvironments. In: Proceeding IEEE international conference oncommunication systems, ICCS ’02. IEEE Computer Society,Washington, pp 957–961

91. Chen J-F, Lin W-C, Bai H-S, Dai S-Y (2005) A message inter-change protocol based on routing information protocol in avirtual world. In: Proceeding the 19th international conference onadvanced information networking and applications, AINA ’05.IEEE Computer Society, Washington, pp 377–384

92. Kawahara Y., Aoyama T, Morikawa H (2004) A peer-to-peermessage exchange scheme for large-scale networked virtualenvironments. Telecommun Syst 25(3–4):353–370

93. Matsumoto N, Kawahara Y, Morikawa H, Aoyama T (2004) Ascalable and low delay communication scheme for networkedvirtual environments. In: Proceeding IEEE international work-shop on networking issues in multimedia entertainment, NIME’04, pp 529–535

94. Hu S-Y, Chen J-F, Chen T-H (2006) VON: a scalable peer-to-peer network for virtual environments. IEEE Netw 20(4):22–31

95. Backhaus H, Krause S (2009) QuON—a quad-tree based over-lay protocol for distributed virtual worlds. In: Proceeding the 2ndinternational workshop on massively multiuser virtual environ-ments, MMVE ’09.

96. Keller J, Simon G (2003) Solipsis: a massively multi-participantvirtual world. In: Proceeding the international conference onparallel and distributed processing techniques and applications,PDPTA ’03, vol 1. CSREA Press, pp 262–268

97. Buyukkaya E, Abdallah M (2009) Efficient triangulation for p2pnetworked virtual environments. Multimedia Tools Appl 45:291–312

98. Ghaffari M, Hariri B, Shirmohammadi S (2009) A delaunay tri-angulation architecture supporting churn and user mobility inMMVEs. In: Proceeding the 18th international workshop on net-work and operating systems support for digital audio and video,NOSSDAV ‘09. ACM, New York, pp 61–66

99. Yao AC-C (1982) On constructing minimum spanning trees ink-dimensional spaces and related problems. SIAM J Comput11(4):721–736

100. Simon G (2004) Conception et realisation d’un systeme pourenvironnement virtuel massivement partage. PhD thesis, Univer-site de Rennes 1

101. Keller J, Simon G (2002) Toward a peer-to-peer shared vir-tual reality. In: Proceeding the 22nd international conference ondistributed computing systems, ICDCSW ’02. IEEE ComputerSociety, Washington, pp 695–700

102. Hu S-Y, Liao G-M (2004) Scalable peer-to-peer networkedvirtual environment. In: Proceeding 3rd ACM SIGCOMM work-shop on network and system support for games, NetGames ‘04.ACM, New York, pp 129–133

103. Aurenhammer F (1991) Voronoi diagrams–a survey of a funda-mental geometric data structure. ACM Comput Surv 23(3):345–405

104. Shun-Yun Hu (2005) Scalable peer-to-peer networked virtualenvironment. Master’s thesis, Tamkang University

105. de Berg M, van Kreveld M, Overmars M, Schwarzkopf O (1997)Computational geometry: algorithms and applications. Springer-Verlag

106. GauthierDickey C, Zappala D, Lo V, Marr J (2004) Low latencyand cheat-proof event ordering for peer-to-peer games. In: Pro-ceeding of the 14th international workshop on Network and oper-ating systems support for digital audio and video, NOSSDAV’04. ACM, New York, pp 134–139

107. Chan M-C, Hu S-Y, Jiang J-R (2008) An efficient and secureevent signature (eases) protocol for peer-to-peer massively mul-tiplayer online games. Comput Netw 52:1838–1845

108. Liu H-I, Lo Y-T (2008) Dacap—a distributed anti-cheating peerto peer architecture for massive multiplayer on-line role playinggame. In: 8th IEEE international symposium, cluster computingand the grid CCGRID ’08, pp 584–589

109. Lin Y, Kemme B, Patino-Martinez M, Jimenez-Peris R (2006)Applying database replication to multi-player online games. In:Proceeding of 5th ACM SIGCOMM workshop on Network andsystem support for games, NetGames ’06., ACM, New York

110. Griwodz C (2002) State replication for multiplayer games.In: Proceedings of the 1st workshop on network and systemsupport for games, NetGames ’02. ACM, New York, pp 29–35

111. OnLive. http://www.onlive.com/112. Gaika. http://www.gaikai.com/113. PeerSon. http://www.peerson.net/

Page 25: A survey of peer-to-peer overlay approaches for networked virtual environments

Peer-to-Peer Netw. Appl.

Eliya Buyukkaya receivedthe B.Sc. degree in ComputerEngineering from Middle EastTechnical University, Turkey,in 2003 and the M.Sc. andPh.D. degrees in ComputerScience from University Paris6, France, in 2006 and 2011,respectively. She worked as apostdoctoral research fellow inthe Computer Science depart-ment at Telecom Bretagne fortwo years. Since September2012, she is research and

teaching fellow in the Computer Science department at ENSSAT, Uni-versity Rennes 1. Her research interests are P2P systems, distributedsystems, graph databases, social networks and virtual environments.

Maha Abdallah is currentlyAssociate Professor in thedepartment of Computer Sci-ence (LIP6) at Pierre andMarie Curie University (Paris,France). She received an MScdegree in Computer Sciencefrom Pierre and Marie CurieUniversity in 1996, and a PhD(Honors) degree in ComputerScience from the Universityof Versailles in 2001. Herresearch interests are in thegeneral area of distributed and

cloud computing, covering distributed algorithms and systems, dis-tributed databases, fault-tolerant systems, and scalable systems andnetworks, as well as the application of these technologies to interactive3D media applications.

Gwendal Simon received hisMaster Degree in ComputerScience in 2000 and his PhDdegree in Computer Sciencein December 2004 from Uni-versity of Rennes 1 (France).During his PhD, he worked atOrange R&D Labs, where hedeveloped a distributed sharedvirtual world, which hasreceived a worldwide atten-tion, including innovationprize, and has turned out to bea start-up. From 2004 to 2006he was a researcher atOrange Labs, where he

worked on spontaneous networks and social media innova-tions. Since 2006, he has been Associate Professor at TelecomBretagne, a graduate engineering school within the InstitutMines-Telecom. He has been a visiting researcher at Univer-sity of Waterloo from September 2011 to September 2012. Hisresearch interests include large-scale distributed networks, peer-to-peer and ad-hoc networks, optimization problems and videodelivery systems.