Upload
gwendal
View
214
Download
0
Embed Size (px)
Citation preview
Peer-to-Peer Netw. Appl.DOI 10.1007/s12083-013-0231-5
A survey of peer-to-peer overlay approaches for networkedvirtual environments
Eliya Buyukkaya · Maha Abdallah · Gwendal Simon
Received: 20 January 2013 / Accepted: 30 August 2013© Springer Science+Business Media New York 2013
Abstract This paper identifies and analyzes the key designapproaches that have been proposed over the last decadeto construct a peer-to-peer (P2P) networked virtual envi-ronment (NVE) systems. We discuss each approach, theiradvantages and drawbacks, and examine several represen-tative P2P-based NVE systems based on the presentedapproaches, thus providing the background and literaturereview of P2P NVEs.
Keywords Networked virtual environments · P2P systems
1 Introduction
During the past decade, the academic community has exten-sively studied peer-to-peer (P2P) networked virtual environ-ments (NVEs). This research activity has been driven bythe phenomenal success of other P2P applications, typifiedby file-sharing applications (e.g., Bit-Torrent), and videostreaming systems (e.g., PPLive). However P2P architec-tures have not yet found their ways in NVEs, partly becausethe development of fully-distributed NVEs has turned out tobe especially challenging. In this paper, we review over tenyears of research related to the design of fully distributedinfrastructure-less NVEs. Our goal is to give a thorough
E. Buyukkaya (�)ENSSAT, University Rennes 1, Rennes, Francee-mail: [email protected]
M. AbdallahLIP6, Pierre and Marie Curie University, Paris, Francee-mail: [email protected]
G. SimonInstitut Telecom, Telecom Bretagne, Plouzane, Francee-mail: [email protected]
description of the main approaches that have been exploredso far, and to emphasize their main advantages and theirweaknesses. We think that it is the right time to survey in afair manner these approaches because technological shifts,in particular the development of cheap computing and stor-age facilities in data-centers, have made P2P NVE systemsless attractive. We discuss the perspectives around theseworks for the next years. We believe that distributed algo-rithms for the management of large-scale NVEs are stillneeded, in a context that is just slightly different. We discussthese perspectives at the end of the paper in a more personalopinion.
1.1 Fundamentals about NVEs
Networked Virtual Environments (NVEs) can be classi-fied into three major categories: multiplayer online games,military simulations, and social virtual worlds.
• Among the most popular NVEs, multiplayer onlinegames are computerized games in which a high num-ber of players play in the same game environment atthe same time. Multiplayer online games can be furtherclassified into First Person Shooter, Real-Time Strat-egy and Role-Playing games. Below are some typicalexamples:
– First Person Shooter: Doom [1], Quake Live [2],Marathon [3], Blake Stone: Aliens of Gold [4], StarWars: Dark Forces [5].
– Real-Time Strategy: Homeworld [6], Starcraft II:Wings of Liberty [7], Company of Heroes [8].
– Role-Playing: Ultima Online [9], EverQuest [10],Final Fantasy XI [11], EVE online [12], World ofWarcraft [13].
Peer-to-Peer Netw. Appl.
• NVEs originated from military simulations designed fortraining purposes. Typical examples are SIMNET [14],DIS [15], HLA [16], FALCON 4.0 [17] and America’sArmy [18].
• Social virtual worlds are online communities livingin computerized virtual environments and aiming atsocializing with other users via mutual interactionswithin the virtual world. Typical examples are Sec-ond Life [19], HiPiHi [20], The Sims Online [21],There [22] and Active Worlds [23].
A typical virtual world is made up of immutable objects(e.g., terrain geometry, textures), mutable objects (e.g.,food, weapons, tools), characters controlled by players (i.e.,avatars) and non-player characters controlled by automatedalgorithms [24]. The goal of a NVE system is to provide toits participants a shared sense of space, time, and presencethrough virtual representations, called avatars. This idea hasderived into a set of non-functional requirements. In partic-ular, users observing the same virtual scene need to havethe same view on data at one time (i.e., consistency). ANVE should respond to users’ events at a fast rate (i.e.,responsiveness), since high latencies degrade user experi-ence; for instance, first person shooter games cannot toleratelatencies above 200 milliseconds [25]. In addition, usersshould be prevented from cheating, and users’ data shouldbe protected from being changed (intentionally or uninten-tionally) by other users (i.e., security). Content within NVEsshould also remain available across different user sessions,so in-world objects’/avatars’ data should be stored on a per-sistent storage device (i.e., persistence). NVEs should beable to survive failures in order to continuously provide theexpected service (i.e., availability). As NVEs are becomingincreasingly popular, they are expected to support a highnumber of simultaneous participants (i.e., scalability), andconsequently deal with potentially highly populated regionsin the virtual world (i.e., load-balancing).
To address these requirements, NVEs have traditionallybeen developed on client/server architectures. The server(or a cluster of servers) maintains the world data, calculatesstate changes and disseminates state updates to the differ-ent users. A user informs the server about the events thathe/she creates and is informed by the server about statechanges of players and objects in the virtual world. Since aplayer is only interested in the portion of the virtual worldthat is located near the player’s avatar, the server usuallyonly sends information about events that occurred within theArea of Interest (AoI).
1.2 Peer-to-peer NVEs
The scientific community has advanced peer-to-peer (P2P)systems as an alternative design choice for building NVEs.
We identify three main motivations behind this choice. Firstis the perspective of cheap scalability. Centralized sys-tems can indeed lead to communication and computationaloverhead on the servers. To support an ever growing userpopulation, the only solution is to increase the number ofservers. To get rid of expensive infrastructure has become akey driver for scientists. The second motivation is related toavoid possible points of failure. Fully distributed architec-tures exhibit extra-ordinary self-healing and self-stabilizingproperties. Since maintaining the redundancy of such large-scale applications is hard, scientists proposed P2P solutions.Finally, the third motivation refers to the libertarian vision.Like in social networks, users are concerned by the factthat a central authority owns the VE, tracks all the activ-ities of users, potentially uses logs for profit, and mightuse censorships on the activities within the virtual world. AP2P infrastructure appears as a way to build an authority-less NVE, where users are free to act and have controlover several important aspects of their participation tothe NVE.
In P2P systems, the overall system load is distributedamong all participating users/peers, which organize them-selves so that they all have an equal role and act as bothclients and servers. Since in a virtual world a player is onlyinterested in its AOI, a key issue in a NVE is to answer thefollowing question: how can each player be informed of thestate changes of other players and objects within its AOIwhile moving around the world? In other words, how can astate update message be sent to relevant players?
To add to complexity, P2P NVEs bring some addi-tional constraints that are inherent to the fact that thesoftware is running on the machines of end-users. Giventhe relatively limited (upload) bandwidth and processingpower of users’ machines, P2P NVE designers tend tolimit the number of connections as well as the numberof exchanged messages between participating peers. More-over, network latency and message losses make maintainingstrict consistency extremely challenging. Finally, direct con-nections between interacting players should be preferredbecause the transmission of a message through severalintermediate peers leads to high message disseminationdelays.
1.3 The idea behind this survey
As illustrated by the number of references in the presentdocument, numerous P2P-based NVE systems have beenproposed. In this paper, we focus on recent efforts to buildP2P overlay networks for the full support of NVEs. Thismeans that, for instance, special mechanisms that deal withphysical network latency are out of the scope of this surveyeven if they refer to both P2P and NVEs. In other words,we do not review the works that improve one specific
Peer-to-Peer Netw. Appl.
non-functional requirement in coordination with a tradi-tional client-server system. Instead, we are interested in sys-tems that build their own software infrastructure by lever-aging exclusively on the machines of end users. At a timewhere large-scale distributed systems are more than everdeployed, we believe that leveraging on lessons learned andexperiences gained from a decade of intensive efforts andscientific works through a comprehensive state-of-the-art isnecessary. Most of these previous works did not envision tobe used in what is now known as the cloud. Nonetheless,they are of high interest in the perspective of deployment ofNVEs in large-scale distributed infrastructures.
We identify six key approaches to construct the over-lay network: the world partitioning, DHT, multicast, fully-connected, neighbor-list exchange and mutual notificationapproaches. Given the plethora of existing efforts in thesedirections, the present paper is not intended to be exhaus-tive of all existing related works, but rather to study, throughrepresentative examples of each approach, the general orga-nization of the overlay and the message schemes that suchoverlay implies. More precisely, we describe (i) the mech-anisms that allow the overall system to stay connected, (ii)
the main schemes for message exchange, and (iii) the distri-bution of data. We summarize at the end of each Section themain advantages and drawbacks of the approach. We discusstheir comparative interest for the future in the Discussionpart.
The remainder of this paper is organized as follows.Section 2 provides a brief overview of the different P2PNVE approaches. Sections 3, 4, 5, 6, 7 and 8 providea detailed discussion of each approach, its advantagesand drawbacks, and examine several representative P2PNVE systems based on the presented approaches. Finally,Section 9 provides a critical analysis of the presentedapproaches and Section 10 concludes this work.
2 P2P NVE approaches: A brief overview
The different P2P NVE approaches are briefly introduced inthe following.
World partitioning approach partitions the virtual worldinto regions, which group a set of players potentially inter-ested in each other. The manager of a region manages peersand objects inside its region. The main issues are effi-cient sized partitioning of the world and the management ofplayer crowding.
DHT approach uses Distributed hash tables (DHTs), whichprovide peer connectivity through a dynamic logical over-lay, as well as efficient lookup through a hash table-likefunctionality and load balancing of the overall system’s
load. The open issues are the fair distribution of the sys-tem load among peers, and latency due to the number ofhops associated with every message over the DHT keyspace.
Multicast approach is used to disseminate update messagesto relevant peers (i.e., multicast group) in the virtual world.A multicast group is formed for each world region or entity(e.g. avatar, object) to inform group members about corre-sponding state changes. P2P-based NVE systems proposeboth IP multicast and application-layer multicast (ALM).However, IP multicast is not widely deployed due to groupand network management issues. In ALM, the managementof multicast groups is still challenging, due to a high num-ber of group members as well as highly dynamic players inthe virtual world.
Fully-connected approach connects all peers to each otherin the VE while using message filtering techniques to limitmessage transmissions. Players thus frequently send mes-sages to only a small group of players, while infrequently toall other players in the world. Fully-connected systems donot scale as each player needs to communicate with all otherplayers.
Neighbor-list exchange approach connects peers closest toeach other in the virtual world. Directly connected peersregularly inform each other about not only themselves butalso their own neighbors through neighbor-list exchange.However, global network connectivity is not guaranteedand constant neighbor-list exchanges may introduce a highmessage overhead.
Mutual notification approach connects nearby peers inthe VE usually based on well-known geometrical struc-tures (e.g., Voronoi diagram). Global network connectivityand neighbor discovery are realized only when necessarythrough mutual notification among direct neighbors. How-ever, the underlying overlay might induce high maintenancecost as it is subject to high connection change rate due to thecontinuous movement of players.
3 World partitioning approach
The partitioning of the virtual world into regions wasthe first used method to construct a P2P NVE system[26–29]. Each region is assigned to a manager peer whichis a peer with sufficient resources in terms of computingpower, bandwidth and storage capacity, responsible for themanagement of the region. A region groups a small numberof entities potentially in relation to each other (e.g., play-ers potentially interested in each other and objects in the
Peer-to-Peer Netw. Appl.
region). However, NVEs can be supported by the world par-titioning approach under the assumption that the resourcesof each manager are sufficient to manage its region.
The manager peer manages the data of objects of itsregion. The data of an avatar on the other hand can be keptby either the manager of the region inside which the avatarresides [30] or the peer controlling the avatar [24]. Themanager of a region determines the relations between peersbased on players’ overlapping AOIs. Regular peers residingin a region inform their manager about the events they create(e.g., position changes). The manager in return informs reg-ular peers about state changes due to events occurring in itsregion. Regular peers directly exchange messages to reducethe load on the manager [31], while neighboring managersexchange their regions information in order to provide acontinuous view to their regular peers [31, 32].
3.1 Different shaped partitioning
The virtual world can be partitioned into different shapedand sized regions. Rectangular [24, 33–36], tree-based [32,36–38], hexagonal [31, 39–42], and polygonal (e.g.,Voronoi) [30, 35, 43–45] regions are the most popularpartitioning forms.
MOPAR [31] partitions the virtual world into hexagonalcells (Fig. 1). Each hexagonal cell is assigned to a homenode through a DHT. A home node is used for assigningthe cell to a master node which is responsible for the cellmanagement (i.e., a DHT is used as a rendezvous pointfor master node registration). The first node entering intoa cell becomes the master node of the cell, otherwise itbecomes a slave node. The master of a cell keeps infor-mation about slave nodes residing inside the cell. Slavenodes inform their master about their position changes inthe VE, and are notified of new neighbors by their master.Slaves directly inform each other about their movements todecrease the load on their master node. Masters exchangetheir slave lists to be informed about the slaves resid-ing in the neighboring cells and thus to completely coverthe AOI of their slaves in order to provide them with a
Fig. 1 Hexagonal partitioning - Mopar [31]
continuous view of the virtual world. However, althoughslave list exchange is performed only between masters, itsfrequency is still a concern. When a master leaves its cell,one of the slaves residing in the cell becomes the new mas-ter, to which the cell data is transferred. The continuous datatransfer to newly assigned cell masters can create a signifi-cant overhead in the system. Besides, the hexagonal zoningdoes not allow the repartitioning of a cell into smallerregions when the cell master becomes overloaded due toslave crowd in the cell. Moreover, in case of crowding,excessive bandwidth requirements beyond slaves’ capabili-ties can occur due to direct connectivity between them.
3.2 Static and dynamic partitioning
Besides different shaped zoning, NVEs based on world par-titioning can follow two types of world division strategieswith different partitioning dynamics: static and dynamic. Instatic partitioning, the virtual world is partitioned into fixed-size regions (i.e., prepartitioning of the virtual world intoregions); thus, the regions do not change with the evolutionof the VE [24, 31, 41, 46, 47]. In dynamic partitioning, onthe other hand, regions can evolve over time due to appli-cation needs. A region can be repartitioned into smallerregions each of which is assigned to a new manager [34,36–38, 45, 48–50]. In the following, COVER [37] and Bad-umna [35] are presented as P2P NVE system examplesfollowing different types of world partitioning strategies.
COVER [37] proposes a dynamic world decomposi-tion through a global quadtree to partition the world byrecursively subdividing it into four quadrants (i.e., regions)depending on the number and position of players (Fig. 2).Each region is assigned to a super-node, the closest playerto the geometric center of the region. Each player is directlyconnected to all other players in its AOI. A player is calledas a covered player if its AOI is completely covered by itsneighbors’ AOI, and as an uncovered player if its AOI isnot completely covered. To provide awareness, each playersends its movement update messages to all its AOI neigh-bors, which in turn propagate the messages to their ownAOI neighbors. Players receiving a message can thus deter-mine their AOI neighborhood relationship with the playersending the message. A super-node is responsible for themanagement of the uncovered players in its region. Uncov-ered players inform not only their AOI neighbors but alsotheir super-node about their position changes in the VE. Asuper-node can thus provide awareness to uncovered play-ers located within its region. However, players’ movementcan produce frequent changes in the underlying quadtreestructure, making the management of the quadtree structuredifficult. When crowding occurs in a small area inside aregion, successive divisions are needed to be able to parti-tion the crowded area, which results in several nearly empty
Peer-to-Peer Netw. Appl.
Fig. 2 COVER [37]
regions. In addition, in case of crowding, direct connectiv-ity to all AOI neighbors can result in excessive bandwidthrequirements beyond players’ capabilities.
Badumna [35, 51] provides both static partitioningthrough fixed-size cells and dynamic partitioning throughdynamic bounded regions, which can change size and posi-tion as required for the management of a dense region(Fig. 3). Badumna is based on both world partitioningand DHT approaches. Both cells and bounded regions areassigned to peers through the underlying DHT. Entitiesquery the DHT to find out about other entities in their AOI.A bounded region containing several entities queries theDHT as a single entity (i.e., one query serving for all entitiesin the region), thus reducing the traffic on the DHT. In addi-tion, a gossip protocol is proposed for the management of ahighly dense area (i.e., a large number of players clusteredin a small area) in order not to overload the peer respon-sible of this area. In a highly dense area, players reducethe frequency of their interaction with their manager peer;instead, players rely on their neighbors to inform each otherabout other players in the region based on the gossip proto-col. However, players’ frequent position changes in the VEmake the management of bounded regions difficult. In addi-tion, the search for interest purposes through queries on theDHT introduces unacceptable latency due to the number ofhops required by the underlying DHT structure.
3.3 Discussion
Table 1 presents a list of the different NVE systems pre-sented in this section. The world partitioning approach iswidely used to construct P2P NVE systems in a simple man-ner, where the global world management is divided into themanagement of small regions containing entities possibly inrelation based on interest. However, the partitioning of thevirtual world, which provides a fair distribution of the VEload among managers, remains a challenging issue. Notethat a large number of players can congregate in a regionfor interest purposes and crowding can thus dynamicallyoccur in different regions due to players’ frequent positionchanges in the VE.
To address this issue, in static world partitioning-basedNVEs, communication is generally constructed based on asecondary mechanism in order to reduce the load on man-agers, such as message dissemination in a region throughload balancing trees in case of crowding (e.g., Distributedevent delivery [33]), application layer multicast based onthe underlying DHT structure (SimMud [24]), direct com-munication between regular peers (e.g., MOPAR [31]), etc.Dynamic partitioning is proposed as an alternative solu-tion, in which overloaded regions due to high density arerepartitioned into smaller regions managed by new man-ager peers. However, region reassignments and management
Fig. 3 Badumna [35]
Peer-to-Peer Netw. Appl.
Table 1 The presented world partitioning based NVE systems
NVE systems Features
MOPAR [31] Static polygonal
Player list exchange between region managers
Direct message exchange between regular
peers in a region
N-Trees [32] Dynamic tree-based
Interregional communication supported by
managers
Direct message exchange between regular
peers in a region
COVER [37] Dynamic tree-based
Management of the uncovered regular peers
by managers
Direct message exchange between regular peers
based on AOI
Badumna [35] Static rectangular and dynamic polygonal
DHT support
Latency due to querying the DHT
of dynamic entities can create a significant overhead onthe system. A dynamic partitioning approach capable ofproviding static assignment of entities to managers is thusthe most suitable world partitioning approach to the NVErequirements, which can also be supported by a secondarycommunication mechanism as in static world partitioning.
4 DHT approach
The use of distributed hash tables (DHTs) [52–54] is anothermethod commonly used to support P2P NVEs. DHTs pro-vide an efficient way of indexing data items by fairly bal-ancing the system load among all participating peers. DHTsystems implement a hash function that maps any given itemto a specific peer in the system. Items and peers are mappedto a key space, which is partitioned among all peers. Eachpeer is responsible of a specific key space partition. DHTsprovide dynamic maintenance of peers, efficient lookup andload balancing of both resources and query messages.
P2P NVE systems partitioning the virtual world intoregions can use a DHT to assign each region to a managerpeer [24, 31, 35, 36, 38, 44, 46, 49, 55]. On the other hand,Colyseus [56] and Loader [57] use a DHT for distributingentities, instead of regions, to peers.
To ensure a fair distribution for load balancing purposes,DHT systems use a uniform hash function that randomlymaps items and peers to a common key space [53, 54]. How-ever, hashing destroys both the physical and logical localityrelation among peers. Pastry [52] takes into account physi-cal network locality, which minimizes the distance that the
messages have to travel. However, peers residing inside thesame region in the virtual world are still placed far-apart inthe DHT key space. Thus, peers probably interacting witheach other are connected via multiple hops over the DHT,which may result in high routing latency. To address thisissue, NL-DHT [58], Loader [57], Walkad [38] take intoconsideration the logical locality (i.e., distance in the vir-tual world) by mapping nearby peers in the virtual world tonearby keys on the DHT space. In addition, Colyseus [56],OPeN [55], Walkad [38] propose range queries on the DHTto find the set of entities located in peers’ area of interest.
To support communication between peers inside aregion, SimMud [24], PastryMMOG [40], Loader [57] andPBT [36] use Scribe [59], the application layer multicastbuilt on the underlying DHT overlay, Pastry [52]. However,to avoid unnecessary network delay due to possible numberof hops on the DHT, [31, 32, 34–36, 46, 49, 56] use DHTsonly as a rendezvous point for peers to discover relevantentities, and propose a communication model independentfrom the DHT routing infrastructure (e.g., direct, multicast,etc.) for update message dissemination.
4.1 Representative systems
This section examines several P2P NVE systems based ona DHT, namely Colyseus [56], SimMud [24] and PBT [36].Colyseus assigns entities to peers, while SimMud and PBTassign regions. Update messages are disseminated to rele-vant peers through a publish-subscribe fashion in Colyseus,and through Scribe [59] in SimMud and PBT.
4.1.1 Colyseus
Colyseus [56] proposes a NVE system based on Mer-cury [60], which is used as both a traditional randomizedDHT and a range-queriable DHT to distribute objects topeers and to discover objects within the AOI of a peer. Eachobject has one primary copy assigned to a peer through theDHT, and may have several replicas on different peers in thesystem to enable quick local access to the object. The pri-mary owner of an object is responsible for the serializationand synchronization of the updates to the object performedon any peer in the system. Peers interested in an object sub-scribe to the object owner through the DHT. The ownerof an object periodically publishes the object data to thesubscribers.
The components of Colyseus nodes are presented inFig. 4a, while an example of publication and subscrip-tion in Mercury is given in Fig. 4b. Hx and Hy are therange-queriable DHT hubs corresponding to the X and Ycoordinates, respectively, of a two dimensional world of320 × 320 units. The publication is sent to both Hx and Hy ,where it is stored at peers b and e respectively. However, a
Peer-to-Peer Netw. Appl.
Fig. 4 Colyseus’ system design
(a) Colyseus nodes [56]
x <= 150
int
int
int
int x >= 50
y <= 250
y >= 150
[0,80)
[240,320)
a
c
d
b[80,160)
[160,240)
int x 100int y 200
(query)
(data−item)
[200, 320)
[100,200)
g
e
f
H
H
x
y[0,100)
(b) Publication and subscription in Mer-cury [60]
subscription is passed to exactly one of the hubs correspond-ing to the required attributes. The subscription thus entersHx at peer d and is processed at peers b and c.
Through interest prediction, a peer prefetches the replicaof the objects likely to be needed in the near future (e.g.,objects entering in the player AOI due to the player move-ment in the world), and subscribes to the owner of theseobjects through the DHT. However, a short-lived object(e.g., missile) might not be consistently prefetched. Toaddress this issue, Colyseus proposes a proactive replica-tion algorithm which attaches a short-lived object to a peerbefore the object is requested by the peer based on the factthat most short-lived objects originate at locations very closeto their creator; thus, peers interested in the creator willprobably be interested in the newly created objects.
Colyseus performs better scalability and load balancingwith a range-queriable DHT compared to a randomizedDHT. However, in a dynamic world based on a range-queriable DHT, the continuous movement of objects intro-duces a high message overhead due to the frequent DHTupdate requirements. In addition, since the neighborhoodrelations between peers are built independently from thelogical locality relations among them in the VE, peersresiding inside the same area in the VE are connected viamultiple hops over the DHT, which may result in highlatency. Finally, in a world with a large number of peers,looking up objects with O(log N) search hops may createunacceptable object discovery latency.
4.1.2 SimMud
SimMud [24] is a distributed NVE system following thestatic world partitioning, DHT and multicast approaches.SimMud’s architecture is based on Pastry [52] and its cor-responding application layer multicast, Scribe [59]. Thevirtual world is partitioned into fixed-size regions contain-ing players and objects (Fig. 5). Using Pastry, each regionis randomly mapped to a coordinator peer that manages allobjects in the region and serves as the root of the multicasttree to which all players residing in the region subscribe.
State changes occurring in a region are periodically multi-cast by the coordinator to all region members. Coordinatorsconnect to each other through Pastry to enable player migra-tion between regions. Due to disjoint region partitioning,SimMud does not provide players a continuous view ofthe virtual world. Since a region’s area may exceed play-ers’ AOI, players receive irrelevant messages beyond theirAOI. If many players crowd in a region, the coordinator canquickly become a bottleneck and suffers from performanceand robustness issues. In most cases, messages are deliveredin less than 6 hops on Pastry’s address space. However, dueto Scribe multicast routing, the number of hops might gobeyond 50 hops, resulting in a delay up to several seconds,since a multicast tree is not constructed based on the AOIconcept, and so may include peers not residing inside theregion.
4.1.3 PBT
PBT [36] is a P2P NVE system following both the static anddynamic world partitioning, DHT and multicast approaches.The whole world is first partitioned into static regions, eachof which is assigned to a region master through the under-lying DHT (Fig. 6). Players interested in a region subscribeto the region through its region master, to be informedabout state changes occurring inside the region. Update
Food
Player
Multicast
Multicast
Direct connection
Region 1
Region 2
Region 3
Fig. 5 SimMud [24]
Peer-to-Peer Netw. Appl.
Fig. 6 PBT’s systemarchitecture [36]
dissemination inside a region is maintained based onScribe [59]. In addition, in order to manage a static regionwith a high number of entities or subscribers, a Predic-tion Binary Tree (PBT) is built on top of the region, whichbecomes the root of the PBT. When necessary, a region isdynamically partitioned into subregions, which correspondto the leaves of the PBT managed by the region master.Each subregion is assigned to a subregion worker throughthe DHT. A new multicast group is constructed for eachsubregion. A subregion worker manages the entities insidethe subregion and multicasts update messages to subscriberpeers. Compared to SimMud [24], PBT additionally pro-vides continuous view of the virtual world and dynamicpartitioning of crowded regions. However, players’ frequentposition change makes it extremely difficult to efficientlymanage a PBT in order to provide an adequate managementof world regions. In addition, PBT is subject to messagetransmission latency, as SimMud, due to the underlyingDHT structure and application layer multicast.
4.2 Discussion
The DHT-based NVE systems presented in this section arelisted in Table 2. A DHT assigns items to peers in a simpleway; thus, VE items can efficiently be distributed amongpeers via a DHT. However, the relation between items andpeers can be destroyed by the hash function due to randommapping of items and peers over the DHT key space. On theother hand, DHT-based NVE systems addressing this issueby mapping items on the DHT key space based on the logi-cal locality can suffer from the high dynamism of the VE interms of position changes in the virtual world. In addition,frequent entity discovery and message dissemination based
on a DHT can cause unacceptable delay for NVE applica-tions. Therefore, a DHT can be used for static assignment ofentities to peers while taking into account the logical rela-tion in the virtual world; however message disseminationbased on an approach other than a DHT might produce moreefficient results in the context of NVEs.
5 Multicast approach
In a VE, when the state of an entity is modified due to anevent, related update messages are disseminated to play-ers interested in the entity [61]. Multicast is a method usedto send an update message to relevant players, namely, amulticast group. A world region or an entity is assigneda multicast group to inform group members about corre-sponding state changes. Players subscribe to the multicast
Table 2 A list of the representative DHT based NVE systems
NVE systems Features
Colyseus [56] Entities assigned to peers
No logical locality relations between peers
on the DHT
Both randomized and range-queriable DHT support
SimMud [24] Regions assigned to peers
Static world partitioning support
Multicast support
PBT [36] Regions assigned to peers
Both static and dynamic world partitioning support
Multicast support
Peer-to-Peer Netw. Appl.
group of regions or entities intersecting with their AOI toreceive relevant update messages. P2P-based NVE systemsexploit both IP multicast and application layer multicast(ALM) for update message dissemination. In the following,we examine different types of multicast schemes proposedby NVE systems.
5.1 IP multicast
Both region and entity-based IP multicasts are proposed byNVE systems to support message dissemination in VEs.In region-based IP multicast, the virtual world is parti-tioned into regions, each of which is assigned a multicastgroup through a multicast address (e.g., NPSNET [26],SPLINE [28], SCORE [62], DIVE [63]). When an eventtakes place inside a region, the related message is multi-cast to all players residing in the region. On the other hand,entity-based IP multicast focuses on the entity, instead of theregion, and assigns each entity (e.g., avatar) a group to mul-ticast its state information to other interested players (e.g.,VELVET [64], Massively Multiplayer Audio Communica-tion [65]).
5.2 Application layer multicast
Application layer multicast (ALM) has been proposed toprovide multicasting functionality as an application serviceinstead of a network service, based on overlay networks.In ALM, a multicast tree is constructed through unicastlinks between overlay network peers forming the multi-cast group [66]. NVE systems propose both physical andlogical locality aware ALM algorithms to support messagedissemination.
5.2.1 Physical locality aware ALM
ALM algorithms typically construct their multicast treebased on the characteristics of unicast links between peers(e.g., end-to-end latency) in order to minimize the additionalrouting overhead compared to IP multicast [67]. Physicallocality aware ALM thus realizes update message dissem-ination based on nearby peers in the physical network interms of latency. NVE systems based on physical localityaware ALM are detailed in the following.
Region-based physical locality aware ALM
This approach has been used in [24, 32, 33, 36, 39, 40,42, 68–70]. These systems follow the region-based multi-cast strategy and propose an ALM scheme to disseminateupdate messages to players inside a region in the virtualworld. SimMud [24] and PastryMMOG [40] are based onPastry [52] and use its corresponding ALM, Scribe [59].
The virtual world is partitioned into regions, each assigneda multicast group with a unique ID. Regions are distributedto peers through Pastry. The peer with an ID closest to aregion ID serves as the manager of the region and the rootof its multicast tree. The manager of a region manages theresiding entities and multicasts messages related to eventsoccurring in the region, to the players residing inside theregion.
Various systems [33, 39, 42, 68–70] have proposed adifferent region-based ALM algorithm, which aims to pro-vide an efficient group management, update transmissiondelay and load balancing. MM-VISA [42, 71, 72] orga-nizes and manages entities by partitioning the virtual worldinto hexagonal regions, each of which is assigned to acoordinator, responsible for multicast-based communica-tion between players inside a region. Coordinators are con-nected to each other to support interregional communicationdue to AOI overlaps of players with different neighborregions. To provide a better multicast group management,entities are clustered into classes according to their move-ment behavior in the virtual world (e.g., stationary, slow,normal and fast moving objects). Multicast trees in a regionare constructed based on one cluster; in other words, themembers of a multicast group belong to the same cluster ofthe region. While constructing a multicast tree, the cumu-lative end-to-end delay between a coordinator and peers istaken into consideration, which results in multicast treeswith limited height. When all tree paths in a region reach themaximum height limit, a new multicast channel is openedfor newly joined/arrived players to the region. Figure 7represents the ALM scheme and message propagation inMM-VISA architecture, where C stands for coordinator,and O-1, O-2, O-3, and O-4, for classes having differentbehaviors.
Entity-based physical locality aware ALM
In entity-based ALM, each entity’s update messages are dis-seminated based on ALM multicast to players interested inthe entity. P2P propagation [73] is a client/server systemusing P2P multicast for message dissemination to reducethe server load. P2P propagation follows the entity-basedmulticast strategy, while proposing its own ALM algorithm.Each entity is assigned a multicast group. The set of peersinterested in an entity is determined by the server. For eachmulticast group, the peer having the minimum delay to allother peers inside the group is selected as relay to forward amessage destined to the group, from the server to the groupmembers. The update message of an entity thus goes fromthe server through the relay of the multicast group to reachthe members (Fig. 8). However, when crowding occurs, arelay can be overloaded due to message delivery to all groupmembers.
Peer-to-Peer Netw. Appl.
Fig. 7 ALM scheme andmessage propagation inMM-VISA [42]
C
O-4
O-2O-1
O-3
C
O-4
O-2O-1
O-3
Local multicast messageInterested message
5.2.2 Logical locality aware ALM
Players’ interest in the virtual world can follow a differentorder than the peers’ layout on the latency-based multi-cast tree. The message transmission delay between nearbyplayers in the virtual world can be more significant thanthe delay between distant players, since distant playershave more tolerance for weak synchronization than nearbyplayers. Logical locality aware ALM is thus proposed toprioritize update message dissemination to players that arenear the event. NVE systems based on logical locality awareALM are detailed in the following.
Region-based logical locality aware ALM
To disseminate update messages first to the most interestedplayers (i.e., nearby players to the event), N-Trees [32] andPIMTRE [74] construct a multicast tree based on the inter-est relation between peers in the multicast group. A peerresiding in a region transmits update messages due to anevent, first to nearby peers residing in the region, then to dis-tant peers residing in the neighbor regions within the event’sscope. Therefore, nearby peers in the world can receivemessages faster.
N-Trees [32] divide the virtual world into nested rectan-gular regions based on a quadtree (i.e., 2-Tree) in order to
Server
Relay Node
Child Node
Fig. 8 P2P propagation [73]
organize players by their interest (Fig. 9). A node in an N-Tree is either a leaf node (i.e., a node with no children) or anode with 2N children. Each node in the N-tree is managedby a selected peer, called leader. Each leaf node in the N-Tree corresponds to a world region. Each successive levelbelow the root node in the N-tree represents a further sub-division of the virtual world by 2N . Players residing insidea region form a group and join the corresponding leaf node.When a player generates an event in a region, the player dis-seminates the event message to other players, including theleader, residing in the same leaf node corresponding to theregion. If the event scope exceeds the region boundaries, themessage is then propagated to the regions (i.e., nodes) inter-secting with the event scope through their leaders. Whenthe number of players in a region, namely the population ofa leaf, exceeds a given threshold, the leaf in the N-Tree issubdivided into N children and the players are placed intothe appropriate leaves. The subdivision of a crowded regioninto subregions decreases the bandwidth requirements ofa player since the number of players to which a playershould send its update messages is reduced. However, lead-ers managing the corresponding leaves and nodes may beoverloaded due to interregional message transmission. Inaddition, players’ movement can produce frequent changesin the underlying N-tree structure, making the managementof the N-tree structure difficult.
Fig. 9 N-Trees [32]
Peer-to-Peer Netw. Appl.
Entity-based logical locality aware ALM
Apolo [75], AOICast [76] and pSense [77] follow a similarapproach to N-Trees and PIMTRE, while using the entity-based multicast strategy. Players multicast their update mes-sages to other players inside their AOI based on the localityrelation between players in the virtual world.
Apolo [75] is the first P2P NVE system achieving mes-sage through AOI-based multicast trees. Apolo connectseach peer to its closest peer in each of the four quadrantsin the virtual world (Fig. 10). A peer p informs other peersinside its AOI about its state change through a multicast treebuilt on top of the overlay (Fig. 10b). The source peer gener-ating the event, namely p, becomes the root of the multicasttree. For a peer q in p’s AOI, the closest peer to p amongq’s closest quadrant neighbors becomes q’s tree parent. Theevent update message is first received by peers closest tothe source peer p, and then relayed to other farther peers.However, the connectivity of each peer to exactly four peerscan lead to multicast trees with long paths, thus increasingthe number of hops (i.e., latency) needed for sending updatemessages to AOI neighbors.
5.3 Discussion
A list of the multicast-based NVE systems given in this sec-tion are listed in Table 3. Multicast provides an efficientway of communication between peers by distributing mes-sage dissemination load among all relevant peers. However,entity-based multicast might create different trees with thesame multicast group members (e.g., multicast tree struc-tures for entities that are nearby in the virtual world), thusoverloading peers participating to multicast trees. On theother hand, in region-based multicast, the region size hasan important role in the efficiency of message delivery. Incase of large regions, since the region may exceed play-ers’ AOI, players receive irrelevant messages beyond theirAOI. With small regions, since players’ AOI may cover sev-eral regions, the same players become participants of severalmulticast trees, thus increasing computation overhead on
peers managing the multicast trees (i.e., tree roots), andplayers participating in the multicast.
IP multicast offers efficient message delivery inside agroup, however it is not widely deployed due to issuesrelated to group management, multicast address allocationand network management [78, 79]. On the other hand,in ALM, peers’ upload bandwidth is a critical resourcethat should be optimized. The depth of the ALM tree isincreased as group size increases, leading to long latencywhich may not be appropriate for NVE requirements [80].In addition, message dissemination may be delayed in ALMalgorithms that do not construct multicast trees based onthe AOI concept; for instance, Scribe [59] can create treeswith long paths which include peers transmitting event mes-sages while not residing in event scopes. As such, theproperties of both the underlying physical network as wellas the logical network built on top of the physical net-work should be considered to provide an efficient multicastalgorithm.
6 Fully-connected approach
In the fully-connected approach, all players in the virtualworld are directly connected to each other. Since everyoneis connected to every-one else, message filtering techniquesare used to limit message transmissions between certainpeers to make the system feasible. Players frequently sendmessages to only a small group of players, while infre-quently to all other players in the world. P2P NVE systemsbased on this approach focuses mainly on the managementof avatars where avatars’ data are stored on the correspond-ing user machines. A world object, if any, can be treated asan avatar and its data are thus managed on the machine ofthe user creating the object.
6.1 Representative systems
Two fully-connected P2P NVE systems, Frontier set [81]and Donnybrook [82] are examined in the following.
Fig. 10 Entity-based logicallocality aware ALM - Apolo [75]
(a) Topology (b) Multicast tree
Peer-to-Peer Netw. Appl.
Table 3 A list of the multicast-based NVE systems
Multicast NVE systems
IP multicast Region based NPSNET [26], SPLINE [28],
SCORE [62], DIVE [63]
Entity based VELVET [64], Massively multiplayer
audio communication [65]
ALM Physical locality aware Region based SimMud [24], Event delivery [33], AVDNet [39],
PastryMMOG [40], MM-VISA [42], Federated P2P [68],
Update dissemination [69]
Entity based P2P propagation [73]
Logical locality aware Region based N-Trees [32], PIMTRE [74]
Entity based Loader [57], Apolo [75], AOICast [76], pSense [77]
Frontier set
Frontier set [81, 83] is inspired from update-free region(UFR) in the geometric message filtering protocols [84, 85]and potentially visible set (PVS) [86]. The UFR concept isused to determine the message exchange protocol betweenplayers, while the PVS concept is used to group playersinterested in each other and thus frequently exchangingmessages.
In the geometric message filtering protocols [84, 85],each player of any pair of players defines its respectiveupdate-free region (UFR) inside which the player does notinform the other about its position change. When a playerleaves its UFR, the pair of players become relevant to eachother. From that time on, update messages are thus mutuallyand regularly transmitted between the two players. Whenthey become irrelevant again, new UFRs are computed. Theplayers do not send any update message until they becomerelevant again.
Potentially visible set (PVS) [86] is based on the fact thatfrom most viewpoints in the world, only a small portion isvisible due to occlusion by opaque surfaces (e.g., walls).The world is partitioned into rectangular cells, the bound-aries of which coincide with opaque surfaces (Fig. 11a).Non-opaque portals are identified on cell boundaries, and
used to form an adjacency graph connecting the world cells.Each cell can thus determine other visible cells from thiscell, based on existing unblocked sightlines between cells.In Fig. 11b, cells B, C, and E are visible to cell A since thereexists a sightline from A to each of them.
Frontier set [81, 83] defines frontier to determine the vis-ibility between cell pairs based on the PVS concept andthe message exchange frequency between players resid-ing in different cells based on UFR concept. Given twocells A and B, a frontier consists of two cell sets, FAB
and FBA, such that no cell in FAB is visible to a cell inFBA and vice-versa. There exists no frontier between twocells visible to each other. The players residing inside themutually visible cells exchange messages. No message isexchanged between players residing inside the cells invis-ible to each other. Therefore, players residing inside cellswithout any frontiers are visible to each other and thus fre-quently exchange messages until they become invisible dueto their movement. Players invisible to each other do notexchange messages until they cross any frontier and thusbecome visible again. Fig. 12 shows the frontier of thecell pairs corresponding to players’ positions in the world.No message is exchanged between players in Figs. 12a, b,and d. However, players exchange messages while residinginside visible cells (Fig. 12c).
Fig. 11 Potential visibility set
(a) Portals and adjacency graph [86]
A
G H I
(b) Visible cells from cell A [83]
D E F
B C
Peer-to-Peer Netw. Appl.
(a) (b) (c) (d)
Fig. 12 Frontier sets [83]. Points represent players. a FAI ={A,B,C, F }, FIA = {G,H, I}. b FEH = {A,B,C,E,F }, FHE ={H, I}. c No frontier set since cell D can see cell H. d FDI =
{A,B,C,D,E,F }, FID = {I}. No message exchange betweenplayers in a, b, and d. Mutual message exchange in c
Although only players visible to each other exchangemessages, computing requirements can exceed peers’ capa-bilities in a VE with a high number of players frequentlychanging position, since each player should check the fron-tier set with each other player in the VE, and calculate newfrontier sets if necessary.
Donnybrook
Donnybrook [82] is a fully-connected NVE system basedon the fact that a person can only focus on a constant num-ber of entities at any moment [87, 88]. Donnybrook thuslimits the number of players in which a player is most inter-ested. Players frequently send their state update messagesto the players which are interested in them. Besides, playerssend infrequent state information to other players (i.e., play-ers with less interest in them). A player p is thus frequentlyinformed about the state of players in which p is inter-ested, and infrequently informed about the state of playersin which p is less/not interested. To extrapolate the behaviorof a remote player q, from which p receives infrequent infor-mation, an intelligent bot runs on p’s local machine. Thisbot approximates q’s behavior through q’s infrequently sentinformation. In case of a difference between q’s approxi-mated and actual state, the bot approaches smoothly to theactual state. Since p is not interested in q, p is unlikely tonotice the difference.
6.2 Discussion
The presented NVE systems based on the fully-connectedapproach are given in Table 4. In the fully-connectedapproach, all players in the virtual world are directly con-nected to each other. The fully-connected approach thusdoes not scale well with the number of players in the VEdue to bandwidth and computing overhead on each peer.In addition, if a large number of peers are interested in(and thus want to directly interact with) each other, thenecessary message exchanges can quickly exceed peers’capabilities.
7 Neighbor-list exchange approach
In P2P systems, using neighbors of neighbors knowledgeis a well-known strategy to acquire required informationand to make a better routing decision [89]. In neighbor-listexchange-based P2P NVEs, peers exchange their knowl-edge of the virtual world in order to be better informed ofthe changes occurring in their surroundings. P2P NVE sys-tems following this approach only consider avatars, the dataof which are managed on corresponding user machines.
7.1 Representative systems
This section examines two P2P NVE systems based on theneighbor-list exchange approach, P2P message exchangescheme [90] and Message interchange protocol [91].
7.1.1 P2P message exchange scheme
P2P message exchange[90] is a fully-distributed system inwhich the VE is supported through regular neighbor-listexchange between peers. Each peer maintains a fixed num-ber of direct connections to the closest neighbors in thevirtual world. A peer p’s direct neighbors are called activeentities (AE). p is regularly informed by its AE about notonly themselves but also their own neighbors. The neigh-bors of p’s AE are called latent entities (LE). p regularlyupdates its AE list based on the information received fromits AE. Figure 13 shows the neighborhood relationship and
Table 4 A list of the representative fully-connected NVE systems
NVE systems Features
Frontier sets [83] Based on update-free region [84] and
potentially visible set [86]
Bottleneck on highly visible players
Donnybrook [82] Based on the fact of the limited number
of focus entities [87, 88]
Bottleneck on popular players
Peer-to-Peer Netw. Appl.
Fig. 13 P2P message exchange scheme [90]
message flow for a peer O, in which O’s active entities areA1, A2, A3, A4, and latent entities are B1, B2, . . . ,B8.
In a world with uncrowded regions, connections to onlya fixed number of nearest peers can divide the networkinto isolated subgroups. [92] is an extension of [90] thatproposes a probabilistic solution to the network partition-ing problem. To ensure global network connectivity throughinformation exchange between isolated subgroups, peersconnect to randomly selected peers at arbitrary time inter-vals through a bootstrap server and exchange informationwith those peers.
To provide an efficient solution to the network parti-tioning problem even in the presence of a high numberof entities, [93] replaces the random message exchangescheme proposed in [92] with a DHT-based scheme toreorganize isolated subgroups through message exchangebetween them. The virtual world is partitioned into regions,each of which is assigned to a manager peer through a DHTbuilt separately from the P2P message exchange protocol.The manager peer of a region keeps information about peersinside its region. Peers thus inform the manager of regionsinside which they reside about their position changes. Whena peer p belonging to an isolated subgroup inquires the man-ager of a region, the manager provides p with the list ofpeers residing inside the region. Then, p asks all peers in thelist their current location in the world and updates its AE listbased on distance. The network is thus reorganized and sub-groups are kept connected. However, this solution requiresthe management of a DHT structure based on peers’ posi-tions in the virtual world in addition to the P2P messageexchange protocol, and thus increases the load on peers.Furthermore, as in exchanged neighbor-list information, thepeer list information provided by the manager of a region toan inquiring peer can contain redundant information, espe-cially in case of player crowding in the region, which cancreate excessive message exchanges.
7.1.2 Message interchange protocol
Message interchange protocol [91] connects each peer to atleast four peers, the closest in each of the four quadrants inthe virtual world (Fig. 14) in a similar way to Apolo [75](cf. Section 5.2.2). However, in Apolo, each peer connectsto exactly four peers (i.e., the closest peers in each of thefour quadrants), which can introduce additional hops to senda message to an AOI neighbor, thus increasing latency.
In Message interchange protocol, a peer p keeps infor-mation about other peers in three tables: Observable Table(OT), Global Table (GT), and Group Leader Table (GLT).The OT contains information about peers interested in p andto which p regularly sends its update messages. The GTcontains information about other peers residing in p’s sur-roundings, however not directly interested in p; this tableis used for accurate AOI neighbor discovery. Peers inter-change their GT within a time period depending on thedistance between them in the virtual world. Nearby peersinterchange their GT more frequently than faraway peers.Besides, the GLT keeps track of the closest peers in each ofthe four quadrants to ensure global connectivity. Peers peri-odically updates their GLT and peers having empty OT andGT interchange their GLT to keep the network connected.Message interchange protocol is, therefore, basically basedon the exchange of peers’ OT, GT and GLT with their neigh-bors, while the OT, GT and GLT are used for state updatedissemination, AOI neighbor discovery and global networkconnectivity, respectively.
Compared to P2P message exchange scheme, Messageinterchange protocol provides better AOI and global net-work connectivity managements. However, similarly to P2Pmessage exchange, exchanged lists may contain redundantinformation. Player crowding causes a high number of peers(with long lists) exchanging their list information with eachother, the bandwidth requirements of which may exceedpeers’ capabilities.
Fig. 14 Message interchange protocol [91]. e connects to at least theclosest peers in each of the four quadrants
Peer-to-Peer Netw. Appl.
7.2 Discussion
Table 5 shows the neighbor-list exchange based NVE sys-tems presented in this section. In the neighbor-list exchangeapproach, each peer has limited number of connectionsindependently from the total number of users in the VE.The direct connectivity between peers in relation minimizesthe delay of message propagation in the VE. However, aspeer connectivity supported by only neighbor-list exchangeis not sufficient to ensure global network connectivity,an auxiliary mechanism is thus needed to avoid networkpartitioning. In addition, constant neighbor-list exchanges,which may contain redundant information for destinationpeers, introduce a high message overhead.
8 Mutual notification approach
In P2P NVE systems based on the mutual notificationapproach, peers maintain direct connections to nearby peerssurrounding them in the VE, usually based on well-knowngeometrical structures such as Voronoi in [94], Yao graphin [77] or quadtree in [95]. In the event of peers’ positionchange in the VE, global network connectivity and neigh-bor discovery are achieved through mutual cooperation andnotification among direct neighbors.
8.1 Representative systems
This section examines several P2P NVE systems basedon the mutual notification approach, namely, pSense [77],Solipsis‘03 [96], VON [94], Relaxed Triangulation [97] andRed-Black Delaunay [98].
8.1.1 pSense
pSense [77] provides an AOI-based peer connectivity.Each peer keeps information about other peers in its AOI(Fig. 15). To ensure global network connectivity and neigh-bor discovery, each peer also connects to sensor peers, the
Table 5 The presented neighbor-list exchange based NVE systems
NVE systems Features
P2P message exchange [90] Global network connectivity
through DHT
Irrelevant and redundant data
transmission
Message interchange Global network connectivity through
protocol [91] closest quadrant peers
Irrelevant and redundant data
transmission
closest peers outside its AOI in each of the six sectors ofequal angle, similarly to Yao graphs [99]. When a peer pmoves in the world, p informs its AOI and sensor neighborsabout its position change. If a sensor peer detects a peer qentering in p’s AOI, it informs p about q which becomes p’snew AOI neighbor. If the total number of neighbors exceedsp’s upload bandwidth capacity, p sends its update messageto only a randomly selected subset of its neighbors. Theneighbors receiving p’s update message then forward themessage to their own neighbors residing in p’s AOI. How-ever, in case of player crowding, many duplicates can beproduced during message forwarding. Also, peers receivinga message may not have enough bandwidth to forward themessage to all their neighbors, leading to a second roundof randomly selected (instead of all) neighbors, and thusincreasing the delay with which some peers will receivemessages.
8.1.2 Solipsis ’03
Solipsis‘03 [96, 101] is a P2P NVE system that proposesa new connectivity structure based on the angular rela-tion between peers’ positions in the virtual world. Thevirtual world is designed as a two dimensional torus toprovide an unbounded surface. Each peer maintains directconnections with the peers residing in its AOI. Neighbordiscovery is achieved through the notification of neighbors.The neighbors of a peer should always form a convex hull(Fig. 16), which guarantees global network connectivity.If a peer detects a convex hull violation through recursivecollaboration with its direct neighbors, the peer establishesnew connections until the convex hull condition is satisfied(Fig. 16). In the event of position change, new neighborsare discovered through mutual cooperation and notifica-tion among direct neighbors. When a peer wants to moveto a specific position in the virtual world upon login orby teleportation, the peer’s request is transmitted from therequesting peer to the peer closest to the specified position(Fig. 16).
However, Solipsis‘03 might lead to inconsistencieswhere some neighboring peers go undetected [100].Figure 17 presents such a situation where e and ef are notconnected although ef is residing inside e’s AOI.
8.1.3 VON
VON [94, 102] proposes a distributed NVE system similarto Solipsis‘03 [96]. VON mainly replaces the geometricalstructure providing peer connectivity in Solipsis‘03 with aVoronoi diagram [103] to resolve the undetected neighborproblem in Solipsis. The virtual world is partitioned usingVoronoi based on peers’ positions in the virtual world. Eachpeer maintains a Voronoi diagram of its AOI neighbors,
Peer-to-Peer Netw. Appl.
Fig. 15 pSense [77]
N
N
N
N
N
N
N
N
S
S
S
S
S
S
N
S
Local Node
Near Node
Sensor Node
Unknown Node
S
S
Vision Range
(a) A peer’s view
S
S
S
S
S
S
S
Uneligible Node
S Sensor Node
Local Node
SVision Range
(b) Sensor node selection
and keeps a direct connection to all of them. Figure 18presents the Voronoi-based partitioning of the virtual worldaccording to the participants’ position and the neighborhoodrelations between them. Each peer keeps connections withits enclosing neighbors whose Voronoi regions have a com-mon edge with its own Voronoi region (even if they resideoutside its AOI). The outermost peers inside the AOI of apeer are called boundary neighbors, and are responsible forneighbor discovery. VON thus optimizes the mutual notifi-cation between peers compared to Solipsis‘03 by reducingthe number of peers participating in the neighbor discoveryfrom all AOI neighbors to only boundary neighbors. VONprovides limited number of connections between peersindependently from the total number of users in the VE,and guarantees global network connectivity since peers arealways connected to their enclosing neighbors even thoughthese neighbors reside outside peers’ AOI.
In the event of player crowding, however, mechanismsthat connect a peer to all its AOI neighbors might leadto situations where almost all peers become connected toeach other due to overlapping AOIs. To deal with this issue,
AOICast [76], built on VON, proposes to construct a span-ning tree over all AOI neighbors of a peer, thus reducingbandwidth usage at the cost of a higher latency.
VON however considers peer movements under theassumption that when a peer moves in the virtual world, allother peers surrounding the moving peer remain stationary.This assumption is needed so that the surrounding peers canbe used as reliable landmarks to update the underlying over-lay structure according to the peer’s movement. In addition,the layout of peers in the VE might overload some peersby forcing them to make direct connections beyond theircapacity. For instance, when in the VE a high number ofpeers are lying on a circle inside the center of which a peerp resides, then the peers on the circle become p’s enclos-ing neighbors and p should manage the direct connectionsto all its enclosing neighbors which may exceed p’s band-width capabilities [104] (Fig. 19). Also, in a highly dynamicenvironment, particularly in the context of P2P NVEs, theunderlying Voronoi structure produces high maintenancecost as it is subject to high connection change rate due tocontinuous players’ movement. As players tend to move in
Fig. 16 Solipsis‘03 [96]
(a) Convex hull (b) Convex hull construction
me
en
me’
e’n
e
e
0
1e
(c) Entrance of e
Peer-to-Peer Netw. Appl.
e
e
Δ1
2Δf
Fig. 17 Solipsis‘03 - Undetected neighbor [100]
the VE, the network topology should continuously adapt toplayers’ movements such that the overlay structure is keptvalid. Maintenance cost can become particularly high whencrowding occurs, and a highly dynamic interaction with thesurrounding takes place.
8.1.4 Relaxed triangulation
To deal with high maintenance cost related to the under-lying overlay in the mutual notification approach, RelaxedTriangulation (RT) [97] proposes a new triangulation algo-rithm to connect peers based on their positions in the virtualworld. Relaxed Triangulation (RT) for a set of vertices Vin a 2-D plane is defined as a triangulation T(V) such thatno vertex of V is inside any triangle in T(V). In order tominimize the maintenance cost resulting from high connec-tion change rate due to user movements, RT maximizes theflip-free area of a node, namely the area inside which thenode’s movement does not trigger any edge flip operation.To this end, RT defines the flip-free area of a node as theregion composed of all the triangles of which the node isa vertex in RT. A flip operation thus occurs only when thenode passes across the base line of one of its triangles. The
Fig. 18 Neighbors in VON [94]
Fig. 19 VON - Peer connectivity in the worst case [104]
gray area in Figs. 20a and b shows point 0’s flip-free area inDelaunay and in RT, respectively. RT provides overlay net-work connectivity to support P2P NVEs while decreasingmaintenance overhead by reducing the number of connec-tion changes due to users’ movement. However, similar toVON, RT considers peer movements under the assumptionthat when a peer moves in the virtual world, all other peerssurrounding the moving peer remain stationary.
8.1.5 Red-Black Delaunay
Red-Black Delaunay [98] proposes a distributed NVE sys-tem based on a Delaunay graph [105] of peers’ positions inthe virtual world. Red-Black Delaunay constructs the sameoverlay structure as VON [94]. Red-Black Delaunay focuseson the construction of a valid Delaunay graph consideringseveral concurrent peers’ movements and clearly defines thegroups of moving peers and of the stationary peers used toupdate the overlay structure. Peers in the virtual world arerandomly grouped into two sets, called red and black. Foreach set, a separate Delaunay subgraph is constructed basedon the position of its peers in the virtual word (Fig. 21).A peer is directly connected not only to its same coloredDelaunay neighbors, but also to a group of different col-ored peers, called parents, which are responsible of updatingthe neighbor connections of the peer during its movement.A peer p in a subgraph becomes the parent of a peer q
(a) Delaunay Triangulation (b) Relaxed Triangulation
Fig. 20 Node 0’s flip-free area [97]
Peer-to-Peer Netw. Appl.
Fig. 21 Red-Black Delaunay [98]
in the other subgraph, if p’s Voronoi region in p’s corre-sponding subgraph intersects with q’s Voronoi region in q’scorresponding subgraph. If the peers in a subgraph movein the virtual world in times T , 2T , 3T , . . ., then the peersin the other subgraph move in times T/2, 3T/2, 5T/2, . . ..Therefore, alternately, when the peers in a subgraph changeposition in the virtual world, the peers in the other subgraphremain stationary and update the subgraph of the movingpeers by advising them about connection changes in theirsubgraph. The overall Delaunay graph including all peers inthe virtual word is then reconstructed based on the red andblack subgraphs.
Red-Black Delaunay constructs a valid Delaunay struc-ture that deals with peer movements based on the sequen-tial synchronized movement of peers in a subgraph (withrespect to peers in the other subgraph) and update ofthe moving subgraph by the stationary peers in the othersubgraph. The feasibility of the proposed scheme strictlydepends on the sequential movement and update phasesof each subgraph. In addition, in the event of crowdingof the same-colored peers, the parent peers are likely tobecome overloaded due to the update requirements of theirsubgraph.
8.2 Discussion
The mutual notification-based NVE systems presented inthis section are given in Table 6. Mutual notification canbe considered as a more specific approach of neighbor-list exchange. In the neighbor-list exchange approach,peers exchange their knowledge of the VE through regu-lar neighbor-list information exchanges regardless of thenecessity of this information on the receiver side. On theother hand, in mutual notification, peers cooperate with eachother on a specific task (e.g., neighbor discovery, globalnetwork connectivity) only when necessary. Since neighbor-list information might contain irrelevant and redundant data,mutual notification provides a more bandwidth efficientsolution. High scalability is achieved since peers maintaina limited number of connections with other peers based onthe underlying geometrical structure independently from thetotal number of peers in the VE. The direct connectivity
Table 6 The representative mutual notification based NVE systems
NVE systems Features
pSense [77] Based on Yao graph
Latency in message transmission
in case of crowding
Solipsis‘03 [96] Based on the angular relation between
peers’ positions
Undetected neighbor problem
VON [94] Based on Voronoi diagram
High overlay maintenance cost
One peer movement at a time
Relaxed Triangulation [97] Based on triangulation
Dealing with the maintenance cost
due to dynamism
Maximizing the flip-free area of a peer
One peer movement at a time
Red-Black Delaunay [98] Based on Delaunay
Focus on several concurrent peers’
movements
Based on sequential synchronized
movement of peer groups
High overlay maintenance cost
between peers in relation minimizes the delay of messagepropagation in the VE. However, the underlying structuremight cause high maintenance cost as it is subject to highconnection change rate due to the continuous movementof players. The management of several concurrent peers’movements remains an open issue. Moreover, the layout ofpeers in the VE might overload some peers by forcing themto make direct connections beyond their capacity.
9 Discussion and future perspectives
Table 7 summarizes the different approaches used to con-struct some of the P2P NVE systems presented in this paper,while Table 8 summarizes the characteristics of the differ-ent approaches discussed in this paper for the constructionof P2P NVE systems. Overall, the world partitioning andfully-connected approaches may suffer from bandwidthconsumption issues, while the DHT approach may intro-duce high latency, and the neighbor-list exchange approachdoes not guarantee global network connectivity. The mutualnotification approach supported by the multicast approachfor message diffusion and the world partitioning approachfor the management of less dynamic entities (in terms ofposition changes) can provide global network connectiv-ity, low latency due to direct connectivity between peersin relation and distributed bandwidth load among peers.
Peer-to-Peer Netw. Appl.
Table 7 P2P NVE systems’ characteristics
NVE systems World par. DHT Multicast Fully-connec. Neigh.-list exchan. Mutual notifi.
SimMud ‘04 [24] X X X
Event delivery ‘05 [33] X X
AVDNet ‘05 [39] X X
Federated P2P ‘04 [68] X X
Zoned federation ‘04 [46] X X
Badumna ‘10 [35] X X X
PBT ‘09 [36] X X X
N-Trees ‘05 [32] X X X
PIMTRE ‘08 [74] X X
COVER ‘06 [37] X
MOPAR ‘05 [31] X X
PastryMmog ‘06 [40] X X X
MM-VISA ‘09 [42] X X
VSM ‘08 [30] X
Solipsis ‘08 [44] X X
Colyseus ‘06 [56] X
Loader ‘08 [57] X X
Update dissemination ‘05 [69] X X
Apolo ‘05 [75] X
Frontier sets ‘06 [81] X
Donnybrook ‘08 [82] X
P2P message exch. ‘02 [90] X
Message intercha. protocol ‘05 [91] X
AOICast ‘08 [76] X X
pSense ‘08 [77] X X
Solipsis ‘03 [96] X
VON ‘06 [94] X
Relaxed triangulation ‘09 [97] X
QuON ‘09 [95] X
However, given the high dynamism of VEs in terms ofposition changes, rigid overlay structures (typified by theVoronoi diagram) introduce a high overlay maintenancecost.
From the above, and despite their several advantages,P2P designs for NVEs still suffer from several weak-nesses that are hindering their adoption by the NVE market.However, we believe that the main reason behind the non-adoption of P2P technologies for the support of NVEs isthe difficulty in providing a high level of availability andtrust. In an open system where nodes join and leave at will,dealing with failures, disconnections, and churn requiresa great degree of redundancy (and its associated consis-tency maintenance cost) to increase service availability.The trust issue is even more challenging in open envi-ronments. In client-server architectures, every user trustsat least the server. However, such trust may not exist ina P2P setting, making security issues very difficult to
solve without centralized control. Although several workshave been lately conducted to provide solutions capableof establishing a higher level of availability and trust inP2P NVEs [106–110], these efforts are still at their earlystages.
However, beyond the very specific characteristics, advan-tages, and drawbacks of P2P NVE approaches, it is prudentto take a step back and observe from a global perspective theplace that a decade of intensive scientific research on thistopic can have in today’s technological landscape. Givencurrent skyrocketing interest in cloud computing as a meansof dynamic provisioning of (virtually infinite) resources,one might wonder whether the P2P technology still has arole to play in today’s computing setting. In other words, thefollowing important question arises: Is there one distributedcomputing paradigm that will define future NVE architec-tures, or will there be a place for multiple paradigms,including P2P?
Peer-to-Peer Netw. Appl.
Tabl
e8
P2P
NV
Eap
proa
ches
’ch
arac
teri
stic
s
App
roac
hes
Adv
anta
ges
Dra
wba
cks
Inte
rest
ing
for
Wor
ldpa
rtit
ioni
ng•G
roup
ing
enti
ties
inte
rest
edin
each
othe
r•E
ffic
ient
size
dpa
rtit
ioni
ngis
sue
•Sta
tic
assi
gnm
ento
fen
titi
esto
man
ager
s
into
regi
ons
•Log
ical
loca
lity
pres
erve
dbe
twee
nen
titi
es•L
oad
bala
ncin
gon
man
ager
sth
roug
hau
xili
ary
mec
hani
sms
•Man
agem
ent
issu
eof
dyna
mic
enti
ties
DH
T•S
impl
eit
em-p
eer
assi
gnm
ent
•Hig
hla
tenc
ydu
eto
hops
onth
eD
HT
key
spac
e•S
tati
cas
sign
men
tof
enti
ties
tom
anag
ers
if
logi
call
ocal
ity
ispr
eser
ved
•Fai
rlo
addi
stri
buti
on•L
ogic
alre
lati
ons
betw
een
enti
ties
may
bede
stro
yed
•Sim
ple
rend
ez-v
ous
poin
tman
agem
ent
Mul
tica
st•F
air
dist
ribu
tion
ofm
essa
gedi
ffus
ion
load
•Man
agem
ent
issu
eof
dyna
mic
grou
pm
embe
rs•M
essa
gedi
ssem
inat
ion
inca
seof
crow
ding
•Hig
hla
tenc
ydu
eto
hops
betw
een
grou
pm
embe
rs
Full
y-co
nnec
ted
•Low
late
ncy
due
todi
rect
conn
ectiv
ity
betw
een
peer
s•O
verh
ead
due
toco
nnec
tivit
yto
allp
eers
•Man
agem
ent
ofpl
ayer
sin
tere
sted
inth
een
tire
VE
•Man
agem
ent
issu
eof
popu
lar
play
ers
Nei
ghbo
r-li
stex
chan
ge•L
imit
ednu
mbe
rof
conn
ecti
ons
betw
een
peer
s•G
loba
lnet
wor
kco
nnec
tivit
yth
roug
h
auxi
liar
ym
echa
nism
s
•Low
late
ncy
due
todi
rect
conn
ectiv
ity
•Hig
hov
erhe
addu
eto
irre
leva
ntan
d
betw
een
peer
sin
rela
tion
redu
ndan
tda
tatr
ansm
issi
on
Mut
ualn
otif
icat
ion
•Lim
ited
num
ber
ofco
nnec
tion
sbe
twee
npe
ers
•Man
agem
ent
issu
eof
conc
urre
nt•A
vata
rm
anag
emen
t
peer
mov
emen
ts
•Low
late
ncy
due
todi
rect
conn
ectiv
ity
•Hig
hov
erla
ym
aint
enan
ceco
st
betw
een
peer
sin
rela
tion
•Ban
dwid
thef
fici
entn
eigh
bor
disc
over
y
(com
pare
dto
neig
hbor
-lis
texc
hang
e)
Peer-to-Peer Netw. Appl.
We strongly believe that no one distributed computingapproach will be able to meet the requirements of NVEapplications. Just like its client-server and P2P counterparts,cloud computing presents several limitations in the contextof NVEs with strong real-time interactivity requirements.Currently, two major cloud gaming models are emerging,with each having its advantages and drawbacks. In the firstmodel, the entire game software is hosted in the cloud,where heavy processing and graphics rendering are exe-cuted and sent back as video streams to users accessing thegame through thin clients [111, 112]. This gaming modelopens NVEs to a much wider user population by makingthe capabilities of users’ machines unimportant. However,streaming high quality interactive video to clients, espe-cially at a massive scale, is confronted with bandwidthlimitations and cost as major bottlenecks. Also, user qualityof experience relies solely on Internet connection quality,including upload/download speeds, connection stability, aswell as latency. Also, traditional delay compensation tech-niques, such as dead reckoning, used on the client side areno longer possible when the complete game software is runremotely in the cloud. This is a critical issue in an Inter-net environment where latencies are not predictable. Thesecond cloud gaming model avoids the above-mentionedissues by hosting in the cloud only the game server softwarerunning the game logic and handling players’ interactions,while keeping graphics rendering on clients’ machines.This, however, makes high-end computers or particularhardware necessary to process the client software for thegame.
Beyond the advantages of one over the other, both cloudgaming models face the following major challenge: howto best exploit the elastic character of cloud computing toanswer the strong scalability requirements of NVEs.
Current solutions deal with the scalability issue throughrigid parallelization techniques of players’ interactions. Themain technique is sharding, which involves creating sev-eral completely separate instances, or shards, of the virtualworld, and assigning each player to a single shard basedon some criteria, such as players’ geographical regions orskill levels. However, this goes against the idea of a “sin-gle unified virtual world” that all users connect to, with thepotential of interacting with millions of other users world-wide instead of being limited to users of the same worldinstance.
In light of the above discussion, we strongly believethat alternative parallelization techniques of players’ inter-actions are required to best exploit the elastic capabili-ties that cloud infrastructures can offer. In particular, thegame server software should be engineered in a way thatallows the server to elastically stretch and shrink its useof cloud resources. In this respect, P2P technologies areextremely relevant in their extraordinary capacities for
self-organization and self-healing in dynamic environmentswhere peers (and their associated resources) continuouslyjoin and leave the network. We note in particular that:
– The world partitioning approach has a huge potentialfor implementation in data-centers. Many of the algo-rithms that have been described in Section 3 couldsignificantly enhance the rigid sharding process whichis implemented today. Flexible area management andload-balancing among area managers are among themost attractive algorithms.
– The DHT-based approach has high potential given thatits main drawback in the context of P2P systems –highlatency– is no longer an issue in data-centers. Similarlyto Facebook’s Cassandra system that is based on a cus-tomized DHT, we envision promising research activitiesin the customization of the previous works described inSection 4 in the context of data-centers.
– The mutual notification and neighbor-list exchangeapproaches are in our opinion less attractive. Though,an architecture where each avatar runs on a virtualmachine in a data-center makes sense. Most of the pre-vious works described in Section 8 focus on hypothesesthat are not accurate in data-centers, namely the absenceof central authority and peer churn, and are thus notdirectly applicable to data-centers.
Another argument in favor of a solution based on bothcloud and P2P computing relates to the game streamingmodel (i.e., the first cloud gaming model discussed above),where latency and bandwidth consumption are the key bot-tlenecks. Given that P2P solutions have shown to be veryeffective for content streaming, we believe that an inter-esting solution would be to augment the cloud resourceswith a P2P streaming solution where the streaming loadcan be offloaded to the P2P network, thus leveraging peers’bandwidth and reducing latency.
Finally, we would like to emphasize that the libertar-ian motivations for building authority-less NVEs is stillviable. Even if this libertarian motivation is less attractivefrom an economical perspective, we believe that there is aneed to perform research with this objective in mind. Therecent proliferation of projects related to peer-to-peer socialnetworks (for example [113]), and the fear that current dom-inant web giants are provoking, emphasizes the critical needfor research efforts on this area.
10 Conclusion
This paper analyzes the key design approaches to con-struct a P2P NVE system. We discuss each approach, itsadvantages and drawbacks, and examine several representa-tive P2P NVE systems based on the presented approaches.
Peer-to-Peer Netw. Appl.
Despite all the nice characteristics of the P2P approach,we believe that a solution that provides all the necessaryrequirements of NVE systems cannot be achieved follow-ing a pure P2P design choice. However, we argue that futureNVE architectures will build on the strengths of severaldistributed computing paradigms to overcome the shortcom-ings of adopting only one. In particular, a very promisingresearch direction would be to look into hybrid solutionswhere the self-organizing and self-healing nature of P2Papproaches can be exploited on top of cloud resources tobest exploit the elastic capabilities of cloud computing.
References
1. Doom. http://en.wikipedia.org/wiki/Doom (video game)2. Quake Live. http://www.quakelive.com/#!home3. Marathon. http://en.wikipedia.org/wiki/Marathon (video game)4. Blake Stone: aliens of gold. http://en.wikipedia.org/wiki/Blake
Stone: Aliens of Gold5. Star wars: dark forces. http://en.wikipedia.org/wiki/Star Wars:
Dark Forces6. Homeworld. http://en.wikipedia.org/wiki/Homeworld7. Starcraft II: wings of liberty. http://us.battle.net/sc2/en/8. Company of heroes. http://www.companyofheroes.com/9. Ultima online. http://www.uo.com/ageofshadows/viscent.html
10. EverQuest. http://everquest.station.sony.com/11. Final fantasy XI. http://www.playonline.com/ff11eu/index.shtml12. EVE online. http://www.eveonline.com/13. World of warcraft. http://us.battle.net/wow/en/14. Calvin J, Dickens A, Gaines B, Metzger P, Miller D, Owen
D (1993) The simnet virtual world architecture. In: Annualinternational symposium virtual reality IEEE, pp 450–455
15. Pullen JM, Wood DC (1995) Networking technology and dis.IEEE 83:1156–1167
16. Institute of Electrical and Electronic Engineers (2000) IEEE stan-dard for modeling and simulation (m&s) high level architecture(hla)—framework and rules. IEEE Std. 1516-2000, pp i–22
17. FALCON 4.0. http://en.wikipedia.org/wiki/Falcon 4.018. America’s army. http://www.americasarmy.com/19. Second life official site. http://secondlife.com/20. HiPiHi. http://www.hipihi.com/21. The sims online. http://thesims.com/en us/home22. There. http://www.there.com/23. Active worlds. http://www.activeworlds.com/24. Knutsson B, Lu H, Xu W, Hopkins B (2004) Peer-to-peer support
for massively multiplayer games. In: Proceeding the 23rd con-ference of the IEEE communications society, INFOCOM ’04, p107
25. Beigbeder T, Coughlan R, Lusher C, Plunkett J, Agu E, ClaypoolM (2004) The effects of loss and latency on user performance inunreal tournament 2003®. In: Proceeding 3rd ACM SIGCOMMworkshop on network and system support for games, NetGames’04. ACM, New York, pp 144–151
26. Macedonia MR, Zyda MJ, Pratt DR, Brutzman DP, Barham PT(1995) Exploiting reality with multicast groups. IEEE ComputGraph Appl 15(5):38–45
27. Funkhouser TA (1995) RING: a client-server system for multi-user virtual environments. In: Proceeding the 1995 symposiumon interactive 3D graphics, SI3D ’95. ACM, New York, pp 85–ff
28. Barrus JW, Waters RC, Anderson DB (1996) Locales: supportinglarge multiuser virtual environments. IEEE Comput Graph Appl16(6):50–57
29. Abrams H, Watsen K, Zyda M (1998) Three-tiered interest man-agement for large-scale virtual environments. In: Proceeding theACM symposium on virtual reality software and technology,VRST ’98. ACM, New York, pp 125–129
30. Hu S-Y, Chang S-C, Jiang J-R (2008) Voronoi state manage-ment for peer-to-peer massively multiplayer online games. In:5th IEEE consumer communications and networking conference,CCNC ’08, pp 1134–1138
31. Yu AP, Vuong ST (2005) MOPAR: a mobile peer-to-peer overlayarchitecture for interest management of massively multiplayeronline games. In: Proceeding the international workshop on net-work and operating systems support for digital audio and video,NOSSDAV ’05. ACM, New York, pp 99–104
32. GauthierDickey C, Lo V, Zappala D (2005) Using N-Trees forscalable event ordering in peer-to-peer games. In: Proceeding theinternational workshop on network and operating systems sup-port for digital audio and video, NOSSDAV ’05. ACM, NewYork, pp 87–92
33. Yamamoto S, Murata Y, Yasumoto K, Ito M (2005) A distributedevent delivery method with load balancing for MMORPG. In:Proceeding 4th ACM SIGCOMM workshop on network and sys-tem support for games, NetGames ’05. ACM, New York, pp 1–8
34. Rieche S, Wehrle K, Fouquet M, Niedermayer H, Petrak L, CarleG (2007) Peer-to-peer-based infrastructure support for massivelymultiplayer online games. In: 4th annual IEEE consumer com-munications and networking conference, CCNC ’07
35. Kulkarni S, Douglas S, Churchill D (2010) Badumna: a decen-tralised network engine for virtual environments. Comput Netw54(12):1953–1967. P2P technologies for emerging wide-areacollaborative services and applications
36. De Chiara R, Erra U, Scarano V (2009) Some considerationson the design of a P2P infrastructure for massive simulations.In: International conference on ultra modern telecommunication,ICUMT ’09
37. Morillo P, Moncho W, Orduna JM, Duato J (2006) Providingfull awareness to distributed virtual environments based on peer-to-peer architectures. In: Proceeding the 24th computer graph-ics international conference, lecture notes in computer science,CGI ’06, vol 4035. Springer, Berlin, pp 116–127
38. Varvello M, Diot C, Biersack E (2009) A walkable Kademlianetwork for virtual worlds. In: International workshop on peer-to-peer systems, IPTPS ’09
39. Correia RCM, Pellegrino SRM (2005) Scalable distributed vir-tual environments using multicast application level. In: Proceed-ing the ninth international conference on computer supportedcooperative work in design, vol 1, pp 140–145
40. Hampel T, Bopp T, Hinn R (2006) A peer-to-peer architecturefor massive multiplayer online games. In: Proceeding 5th ACMSIGCOMM workshop on network and system support for games,NetGames ’06. ACM, New York, pp 48
41. Chen A, Muntz RR (2006) Peer clustering: a hybrid approach todistributed virtual environments. In: Proceeding 5th ACM SIG-COMM workshop on network and system support for games,NetGames ’06. ACM, New York, p 11
42. Ahmed DT, Shirmohammadi S, Oliveira JC (2009) A hybrid p2pcommunications architecture for zonal mmogs. Multimed ToolsAppl 45(1–3):313–345
43. Tumbde A, Venugopalan S (2004) A voronoi partitioningapproach to support massively multiplayer online games. Tech-nical Report CS 740 Project. University of Wisconsin, Madison
44. Frey D, Royan J, Piegay R, Kermarrec A-M, AnceaumeE, Le Fessant F (2008) Solipsis: a decentralized architec-ture for virtual environments. In: Proceeding the 1st interna-tional workshop on massively multiuser virtual environments,MMVE ’08
Peer-to-Peer Netw. Appl.
45. Denault A, Canas C, Kienzle J, Kemme B (2011) Triangle-basedobstacle-aware load balancing for massively multiplayer games.In: Proceedings of the 10th annual workshop on network and sys-tems support for games, NetGames ’11. IEEE Press, Piscataway,pp 4:1–4:6
46. Iimura T, Hazeyama H, Kadobayashi Y (2004) Zoned federationof game servers: a peer-to-peer approach to scalable multi-playeronline games. In: Proceeding ACM SIGCOMM workshop onnetwork and system support for games, NetGames ’04. ACM,New York, pp 116–120
47. Chan L, Yong J, Bai J, Leong B, Tan R (2007) Hydra: amassively-multiplayer peer-to-peer architecture for the gamedeveloper. In: Proceeding the 6th ACM SIGCOMM workshopon network and system support for games, NetGames ’07. ACM,New York, pp 37–42
48. Kim KC, Yeom I, Lee J (2004) HYMS: a hybrid MMOG serverarchitecture. IEICE Trans Inf Syst E87-D(12):2706–2713
49. Fan L, Taylor H, Trinder P (2007) Mediator: a design frame-work for P2P MMOGs. In: Proceeding the 6th ACM SIGCOMMworkshop on network and system support for games, NetGames’07. ACM, New York, pp 43–48
50. Botev J, Hohfeld A, Schloss H, Scholtes I, Sturm P, Esch M(2008) The hyperVerse—concepts for a federated and torrent-based ‘3D Web’. Int J Adv Media Commun 2(4):331–350
51. Kulkarni S (2009) Badumna network suite: a decentralized net-work engine for massively multiplayer online applications. In:9th international conference on peer-to-peer computing, IEEEP2P ’09
52. Rowstron A, Druschel P (2001) Pastry: scalable, distributedobject location and routing for large-scale peer-to-peer systems.In: Proceeding IFIP/ACM international conference on distributedsystems platforms. Middleware, pp 329–350:329–350
53. Ratnasamy S, Francis P, Handley M, Karp R, Schenker S (2001)A scalable content-addressable network. In: Proceeding the 2001conference on applications, technologies, architectures, and pro-tocols for computer communications, SIGCOMM ’01. ACM,New York, pp 161–172
54. Stoica I, Morris R, Karger D, Kaashoek MF, Balakrishnan H(2001) Chord: a scalable peer-to-peer lookup service for internetapplications. In: Proceeding the 2001 conference on applica-tions, technologies, architectures, and protocols for computercommunications, SIGCOMM ’01. ACM, New York, pp 149–160
55. Tanin E, Harwood A, Samet H, Nayar D, Nutanong S (2006)Building and querying a P2P virtual world. Geoinformatica10(1):91–116
56. Bharambe A, Pang J, Seshan S (2006) Colyseus: a distributedarchitecture for online multiplayer games. In: Proceedings the3rd conference on networked systems design & implementation,NSDI ’06. USENIX Association, Berkeley, pp 12–12
57. Hariri B, Shirmohammadi S, Pakravan MR (2008) LOADER:a location-aware distributed virtual environment architecture.In: IEEE conference on virtual environments, human-computerinterfaces and measurement systems, VECIMS ’08, pp 97–101
58. Ratti S, Hariri B, Shirmohammadi S (2008) NL-DHT: a non-uniform locality sensitive DHT architecture for massively multi-user virtual environment applications. In: Proceeding the 200814th IEEE international conference on parallel and distributedsystems, ICPADS ’08. IEEE Computer Society, Washington,pp 793–798
59. Castro M, Druschel P, Kermarrec AM, Rowstron AIT (2002)SCRIBE: a large-scale and decentralized application-level mul-ticast infrastructure. IEEE J Sel Areas Commun 20(8):1489–1499
60. Bharambe AR, Agrawal M, Seshan S (2004) Mercury: support-ing scalable multi-attribute range queries. SIGCOMM ComputCommun Rev 34(4):353–366
61. Delaney D, Ward T, McLoone S (2006) On consistency and net-work latency in distributed interactive applications: a survey–partI. Presence: Teleoper Virtual Environ 15(2):218–234
62. Lety E, Turletti T, Baccelli F (2004) SCORE: a scalablecommunication protocol for large-scale virtual environments.IEEE/ACM Trans Networking 12(2):247–260
63. Frecon E (2004) DIVE: communication architecture and pro-gramming model. IEEE Commun Mag 42(4):34–40
64. de Oliveira JC, Georganas ND (2002) VELVET: an adaptivehybrid architecture for very large virtual environments. In: IEEEinternational conference on communications, ICC ’02, vol 4,pp 2491–2495
65. Nguyen CD, Safaei F, Boustead P (2004) Performance evalua-tion of a proxy system for providing immersive audio communi-cation to massively multi-player games. In: IEEE global telecom-munications conference workshops, GlobeCom workshops 2004,pp 192–199
66. Chu Y-h, Rao SG, Seshan S, Zhang H (2002) A case for endsystem multicast. IEEE J Sel Areas Commun 20(8):1456–1471
67. Vogel J, Widmer J, Farin D, Mauve M, Effelsberg W (2003)Priority-based distribution trees for application-level multicast.In: Proceeding the 2nd workshop on network and system supportfor games, NetGames ‘03. ACM, New York, pp 148–157
68. Rooney S, Bauer D, Deydier R (2004) A federated peer-to-peernetwork game architecture. IEEE Commun Mag 42(5):114–122
69. Chen F, Kalogeraki V (2005) Adaptive real-time update dis-semination in distributed virtual simulation environments. In:Proceeding the eighth IEEE international symposium on object-oriented real-time distributed computing, ISORC ‘05. IEEEComputer Society, Washington, pp 233–236
70. Singhal SK (2007) Scalable networked virtual environmentsusing unstructured overlays. In: Proceeding the 13th internationalconference on parallel and distributed systems, ICPADS ’07.IEEE Computer Society, Washington, pp 1–8
71. Ahmed DT, Shirmohammadi S, de Oliveira JC (2006) A novelmethod for supporting massively multi-user virtual environ-ments. In: IEEE international workshop on haptic audio visualenvironments and their applications, HAVE ’06, pp 72–77
72. Ahmed DT, Shirmohammadi S (2010) A fault tolerance proce-dure for p2p online games. In: 10th international conference,information sciences signal processing and their applications(ISSPA), 2010, pp 614–617
73. Ito S, Saito H, Sogawa H, Tobe Y (2006) A propagation of virtualspace information using a peer-to-peer architecture for massivelymultiplayer online games. In: 26th IEEE international conferenceon distributed computing systems workshops, ICDCS workshops2006, p 44
74. Xiang-bin S, Yue W, Qiang L, Ling D, Fang L (2008) An inter-est management mechanism based on N-Tree. In: Proceeding the2008 9th ACIS international conference on software engineer-ing, artificial intelligence, networking, and parallel/distributedcomputing, SNPD ’08. IEEE Computer Society, Washington,pp 917–922
75. Lee J, Lee H, Ihm S, Gim T, Junehwa S (2005) APOLO:ad-hoc peer-to-peer overlay network for massively multi-playeronline games. Technical report cs/tr-2005-248. Korea AdvancedInstitute Of Science And Technology, (KAIST)
76. Jiang J-R, Huang Y-L, Hu S-Y (2008) Scalable AOI-cast forpeer-to-peer networked virtual environments. In: Proceeding the28th international conference on distributed computing systemsworkshops, ICDCSW ’08. IEEE Computer Society, Washington,pp 447–452
77. Schmieg A, Stieler M, Jeckel S, Kabus P, Kemme B, BuchmannA (2008) pSense - maintaining a dynamic localized peer-to-peerstructure for position based multicast in games. In: Proceeding
Peer-to-Peer Netw. Appl.
the 2008 eighth international conference on peer-to-peer comput-ing, P2P ’08. IEEE Computer Society, Washington, pp 247–256
78. Diot C, Levine BN, Lyles B, Kassem H, Balensiefen D (2000)Deployment issues for the IP multicast service and architecture.IEEE Netw 14(1):78–88
79. Hosseini M, Ahmed DT, Shirmohammadi S, Georganas ND(2007) A survey of application-layer multicast protocols. IEEECommun Surv Tutor 9(3):58–74
80. Lao L, Cui J-H, Gerla M, Chen S (2007) A scalable overlaymulticast architecture for large-scale applications. IEEE TransParallel Distrib Syst 18:449–459
81. Steed A, Angus C (2006) Enabling scalability by partitioning vir-tual environments using frontier sets. Presence: Teleoper VirtualEnviron 15(1):77–92
82. Bharambe A, Douceur JR, Lorch JR, Moscibroda T, Pang J,Seshan S, Zhuang X (2008) Donnybrook: enabling large-scale,high-speed,peer-to-peer games. In: Proceeding the ACM SIG-COMM 2008 conference on data communication, SIGCOMM’08. ACM, New York, pp 389–400
83. Steed A, Angus C (2005) Supporting scalable peer to peer vir-tual environments using frontier sets. In: Proceeding the 2005IEEE conference 2005 on virtual reality, VR ’05. IEEE ComputerSociety, pp 27–34
84. Makbily Y, Gotsman C, Bar-Yehuda R (1999) Geometric algo-rithms for message filtering in decentralized virtual environ-ments. In: Proceeding the 1999 symposium on Interactive 3Dgraphics, I3D ’99. ACM, New York, pp 39–46
85. Goldin A, Gotsman C (2004) Geometric message-filtering proto-cols for distributed multiagent environments. Presence: TeleoperVirtual Environ. 13(3):279–295
86. Teller SJ, Sequin CH (1991) Visibility preprocessing for interac-tive walkthroughs. SIGGRAPH Comput Graph 25(4):61–70
87. Robson JG, Graham N (1981) Probability summation andregional variation in contrast sensitivity across the visual field.Vis Res 21(3):409–418
88. Cowan N (2001) The magical number 4 in short-term memory:a reconsideration of mental storage capacity. Behav Brain Sci24(01):87–114
89. Manku GS, Naor M, Wieder U (2004) Know thy neighbor’sneighbor: the power of lookahead in randomized P2P networks.In: Proceeding the thirty-sixth annual ACM symposium on the-ory of computing, STOC ’04. ACM, New York, pp 54–63
90. Kawahara Y, Morikawa H, Aoyama T (2002) A peer-to-peermessage exchange scheme for large scale networked virtualenvironments. In: Proceeding IEEE international conference oncommunication systems, ICCS ’02. IEEE Computer Society,Washington, pp 957–961
91. Chen J-F, Lin W-C, Bai H-S, Dai S-Y (2005) A message inter-change protocol based on routing information protocol in avirtual world. In: Proceeding the 19th international conference onadvanced information networking and applications, AINA ’05.IEEE Computer Society, Washington, pp 377–384
92. Kawahara Y., Aoyama T, Morikawa H (2004) A peer-to-peermessage exchange scheme for large-scale networked virtualenvironments. Telecommun Syst 25(3–4):353–370
93. Matsumoto N, Kawahara Y, Morikawa H, Aoyama T (2004) Ascalable and low delay communication scheme for networkedvirtual environments. In: Proceeding IEEE international work-shop on networking issues in multimedia entertainment, NIME’04, pp 529–535
94. Hu S-Y, Chen J-F, Chen T-H (2006) VON: a scalable peer-to-peer network for virtual environments. IEEE Netw 20(4):22–31
95. Backhaus H, Krause S (2009) QuON—a quad-tree based over-lay protocol for distributed virtual worlds. In: Proceeding the 2ndinternational workshop on massively multiuser virtual environ-ments, MMVE ’09.
96. Keller J, Simon G (2003) Solipsis: a massively multi-participantvirtual world. In: Proceeding the international conference onparallel and distributed processing techniques and applications,PDPTA ’03, vol 1. CSREA Press, pp 262–268
97. Buyukkaya E, Abdallah M (2009) Efficient triangulation for p2pnetworked virtual environments. Multimedia Tools Appl 45:291–312
98. Ghaffari M, Hariri B, Shirmohammadi S (2009) A delaunay tri-angulation architecture supporting churn and user mobility inMMVEs. In: Proceeding the 18th international workshop on net-work and operating systems support for digital audio and video,NOSSDAV ‘09. ACM, New York, pp 61–66
99. Yao AC-C (1982) On constructing minimum spanning trees ink-dimensional spaces and related problems. SIAM J Comput11(4):721–736
100. Simon G (2004) Conception et realisation d’un systeme pourenvironnement virtuel massivement partage. PhD thesis, Univer-site de Rennes 1
101. Keller J, Simon G (2002) Toward a peer-to-peer shared vir-tual reality. In: Proceeding the 22nd international conference ondistributed computing systems, ICDCSW ’02. IEEE ComputerSociety, Washington, pp 695–700
102. Hu S-Y, Liao G-M (2004) Scalable peer-to-peer networkedvirtual environment. In: Proceeding 3rd ACM SIGCOMM work-shop on network and system support for games, NetGames ‘04.ACM, New York, pp 129–133
103. Aurenhammer F (1991) Voronoi diagrams–a survey of a funda-mental geometric data structure. ACM Comput Surv 23(3):345–405
104. Shun-Yun Hu (2005) Scalable peer-to-peer networked virtualenvironment. Master’s thesis, Tamkang University
105. de Berg M, van Kreveld M, Overmars M, Schwarzkopf O (1997)Computational geometry: algorithms and applications. Springer-Verlag
106. GauthierDickey C, Zappala D, Lo V, Marr J (2004) Low latencyand cheat-proof event ordering for peer-to-peer games. In: Pro-ceeding of the 14th international workshop on Network and oper-ating systems support for digital audio and video, NOSSDAV’04. ACM, New York, pp 134–139
107. Chan M-C, Hu S-Y, Jiang J-R (2008) An efficient and secureevent signature (eases) protocol for peer-to-peer massively mul-tiplayer online games. Comput Netw 52:1838–1845
108. Liu H-I, Lo Y-T (2008) Dacap—a distributed anti-cheating peerto peer architecture for massive multiplayer on-line role playinggame. In: 8th IEEE international symposium, cluster computingand the grid CCGRID ’08, pp 584–589
109. Lin Y, Kemme B, Patino-Martinez M, Jimenez-Peris R (2006)Applying database replication to multi-player online games. In:Proceeding of 5th ACM SIGCOMM workshop on Network andsystem support for games, NetGames ’06., ACM, New York
110. Griwodz C (2002) State replication for multiplayer games.In: Proceedings of the 1st workshop on network and systemsupport for games, NetGames ’02. ACM, New York, pp 29–35
111. OnLive. http://www.onlive.com/112. Gaika. http://www.gaikai.com/113. PeerSon. http://www.peerson.net/
Peer-to-Peer Netw. Appl.
Eliya Buyukkaya receivedthe B.Sc. degree in ComputerEngineering from Middle EastTechnical University, Turkey,in 2003 and the M.Sc. andPh.D. degrees in ComputerScience from University Paris6, France, in 2006 and 2011,respectively. She worked as apostdoctoral research fellow inthe Computer Science depart-ment at Telecom Bretagne fortwo years. Since September2012, she is research and
teaching fellow in the Computer Science department at ENSSAT, Uni-versity Rennes 1. Her research interests are P2P systems, distributedsystems, graph databases, social networks and virtual environments.
Maha Abdallah is currentlyAssociate Professor in thedepartment of Computer Sci-ence (LIP6) at Pierre andMarie Curie University (Paris,France). She received an MScdegree in Computer Sciencefrom Pierre and Marie CurieUniversity in 1996, and a PhD(Honors) degree in ComputerScience from the Universityof Versailles in 2001. Herresearch interests are in thegeneral area of distributed and
cloud computing, covering distributed algorithms and systems, dis-tributed databases, fault-tolerant systems, and scalable systems andnetworks, as well as the application of these technologies to interactive3D media applications.
Gwendal Simon received hisMaster Degree in ComputerScience in 2000 and his PhDdegree in Computer Sciencein December 2004 from Uni-versity of Rennes 1 (France).During his PhD, he worked atOrange R&D Labs, where hedeveloped a distributed sharedvirtual world, which hasreceived a worldwide atten-tion, including innovationprize, and has turned out to bea start-up. From 2004 to 2006he was a researcher atOrange Labs, where he
worked on spontaneous networks and social media innova-tions. Since 2006, he has been Associate Professor at TelecomBretagne, a graduate engineering school within the InstitutMines-Telecom. He has been a visiting researcher at Univer-sity of Waterloo from September 2011 to September 2012. Hisresearch interests include large-scale distributed networks, peer-to-peer and ad-hoc networks, optimization problems and videodelivery systems.