Embed Size (px)
Citation preview
1
Scalable Peer-to-Peer Networked Virtual Environment
Master Thesis Oral Examination
Dept. of CSIE, Tamkang Univ.
Advisor: Dr. Chen Jui-Fa
Shun-Yun Hu
2005/01/07
2
Outline
Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion
3
What is Networked Virtual Environment (NVE)?
Virtual Reality + Internet
3D environment with people (avatar), objects, terrain, agents
Military simulations (’80) Massively Multiplayer Online Games (mid-‘90)
Trends: larger scale, more realistic simulation
4
5
NVE: A Shared Space
6
Issues for Creating NVE
Consistency (events/states)
Responsiveness multiplayer Security
Scalability Persistency massively multiplayer Reliability (Fault-tolerance)
7
The Scalability Problem
Many nodes on a 2D plane ( > 1,000) Message exchange with those within Area of Interest (AOI) How does each node receive the relevant messages?
Area of Interest
8
A simple solution (point-to-point)
N * (N-1) connections ≈ O(N2) Not scalable!
Source: [Funkhouser95]
9
A better solution (client-server)
Message filtering at server to reduce trafficN connections = O(N) server is bottleneck
Source: [Funkhouser95]
10
Current solution (server-cluster)
Still limited by servers. Expansive to deploy & maintain.
Source: [Funkhouser95]
11
Scalability Analysis
Scalability constrains Computing resource (CPU) Network resource (Bandwidth)
Non-scalable system vs. Scalable system
x: number of entities
y: resource consumption at the limiting system component
Resource limit
12
What Next?
Strategies Increase resource More servers Decrease consumption Message filtering
Architectures Scale Point-to-point (LAN) tens 10^1 Client-server hundreds 10^2 Server-cluster thousands 10^3 ? millions 10^6 …
Peer-to-Peer
13
What is Peer-to-Peer (P2P)?
[Stoica et al. 2003] Distributed systems without any centralized control
or hierarchical organization Runs software with equivalent functionality
Examples File-sharing: Napster, Gnutella, eDonkey Distributed computing: SETI@Home (UC Berkeley) VoIP: Skype
14
Peer-to-Peer Overlay
A P2P overlay network source: [Keller & Simon 2003]
15
Promise & Challenge of P2P
Promises Growing resource, decentralized
Scalable Commodity hardware Affordable
Challenges Topology maintenance dynamic join/leave Efficient content retrieval no global knowledge
16
Issues for Creating P2P NVE
Consistency (events/states)
Responsiveness multiplayer Security
Scalability Persistency massively multiplayer Reliability (Fault-tolerance)
Consistency (topology) P2P NVE
17
Related Works (1): SimMUD
[Knutsson et al. 2004] (Univ. of Pennsylvania)
Pastry + Scribe Regions Coordinators
(super-nodes)
Fixed-size region Relay overhead
18
Related Works (2)
[Kawahara et al. 2004] (Univ. of Tokyo)
Fully-distributed Nearest-neighbors List exchange
High transmission Overlay partition
19
Related Works (3): Solipsis
[Keller & Simon 2003] (France Telecomm R&D)
Links with AOI neighbor Mutual cooperation Inside convex hull
Potentially slow discovery Inconsistent topology
20
Outline
Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion
21
Design Goals
Observation: for virtual environment applications, the contents we want
are messages from AOI neighbors Content discovery is a neighbor discovery problem
Solve the Neighbor Discovery Problem in a fully-distributed, message-efficient manner.
Specific goals: Scalable Limit & minimize message traffics Responsive Direct connection with AOI neighbors
22
Voronoi Diagram
2D Plane partitioned into regions by sites, each region contains all the points closest to its site
Can be used to find k-nearest neighbor easily
Neighbors
Site
Region
23
Design Concepts
Identify enclosing and boundary neighbors Each node constructs a Voronoi of its neighbors Enclosing neighbors are minimally maintained Mutual collaboration in neighbor discovery
Circle Area of Interest (AOI)
White self
Yellow enclosing neighbor (E.N.)
L. Blue boundary neighbor (B.N.)
Pink E.N. & B.N.
Green AOI neighbor
D. Blue unknown neighbor
Use Voronoi to solve the neighbor discovery problem
24
Procedure (JOIN)
1) Joining node sends coordinates to any existing node
Join request is forwarded to acceptor
2) Acceptor sends back its own neighbor list
joining node connects with other nodes on the list
Acceptor’s region
Joining node
25
Procedure (MOVE)
1) Positions sent to all neighbors, mark messages to B.N.
B.N. checks for overlaps between mover’s AOI and its E.N.
2) Connect to new nodes upon notification by B.N.
Disconnect any non-overlapped neighbor
Boundary neighbors
New neighbors
Non-overlapped neighbors
26
Procedure (LEAVE)
1) Simply disconnect
2) Others then update their Voronoi
new B.N. is discovered via existing B.N.
Leaving node (also a B.N.)
New boundary neighbor
27
Dynamic AOI
Crowding within AOI can overload a particular node
It’s better if AOI-radius can be adjusted in real time
28
Adjustment Conditions
AOI-radius decrease Number of connections > maximum allowable connections
AOI-radius increase Maximum connections not exceeded Current AOI-radius < preferred AOI-radius
Delay counter To avoid fluctuations
29
Demonstration
Simulation video General movements (20 nodes, 800x600 world) Local vs. global view Dynamic AOI adjustment
31
Outline
Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion
32
Simulation Method C++ implementation of Voronoi-based algorithm
World size: 1000 x 1000, AOI: 150 Trials from 10 – 250 nodes Connection limit per node: 10 1000 time-steps
(~ 100 simulated seconds, assuming 10 updates/seconds)
Behavior model Random movement: random direction Constant velocity: 5 units/step Movement duration: random (1 – 25 steps)
33
Consistency Metrics
Topology Consistency [Kawahara, 2004]
Number of observed AOI neighbors
Number of actual AOI neighbors
Drift Distance [Diot, 1999]Distance between observed position and actual position
(average over all nodes)
34
Basic Model
Topology Consistency
Topology Consistency Measurements
90
9192
9394
95
9697
9899
100
0 50 100 150 200 250
Number of Nodes
To
po
log
y C
on
sis
ten
cy
(%
)
35
Basic Model
Scalability (1)
Transmission Size Per Node Per Second
0.0
1.0
2.0
3.0
4.0
5.0
6.0
0 25 50 75 100 125 150 175 200 225 250
Number of Nodes
Siz
e (k
b)
send (max)
send (avg)
recv (max)
recv (avg)
36
Basic Model
Scalability (2)
Average Neighbor Size Measurements
0
2
4
6
8
10
12
14
16
18
0 50 100 150 200 250
Number of Nodes
Nei
gh
bo
r S
ize
connected
AOI
37
Dynamic AOI Model
Effect of Node Increase on Average AOI-radius
0
20
40
60
80
100
120
140
160
0 50 100 150 200 250
Number of Nodes
Av
g. A
OI R
ad
ius
38
Dynamic AOI
Scalability (1)
Average Neighbor Size Measurements
0
1
2
3
4
5
6
7
8
9
10
0 50 100 150 200 250
Number of Nodes
Nei
gh
bo
r S
ize
connected
AOI
39
Dynamic AOI
Scalability (2)
Transmission Size Per Node Per Second
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0 25 50 75 100 125 150 175 200 225 250
Number of Nodes
Siz
e (
kb
)
send (max)
send (avg)
recv (max)
recv (avg)
40
Dynamic AOI
Scalability (3)
Comparison of Voronoi-based P2P and Client-Server
0
20
40
60
80
100
120
140
160
180
0 25 50 75 100 125 150 175 200 225 250
Number of Nodes
Siz
e (
kb
)
send (avg)
recv (avg)
CS-send (avg)
CS-recv (avg)
41
Dynamic AOI
Topology Consistency (1)
Topology Consistency Measurements
90
91
92
93
94
95
96
97
98
99
100
10 30 50 70 90 110 130 150 170 190 210 230 250
Number of Nodes
To
po
log
y C
on
sist
ency
(%
)
42
Dynamic AOI
Topology Consistency (2)
Drift Distance Measurements
0
20
40
60
80
100
0 50 100 150 200 250
Number of Nodes
Dri
ft D
ista
nc
e
max
avg
43
Dynamic AOI
Reliability (1)
Change in Consistency of over 1000 time-steps (step 150 - 250)
70
80
90
100
150 160 170 180 190 200 210 220 230 240 250
time-step
To
po
log
y C
on
sist
ency
(%
)
per step
node 1
node 91
44
Dynamic AOI
Reliability (2)
Average Number of Steps to Recover from Inconsistency
0
1
2
3
4
0 50 100 150 200 250
Number of Nodes
Ste
ps
45
Outline
Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion
46
Analysis of Design
Consistency (Topology) Topology is fully connected & consistent enclosing neighbors
Responsiveness Lowest latency direct connection, no relay
Scalability Resource-growing & decentralized resource consumption
Reliability Self-organizing for small number of node failures
47
P2P NVE Comparisons
SimMUD Neighbor-list exchange
Solipsis VON
Consistency (topology)
Supernode Neighbor list-exchange (partitioning)
Neighbor notify&query (undiscovery)
Neighbor notify (consistent)
Responsive-ness
High overhead
High overhead
Medium overhead
Low overhead
Scalability Relied on supernode
Fully-distributed
Fully-distributed
Fully-distributed
Reliability Long up-time
N/A N/A Self-organizing
48
Problems of Voronoi Approach
Message traffic Circular round-up of nodes Redundant message sending
(inherent to fully-distributed design)
Incomplete neighbor discovery Can happen with inconsistent / incorrect neighbor list Fast moving node
49
Outline
Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion
50
Conclusion
NVE scalability is achievable with P2P architecture and is a neighbor discovery problem
A promising solution: Voronoi-based P2P Overlay Leverage knowledge of each peer to maintain topology
Properties Scalable: fully-distributed, dynamic AOI Efficient: low irrelevant messages, zero relay Robust: consistent and self-organizing topology
51
Potential Applications
Online gamesRelieve server from position updates in current MMOGs
MilitaryEnable large-scale, affordable military training simulation
3D WebProvide multi-user interactivity to static 3D world
Scientific simulationsDistribute spatial simulation requiring frequent synchronization
52
Future Works
Short-term Reliability measurements latency, packet loss, node fail Distributed event/state consistency Recovery from overlay partition
Long-term Persistency issue (P2P-based database) Security issue (protection from malicious nodes) 3D content distribution (3D streaming on P2P)
Massive, persistent 3D environment sharable by all!
53
Acknowledgements Dr. Jui-Fa Chen (陳瑞發老師 ) Dr. Wei-Chuan Lin (林偉川老師 ) Members of the Alpha Lab, TKU CS
Guan-Ming Liao (廖冠名 ) Dr. Chin-Kun Hu (胡進錕老師 ) LSCP, Institute of Physics, Academia Sinica
Joaquin Keller (France Telecomm R&D, Solipsis) Bart Whitebook(butterfly.net) Jon Watte (there.com)
Dr. Wen-Bing Horng (洪文斌老師 ) Dr. Jiung-yao Huang (黃俊堯老師 )
54
Inconsistency caused by dAOI
55
Reliability (0-500 steps)
Change in Consistency of over 1000 time-steps (step 1 - 500)
70
80
90
100
0 100 200 300 400 500
time-step
To
po
log
y C
on
sist
ency
(%
)
per step
node 1
node 91
56
Reliability (501-1000 steps)
Change in Consistency of over 1000 time-steps (step 501 - 1000)
70
80
90
100
500 600 700 800 900 1000
time-step
To
po
log
y C
on
sist
ency
(%
)
per step
node 1
node 91
