1 Scalable Peer-to-Peer Networked Virtual Environment Master Thesis Oral Examination Dept. of CSIE,...

1

Scalable Peer-to-Peer Networked Virtual Environment

Master Thesis Oral Examination

Dept. of CSIE, Tamkang Univ.

Advisor: Dr. Chen Jui-Fa

Shun-Yun Hu

2005/01/07

2

Outline

Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion

3

What is Networked Virtual Environment (NVE)?

Virtual Reality + Internet

3D environment with people (avatar), objects, terrain, agents

Military simulations (’80) Massively Multiplayer Online Games (mid-‘90)

Trends: larger scale, more realistic simulation

4

5

NVE: A Shared Space

6

Issues for Creating NVE

Consistency (events/states)

Responsiveness multiplayer Security

Scalability Persistency massively multiplayer Reliability (Fault-tolerance)

7

The Scalability Problem

Many nodes on a 2D plane ( > 1,000) Message exchange with those within Area of Interest (AOI) How does each node receive the relevant messages?

Area of Interest

8

A simple solution (point-to-point)

N * (N-1) connections ≈ O(N2) Not scalable!

Source: [Funkhouser95]

9

A better solution (client-server)

Message filtering at server to reduce trafficN connections = O(N) server is bottleneck

Source: [Funkhouser95]

10

Current solution (server-cluster)

Still limited by servers. Expansive to deploy & maintain.

Source: [Funkhouser95]

11

Scalability Analysis

Scalability constrains Computing resource (CPU) Network resource (Bandwidth)

Non-scalable system vs. Scalable system

x: number of entities

y: resource consumption at the limiting system component

Resource limit

12

What Next?

Strategies Increase resource More servers Decrease consumption Message filtering

Architectures Scale Point-to-point (LAN) tens 10^1 Client-server hundreds 10^2 Server-cluster thousands 10^3 ? millions 10^6 …

Peer-to-Peer

13

What is Peer-to-Peer (P2P)?

[Stoica et al. 2003] Distributed systems without any centralized control

or hierarchical organization Runs software with equivalent functionality

Examples File-sharing: Napster, Gnutella, eDonkey Distributed computing: SETI@Home (UC Berkeley) VoIP: Skype

14

Peer-to-Peer Overlay

A P2P overlay network source: [Keller & Simon 2003]

15

Promise & Challenge of P2P

Promises Growing resource, decentralized

Scalable Commodity hardware Affordable

Challenges Topology maintenance dynamic join/leave Efficient content retrieval no global knowledge

16

Issues for Creating P2P NVE

Consistency (events/states)

Responsiveness multiplayer Security

Scalability Persistency massively multiplayer Reliability (Fault-tolerance)

Consistency (topology) P2P NVE

17

Related Works (1): SimMUD

[Knutsson et al. 2004] (Univ. of Pennsylvania)

Pastry + Scribe Regions Coordinators

(super-nodes)

Fixed-size region Relay overhead

18

Related Works (2)

[Kawahara et al. 2004] (Univ. of Tokyo)

Fully-distributed Nearest-neighbors List exchange

High transmission Overlay partition

19

Related Works (3): Solipsis

[Keller & Simon 2003] (France Telecomm R&D)

Links with AOI neighbor Mutual cooperation Inside convex hull

Potentially slow discovery Inconsistent topology

20

Outline

Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion

21

Design Goals

Observation: for virtual environment applications, the contents we want

are messages from AOI neighbors Content discovery is a neighbor discovery problem

Solve the Neighbor Discovery Problem in a fully-distributed, message-efficient manner.

Specific goals: Scalable Limit & minimize message traffics Responsive Direct connection with AOI neighbors

22

Voronoi Diagram

2D Plane partitioned into regions by sites, each region contains all the points closest to its site

Can be used to find k-nearest neighbor easily

Neighbors

Site

Region

23

Design Concepts

Identify enclosing and boundary neighbors Each node constructs a Voronoi of its neighbors Enclosing neighbors are minimally maintained Mutual collaboration in neighbor discovery

Circle Area of Interest (AOI)

White self

Yellow enclosing neighbor (E.N.)

L. Blue boundary neighbor (B.N.)

Pink E.N. & B.N.

Green AOI neighbor

D. Blue unknown neighbor

Use Voronoi to solve the neighbor discovery problem

24

Procedure (JOIN)

1) Joining node sends coordinates to any existing node

Join request is forwarded to acceptor

2) Acceptor sends back its own neighbor list

joining node connects with other nodes on the list

Acceptor’s region

Joining node

25

Procedure (MOVE)

1) Positions sent to all neighbors, mark messages to B.N.

B.N. checks for overlaps between mover’s AOI and its E.N.

2) Connect to new nodes upon notification by B.N.

Disconnect any non-overlapped neighbor

Boundary neighbors

New neighbors

Non-overlapped neighbors

26

Procedure (LEAVE)

1) Simply disconnect

2) Others then update their Voronoi

new B.N. is discovered via existing B.N.

Leaving node (also a B.N.)

New boundary neighbor

27

Dynamic AOI

Crowding within AOI can overload a particular node

It’s better if AOI-radius can be adjusted in real time

28

Adjustment Conditions

AOI-radius decrease Number of connections > maximum allowable connections

AOI-radius increase Maximum connections not exceeded Current AOI-radius < preferred AOI-radius

Delay counter To avoid fluctuations

29

Demonstration

Simulation video General movements (20 nodes, 800x600 world) Local vs. global view Dynamic AOI adjustment

31

Outline

Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion

32

Simulation Method C++ implementation of Voronoi-based algorithm

World size: 1000 x 1000, AOI: 150 Trials from 10 – 250 nodes Connection limit per node: 10 1000 time-steps

(~ 100 simulated seconds, assuming 10 updates/seconds)

Behavior model Random movement: random direction Constant velocity: 5 units/step Movement duration: random (1 – 25 steps)

33

Consistency Metrics

Topology Consistency [Kawahara, 2004]

Number of observed AOI neighbors

Number of actual AOI neighbors

Drift Distance [Diot, 1999]Distance between observed position and actual position

(average over all nodes)

34

Basic Model

Topology Consistency

Topology Consistency Measurements

90

9192

9394

95

9697

9899

100

0 50 100 150 200 250

Number of Nodes

To

po

log

y C

on

sis

ten

cy

(%

)

35

Basic Model

Scalability (1)

Transmission Size Per Node Per Second

0.0

1.0

2.0

3.0

4.0

5.0

6.0

0 25 50 75 100 125 150 175 200 225 250

Number of Nodes

Siz

e (k

b)

send (max)

send (avg)

recv (max)

recv (avg)

36

Basic Model

Scalability (2)

Average Neighbor Size Measurements

0

2

4

6

8

10

12

14

16

18

0 50 100 150 200 250

Number of Nodes

Nei

gh

bo

r S

ize

connected

AOI

37

Dynamic AOI Model

Effect of Node Increase on Average AOI-radius

0

20

40

60

80

100

120

140

160

0 50 100 150 200 250

Number of Nodes

Av

g. A

OI R

ad

ius

38

Dynamic AOI

Scalability (1)

Average Neighbor Size Measurements

0

1

2

3

4

5

6

7

8

9

10

0 50 100 150 200 250

Number of Nodes

Nei

gh

bo

r S

ize

connected

AOI

39

Dynamic AOI

Scalability (2)

Transmission Size Per Node Per Second

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0 25 50 75 100 125 150 175 200 225 250

Number of Nodes

Siz

e (

kb

)

send (max)

send (avg)

recv (max)

recv (avg)

40

Dynamic AOI

Scalability (3)

Comparison of Voronoi-based P2P and Client-Server

0

20

40

60

80

100

120

140

160

180

0 25 50 75 100 125 150 175 200 225 250

Number of Nodes

Siz

e (

kb

)

send (avg)

recv (avg)

CS-send (avg)

CS-recv (avg)

41

Dynamic AOI

Topology Consistency (1)

Topology Consistency Measurements

90

91

92

93

94

95

96

97

98

99

100

10 30 50 70 90 110 130 150 170 190 210 230 250

Number of Nodes

To

po

log

y C

on

sist

ency

(%

)

42

Dynamic AOI

Topology Consistency (2)

Drift Distance Measurements

0

20

40

60

80

100

0 50 100 150 200 250

Number of Nodes

Dri

ft D

ista

nc

e

max

avg

43

Dynamic AOI

Reliability (1)

Change in Consistency of over 1000 time-steps (step 150 - 250)

70

80

90

100

150 160 170 180 190 200 210 220 230 240 250

time-step

To

po

log

y C

on

sist

ency

(%

)

per step

node 1

node 91

44

Dynamic AOI

Reliability (2)

Average Number of Steps to Recover from Inconsistency

0

1

2

3

4

0 50 100 150 200 250

Number of Nodes

Ste

ps

45

Outline

Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion

46

Analysis of Design

Consistency (Topology) Topology is fully connected & consistent enclosing neighbors

Responsiveness Lowest latency direct connection, no relay

Scalability Resource-growing & decentralized resource consumption

Reliability Self-organizing for small number of node failures

47

P2P NVE Comparisons

SimMUD Neighbor-list exchange

Solipsis VON

Consistency (topology)

Supernode Neighbor list-exchange (partitioning)

Neighbor notify&query (undiscovery)

Neighbor notify (consistent)

Responsive-ness

High overhead

High overhead

Medium overhead

Low overhead

Scalability Relied on supernode

Fully-distributed

Fully-distributed

Fully-distributed

Reliability Long up-time

N/A N/A Self-organizing

48

Problems of Voronoi Approach

Message traffic Circular round-up of nodes Redundant message sending

(inherent to fully-distributed design)

Incomplete neighbor discovery Can happen with inconsistent / incorrect neighbor list Fast moving node

49

Outline

Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion

50

Conclusion

NVE scalability is achievable with P2P architecture and is a neighbor discovery problem

A promising solution: Voronoi-based P2P Overlay Leverage knowledge of each peer to maintain topology

Properties Scalable: fully-distributed, dynamic AOI Efficient: low irrelevant messages, zero relay Robust: consistent and self-organizing topology

51

Potential Applications

Online gamesRelieve server from position updates in current MMOGs

MilitaryEnable large-scale, affordable military training simulation

3D WebProvide multi-user interactivity to static 3D world

Scientific simulationsDistribute spatial simulation requiring frequent synchronization

52

Future Works

Short-term Reliability measurements latency, packet loss, node fail Distributed event/state consistency Recovery from overlay partition

Long-term Persistency issue (P2P-based database) Security issue (protection from malicious nodes) 3D content distribution (3D streaming on P2P)

Massive, persistent 3D environment sharable by all!

53

Acknowledgements Dr. Jui-Fa Chen (陳瑞發老師 ) Dr. Wei-Chuan Lin (林偉川老師 ) Members of the Alpha Lab, TKU CS

Guan-Ming Liao (廖冠名 ) Dr. Chin-Kun Hu (胡進錕老師 ) LSCP, Institute of Physics, Academia Sinica

Joaquin Keller (France Telecomm R&D, Solipsis) Bart Whitebook(butterfly.net) Jon Watte (there.com)

Dr. Wen-Bing Horng (洪文斌老師 ) Dr. Jiung-yao Huang (黃俊堯老師 )

54

Inconsistency caused by dAOI

55

Reliability (0-500 steps)

Change in Consistency of over 1000 time-steps (step 1 - 500)

70

80

90

100

0 100 200 300 400 500

time-step

To

po

log

y C

on

sist

ency

(%

)

per step

node 1

node 91

56

Reliability (501-1000 steps)

Change in Consistency of over 1000 time-steps (step 501 - 1000)

70

80

90

100

500 600 700 800 900 1000

time-step

To

po

log

y C

on

sist

ency

(%

)

per step

node 1

node 91