1

A Network Communication System for Large-Scale, High-Speed Games

Ashwin Bharambe, Jeff Pang, Srini Seshan, Xinyu ZhangCarnegie Mellon

John Douceur, Jacob R. Lorch, Thomas MoscibrodaMicrosoft Research

2

P2P Games

InternetPrimary object

Local View

Replica objects

3

P2P Games

InternetPrimary object

Local View

Replica objects

4

P2P Games

InternetPrimary object

Local View

Replica objects

0

50

100

150

200

0 100 200 300 400 500

Players

To

tal

Ba

nd

wid

th (

Mb

ps

)Goal: 1000 players in the

same battle at the same time

5

Problem 1: Insufficient Capacity

• Update Size: ~35 bytes/frame

• Update Rate: 20 frames/sec

Need 14 Mb/s upload BW per peer

• Real P2P topologies: – 800 Kb/s to 1.5Mb/s per peer (on average)

6

Solution 1: Subscriber Set

• Players can only focus on ~5 objects at the same time

• Each player divides other peers into two classes:– Subscriber Set: players focused on me

• Update every frame– Best Effort: everyone else

• Update with “guidance” once per second

• Who goes in my Subscriber Set?– Attention(i,j) = how much a player i is focused on player j– Attention(i,j) sent from peer i to peer j periodically – Peers with the highest Attention(i,j) go in peer j’s Set

7

Solution 1: Subscriber Set

• Who goes in my Subscriber Set?– Attention(i,j) = how much a player i is focused on player j– Attention(i,j) sent from peer i to peer j periodically – Peers with the highest Attention(i,j) go in peer j’s Set

d1d2

θ1

θ2

Attention(i,j) =

fproximity(d1) faim(θ1) finteraction-recency(t1)+ +

j i

8

Projected Scalability

0

200

400

600

800

1000

1200

1400

0 128 256 384 512 640 768 896 1024 1152 1280 1408 1536

Upload bandwidth per peer (kbps)

Ma

x #

of

pla

ye

rs

With Donnybrook

Without Donnybrook

9

Problem 2: Bandwidth Heterogeneity

• Best Effort updates still scale quadratically• Still need ~1 Mb/s per peer Limited by low bandwidth peers

10

Solution 2: Guidance Forwarding

• Static Overlay Multicast– Need low latency limit tree depth to 2– Can’t queue updates have senders send to

forwarders at 20 updates/sec– Generally have centralized match making

(replicated) centralized coordinator picks forwarders

11

Projected Scalability

12

• Some players more interesting than others More players focus on them They need to send more frequent updates

• Subscriber sets change dynamically (~60% turnover per second!) Can’t use static overlay multicast

Limited by BW of popular players

Problem 3: Interest Heterogeneity

13

Solution 3: Update Dissemination

• Alterative 1:– Static forwarder groups

• Alternative 2:– Centralized forwarder

assignment

• Alternative 3:– Uncoordinated

forwarder assignment

A B

I need BW!

Use B

A BPick random

14

Projected Scalability

15

Evaluation

• Detailed packet level simulation– Empirical simulation models

• E2E latency• Jitter• Access Bandwidth• Packet Loss• Quake III-like CTF workload• ... many other details …• But no churn

16

• Metric: updates delivered roughly on time– updates delivered too late are useless

• 3-5% loss rate acceptable for games

Alternative Comparison

17

Performance Scaling

18

Summary