Networking in Games In-game networking: networking topology

Networking in Games

In-game networking:• networking topology: client-server vs. peer-to-peer• computing model: distributed object vs. message passing• which protocol to use? tcp, udp, reliable udp• bandwidth limitation• latency limitation• consistency• cheat proofing• socket programming WinSocks

Machine 1 Machine 2

in library/kernel(EECS 489)

in game(more in EECS 591/491)

Network: LAN or Internet

Distributed Systems

Networking

Distributed Systems

Networking

Sugih Jamin ([email protected])

Consistency

Problem statement:

time

player1 player2move

fire

movefire

How do you differentiate the two cases, at both player1 and player2?


Synchronization

Synchronization: order moves by their times of occurence

Assume globally synchronized clocks

Out-of-synch worlds are inconsistent

Small inconsistencies not corrected can lead to large compounded errorslater on (deer not speared means one less villager means slower barrackbuild, etc.)


When to Render a Move?

How long do you have to wait for the other players’ moves beforerendering them?

time

player1 player2move

fire

movefire

movefire

X


Lock-step Protocol

• each player receives all otherplayers’ moves before renderingnext frame

Problems:• long Internet latency• variable latencies• speed determined by the slowest

player

time

Player1 Player2

synchronizemoves andrender scene


Bucket Synchronization

• buffer both local and remotemoves

• play them in the future• each bucket is a turn, say for

about 200 ms• bucket size can be adapted to

measured rtt

Problems:• game speed (bucket size) deter-

mined by slowest player• what if a move is lost or late?

time

Player1 Player2

synch movesand render scene

a player can havemultiple movesper turn

X


Pessimistic Consistency

Every player must see the EXACT same world

AoE/AoK/AoM:

• each player simulates its own copy of the world

• all the worlds must be in sync.

• uses bucket synchronization

• each player sends moves to all other players

• dropped packets retransmitted• a designated host collect measured rtts from

all players and set future bucket sizes

Problems:

• variable latencies

• speed determined by the slowest player


Dead Reckoning

• a.k.a. client-side prediction• extrapolate next move based

on prior moves• compute the velocity and ac-

celeration of objects to deadreckon

• players can help by sending thisinfo along

• obviously, only works if ve-locity and acceleration haven’tchanged

v

move lost or late,dead reckoned


Roll-back

In case of inconsistency:

• server always have authoritative view

• when clients correct inconsistent views,players may experience “warping”

• can players’ decisions be dead reckoned?See http://www.sim.inf.ethz.ch/projects/alpsim/


Optimistic Consistency with Roll-back

Observation: dead reckoning doesn’t have to be limited to lost packets!

Half-Life:

• each client plays back its own moves immediately and send themoves to server

• each client also dead reckons the other players’ moves

• server computes world and sends its authoritative version to all clients

• clients reconcile dead reckoned world with server’s version

• can result in some jerkiness and perception of “shooting aroundcorner”

• only need to synchronize important events, but must be careful thatdead reckoning error doesn’t get compounded over time


Consistency: Correctness

For consistency ALL user inputMUST pass through the synchro-nization module

Be careful with random numbergenerators. Isolate the one used forgame-state updating from other uses(ambient noise etc.)

Design for multiplayer from the start.Single-player becomes a specialcase of single-client multiplayergame

Initialization Overall GameControl

Game SessionControl

Local Player Input

Receive Remote Player(s) Input

Main Logic:- consistency- game AI- physics- collision

SendLocal Input

Render sceneto buffer

Copy buffer todisplay

Time sync


Consistency: Smoothness

For smoother playback, decouple bucket size from frame rate(even AoE does this)

Immediately render local moves

Modify game design to allow for latency and loss, e.g.:

• make players wait for elevator

• teleportation takes time

• require multiple hits per kill

• let bullet/missile have flying time

• build in inertia, don’t allow sudden change in facing


Reducing Consistency Check

Do area-of-interest management (a.k.a. relevance filtering):

• aura: how far you can be sensed(cloaked ships have 0 aura)

• nimbus: how far you can sense(use quantum-sensor to detect cloaked ships)

Aura and nimbus are defined for a given set of “technology”(e.g., cloaking device, quantum sensor, etc.)

Perform consistency check only when B is within A’s nimbus andA is within B’s aura


Cheating

AoE doesn’t need cheat-proofing because each player simulates eachmove in lock stepAll moves are simulated, not just collisions

Half-Life synchronizes only collisions, higher probability for cheating

Cheats (more at megagames.com):

• superhuman cheat: auto-aim, auto-position

• game-state editing: boost player’s profile

• rule bending: see/walk through walls

• sixth-sense cheat

• lookahead cheat: claim to be behind slow link

• suppress-correct cheat: exploit dead-reckoning, claim moves werelost, then “reconstruct” advantageous moves based on others’ moves


Lookahead Cheat

time (ms)

b) C is a cheater 50ms from H claiming to be 150ms from H

a) C is an honest player 150ms from H

Player C Player H

0

50

100

200

150

250

0

50

100

200

150

250

t=0

t=50

t=100

t=0

t=50

t=100

Player C Player H

0

50

100

200

150

250

0

50

100

200

150

250

t=0

t=50

t=100t=0

t=50

t=100


Suppress-correct Cheat

At time 150, C sends out a move consistent with fake moves at time 0, 50,100 that were actually computed upon receiving packets from Player H

Player C Player H

0

50

100

200

150

250

0

50

100

200

150

250

t=0

t=50

t=100

XXX


Distributed Computing Model

Message passing:• player inputs (either button

pushes or higher-level move-ments) are sent to other players(or server)

• all players update their owngame state

• or server updates the globalgame state and send it out to allplayers


Game SessionControl

Local Player Input

Receive Remote Player(s) Input

Main Logic:- update states- game AI- physics- collision

SendLocal Input



Time sync

Player inputs sent/received using sockets


Distributed Computing Model

Distributed objects:• characters and environment

maintained as objects• player inputs are applied to ob-

jects (at server)• changes to objects propagated

to all players at end of game loop• object update usually imple-

mented as one or more librarycalls


Game SessionControl

Local Player Input

Send & ReceiveObject Updates

Main Logic:- update objects- game AI- physics- collision



Time sync

Object update library implemented on sockets


Socket Programming

What is a socket?

How to use socket for client-server computing?


Socket Programming

socket: a data structure containing connection information

Connection identifying information:

• client IP (Internet Protocol) address

• client port number

• source IP address

• source port number

Client-server connection:

• server creates a socket and listens for connections on a well-knownport number

• client creates a socket and connects to the server address at thewell-known port number


TCP ConnectionWinSock API TCP Server

socket()

bind()

listen()

accept()

closesocket()

WSAStartup()WSAEnumProtocols()

WSACleanup()

blocks untilclient connects

WinSock API TCP Client

socket()

connect()


recv()

send()

processrequest

send()

recv()

connection establishment

data request

data reply

closesocket()

WSACleanup()


server.cint visits;intmain(int argc, char *argv[]){

struct sockaddr_in self, client;struct hostent *cp;int sd, td, len;char buf[BLEN];

sd = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);memset((char *) &self, 0, sizeof(struct sockaddr_in));self.sin_family = AF_INET;self.sin_addr.s_addr = INADDR_ANY;self.sin_port = htons((u_short) PORT);bind(sd, (struct sockaddr *) &self, sizeof(struct sockaddr_in));listen(sd, QLEN);

while (1) {len = sizeof(struct sockaddr_in);td = accept(sd, (struct sockaddr *) &client, &len);cp = gethostbyaddr((char *) &client.sin_addr, sizeof(struct in_addr), AF_INET);printf("Connected from %s\n", cp->h_name);visits++;sprintf(buf, "This server has been contacted %d time(s).\n", visits);send(td, buf, strlen(buf), 0);close(td);

}}


client.cintmain(int argc, char *argv[]){

struct sockaddr_in server;struct hostent *sp;int sd;int n;char buf[BLEN];

sd = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);

memset((char *) &server, 0, sizeof(struct sockaddr_in));server.sin_family = AF_INET;server.sin_port = htons((u_short) PORT);sp = gethostbyname(SERVER);memcpy(&server.sin_addr, sp->h_addr, sp->h_length);

connect(sd, (struct sockaddr *) &server, sizeof(struct sockaddr_in));

n = recv(sd, buf, sizeof(buf), 0);while (n > 0) {

write(1, buf, n);n = recv(sd, buf, sizeof(buf), 0);

}

close(sd);exit(0);

}


includes and defines

To be prepended to both server.c and client.c:

#include <stdio.h>#include <stdlib.h>#include <string.h>#include <unistd.h>#include <sys/types.h>#include <sys/socket.h>#include <netinet/in.h>#include <arpa/inet.h>#include <netdb.h>

#define SERVER "localhost"#define PORT 4897#define BLEN 256#define QLEN 200


Socket APIs Highlights

WinSock APIs:http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/winsock functions.asp

socket: creates a socket data structure

Then we need to populate the structure with the connection identifyinginformation:

• client IP (Internet Protocol) address

• client port number

• source IP address

• source port number


TCP Socket Addresses

In the socket structure:

match incomingpkts’ destination

copy to outgoingpkts’ destination

bind()

connect()

IP address Port#

bind: used by server only, gives the server socket an IP address and/orport#

connect:

• TCP: initiates connection

• udp: remembers remote address


TCP Socket Addresses

TCP Server:IP address Port#

INADDR_ANY well-known

client’s address ephem-eral

TCP Client:IP address Port#

server’s address

client’s address ephem-eral

well-known


NAT and Firewalls

What are NAT (Network Address Translation) and firewalls?


NAT and Firewalls

NAT boxes remap port numbers (Why?)

A1 3

Ethernet LAN

N1 2 3 4

The Internet

port# mapping123

A.1A.3B.2

B2 5

R2 5

Firewalls may filter out all unknown ports and all UDP packets



socket()

bind()

listen()

accept()

closesocket()


WSACleanup()



socket()

connect()


recv()

send()

processrequest

send()

recv()


data request

data reply

closesocket()

WSACleanup()


Socket APIs Hightlights (cont)

listen:

• specifies max # of pending TCP connections

• only useful for connection oriented services

• TCP SYN denial of service attack

accept:

• waits for client connection

• returns a connected socket (different from the listening socket)



send:

• returns how many bytes are actually sent

• must loop to make sure that all is sent(except for blocking I/O, see UNP Section 6.2)

What is blocking and non-blocking I/O?

Why do you want to use non-blocking I/O?


Different Types of I/O

Synchronous: blocks (puts process to sleep) until I/O is readyBy default operations on sockets are blocking

Waiting for I/O:

1. wait for device availability

2. wait for I/O completion


Non-blocking I/O

Non-blocking I/O: keeps on checking (polling) until device is available

• set socket non-blocking:int on = 1; ioctlsocket(socket, FIONBIO, &on);

• call select on non-blocking socket

Signal-driven I/O: process gets a signal when device is available

• use WSAAsyncSelect() for signals tied to a window

• or WSAEventSelect() for signals not tied to a window

Asynchronous I/O: process notified when I/O completed

• Not widely supported yet

(See UNP Section 6.2 for more info)



socket()

bind()

listen()

accept()

closesocket()


WSACleanup()



socket()

connect()


recv()

send()

processrequest

send()

recv()


data request

data reply

closesocket()

WSACleanup()



recv:

• returns how many bytes are received

• 0 if connection is closed, -1 on error

• if non-blocking: -1 if no data with errno set to EWOULDBLOCK

• must loop to make sure that all is received (in TCP case)

• How do you know you have received everything sent?fixed size (part of protocol definition), prior handshake


Select

select(maxfd, readset, writeset, acceptset, timeout)

• synchronous (blocking) I/O multiplexing

• maxfd is the maximum file descriptor number + 1,so if you have only one descriptor, number 5, maxfd is 6.

• descriptor sets provided as bit mask. UseFD ZERO, FD SET, FD ISSET, and FD CLR

to work with the descriptor sets

• (the fourth parameter is usually called the exceptset)


Select (cont)

select(maxfd, readset, writeset, acceptset, timeout)

• returns as soon as one of the specified socket is ready for I/O

• returns # of ready sockets, -1 on error, 0 if timed out and no device isready (what for?)


recv with select vs. Polling

Which of the following would you use? Why?

loop {

select(. . . , timeout);

recv();

} till done;

or:

loop {

sleep(seconds)

recv();

} till done;



closesocket:

• marks socket unusable

• actual tear down depends on TCPif bind() fails, check WSAGetLastError() for WSEADDRINUSE


Socket Options: getsockopt and setsockopt (UNP Ch. 7)

• SO REUSEADDR: allows server to restart or multiple servers to bind tothe same port with different IP addresses

• SO LINGER: whether close should return immediately or abortconnection or wait for termination

• SO RCVBUF and SO SNDBUF: set buffers sizes

• SO KEEPALIVE: server pings client periodically


UDP Socket Programming

Server must always call bind(), but not listen() nor accept().

Client doesn’t need to call connect().

Use sendto() instead of send().

However, connect() can still be used to tell the system to remember theremote address. Then send() instead of sendto() can be used.

Call either recv() or recvfrom() to recv. recvfrom() also returnsthe address of the client.

UDP packets have boundary, not a byte-stream as in TCP, so recv()

retrieves one message at a time, i.e. no need to call recv() in a loop.


UDP DatagramWinSock API UDP Server

socket()

bind()

closesocket()


WSACleanup()

blocks untildatagram received

WinSock API UDP Client

socket()


sendto()

processrequest

sendto()data request

data reply

closesocket()

WSACleanup()

recvfrom()

recvfrom()


UDP Socket Addresses

UDP Server:

match incomingpkts’ destination

To be filled inwith sender’s addr.by kernel

bind()

IP address Port#

239.4.8.9 9489

UDP Client:

To be filled in withhost’s IP addr. andephemeral port by kernel

copied to outgoingpkts’ destination

connect()

IP address Port#

239.4.8.9 9489


UDP with connect()WinSock API UDP Server

socket()

bind()

closesocket()


WSACleanup()

blocks untildatagram received

WinSock API UDP Client

socket()


sendto()

processrequest

send()data request

data reply

closesocket()

WSACleanup()

recvfrom()

recv()

connect()


Byte Ordering

Big-endian: Most Significant Byte (MSB) in low address (sent/arrives first)(Sun Sparc, HP-PA)

Little-endian: MSB in high address (sent/arrives later) (Intel x86, PS2)

PowerPC and Alpha can be set to either mode

MMORG servers and backend servers may live on big-endian machines


Byte Ordering (cont)

Actual Value 1:

00000000 00000001MSB LSB

sent withouthtons and ntohs

little endian

MSB

LSB

big endianvalue: 2^8

LSB

MSB

00000000

00000001

A+1

A

00000000

00000001

A+1

A

00000000

00000001


Byte Ordering (cont)

To ensure interoperability, ALWAYS translateshort, long, int

to (from) “network byte order” before (after) transmissionby using these macros:

htons(): host to network shorthtonl(): host to network longntohs(): network to host shortntohl(): network to host long


Protocol Layers

Physical

Data Link

Network

Transport

Session

Presentation

Application

socket API

Domain NameSystem (DNS)


Naming and Addressing

DNS (Domain Name System) name in ASCII string:irl.eecs.umich.edu

IP address written out in dotted-decimal (dd) ASCII string:141.213.8.193

IP address in 32-bit binary representation:10001101 11010101 00001000 11000001

Use names instead of addresses:symbolic, easier to remember, variable length string

But fixed-length address provides more efficient handling and fastercomparison, uses less memory and less bandwidth (bw)


Name and Adress Manipulation

Syscalls to map name to/from address:

• dns to b: gethostbyname

• b to dns: gethostbyaddress

and to change representation:

• dd to b: inet addr (inet aton)

• b to dd: inet ntoa

dns to dd: gethostbyname plus inet ntoa


Other useful functions:

• gethostname: returns DNS name of current host

• getsockname: returns IP address bound to socket (in binary) Usedwhen addr and/or port is not specified (INADDR ANY), to find out theactual addr and/or port used

• getpeername: returns IP address of peer (in binary)

Debugging Tools: use tcpdump to look at packets on the networkhttp://windump.polito.it/install/


tcpdump Output

% sudo tcpdump -i fxp0 host tailtcpdump: listening on fxp008:52:07.913485 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: S 1334090569:1334090569(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 28019945 0> (DF)08:52:07.913766 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: S 1738389661:1738389661(0) ack 1334090570 win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 126308216 28019945> (DF)08:52:07.913820 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: . ack 1 win 57920 <nop,nop,timestamp 28019945 126308216> (DF)08:52:07.965499 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: P 1:41(40) ack 1 win 57920 <nop,nop,timestamp 126308221 28019945> (DF)08:52:07.965857 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: P 1:40(39) ack 41 win 57920 <nop,nop,timestamp 28019950 126308221> (DF)08:52:07.966126 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: . ack 40 win 57881 <nop,nop,timestamp 126308221 28019950> (DF)08:52:07.966392 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: P 40:584(544) ack 41 win 57920 <nop,nop,timestamp 28019950 126308221> (DF)08:52:07.966842 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: . ack 584 win 57376 <nop,nop,timestamp 126308221 28019950> (DF)08:52:07.995417 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: P 41:577(536) ack 584 win 57376 <nop,nop,timestamp 126308224 28019950> (DF)08:52:07.995842 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: P 584:608(24) ack 577 win 57920 <nop,nop,timestamp 28019953 126308224> (DF)08:52:07.996143 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: . ack 608 win 57896 <nop,nop,timestamp 126308224 28019953> (DF)08:52:08.053504 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: P 577:1001(424) ack 608 win 57920 <nop,nop,timestamp 126308230 28019953> (DF)08:52:08.146672 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: . ack 1001 win 57920 <nop,nop,timestamp 28019969 126308230> (DF)08:52:08.182531 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: P 608:1024(416) ack 1001 win 57920 <nop,nop,timestamp 28019972 126308230> (DF)08:52:08.183112 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: . ack 1024 win 57504 <nop,nop,timestamp 126308243 28019972> (DF)08:52:08.566220 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: P 1001:1929(928) ack 1024 win 57920 <nop,nop,timestamp 126308281 28019972> (DF)08:52:08.656695 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: . ack 1929 win 57920 <nop,nop,timestamp 28020020 126308281> (DF)08:52:08.755094 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: P 1024:1040(16) ack 1929 win 57920 <nop,nop,timestamp 28020029 126308281> (DF)08:52:08.755369 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: . ack 1040 win 57904 <nop,nop,timestamp 126308300 28020029> (DF)08:52:08.755452 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: P 1040:1088(48) ack 1929 win 57920 <nop,nop,timestamp 28020029 126308300> (DF)08:52:08.755683 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: . ack 1088 win 57872 <nop,nop,timestamp 126308300 28020029> (DF)08:52:08.756357 tail.eecs.umich.edu.ssh > irl.eecs.umich.edu.3465: P 1929:1977(48) ack 1088 win 57920 <nop,nop,timestamp 126308300 28020029> (DF)08:52:08.756654 irl.eecs.umich.edu.3465 > tail.eecs.umich.edu.ssh: P 1088:1152(64) ack 1977 win 57920 <nop,nop,timestamp 28020029 126308300> (DF)


Sources

Stevens, R., Unix Network Programming, 2nd. or 3rd. ed., Prentice-Hall,2004. All you ever want to know about socket programming, even if you’reusing WinSock.

Mulholland & Hakala, Programming Multiplayer Games, WordwarePublishing, 2004. Useful mainly for the WinSock coverage.

Bettner, P. and Terrano, M., “1500 Archers on a 28.8: NetworkProgramming in Age of Empires and Beyond,” GDC 2001

Bernier, Y.W., “Latency Compensating Methods in Client/Server In-gameProtocol Design and Optimization,” GDC 2001

Rothschild, J., “Mpath: Technical Considerations,” Mpath Interactive, 1997


Sources (cont)

Spohn, D., “Internet Game Timeline,” 2003http://internetgames.about.com/library/weekly/aatimelinea.htmhttp://internetgames.about.com/cs/gamingnews/a/goty2003.htm

Bartle, R., “Early MUD History,” 1990http://www.mud.co.uk/richard/mudhist.htm

Ng, Y-S., “Internet Game Design,” Gamasutra, Aug. 1, 1997

Ng, Y-S., “Designing Fast Action Games for the Internet,” Gamasutra,Sept. 5, 1997

Rosedale and Ondrejka, “Enabling Player-Created Online Worlds withGrid Computing and Streaming,” Gamasutra, Sep. 18, 2003


Sources (cont)

Filstrup, B., Cronin, E., and Jamin, S., “An Evaluation of Cheat-ProofingMethods for Multiplayer Games,” NetGames 2002

Brockington and Greig, “Neverwinter Nights Client/Server Postmortem,”GDC 2003, Gamasutra Mar. 6, 2003

Isensee and Ganem, “Developing Online Console Games,” Gamasutra,Mar. 28, 2003

Palm, “The Birth of the Mobile MMOG,” Gamasutra, Sep. 19, 2003


Documents

Networking in Games In-game networking: networking topology