Embed Size (px)
DESCRIPTION
At talk in jsconf asia
Citation preview
Building fast, scalable game server in node.js
Charlie Crane@xiecc
Who am I
NASDAQ: NTES
Senior engineer, architect(8 years) in NetEase Inc. , developed many web, game products
Recently created the open source game server framework in node.js: pomelo
Top cities for npm moduleshttps://gist.github.com/substack/7373338
AgendaState of pomeloScalabilityFrameworkPerformanceRealtime application
The state of pomelo
Fast, scalable, distributed game server framework for node.js
Open sourced in 2012.11.20 Newest version: V0.7
https://github.com/NetEase/pomelo
Pomelo position Game server framework
Mobile gameWeb gameSocial gameMMO RPG(middle scale)
Realtime application server framework
Realtime Multiple User Interaction
State of pomeloPomelo is not a single projectAlmost 40 repos in total
Framework ---
https://github.com/NetEase/pomelo
Success casesCard Game — ShenMu
http://shenmu.iccgame.com/index.shtml
Q&A
Success cases
http://wap.youth.cn/bangbang/
Realtime Communication Tools —ZhongQingBangBang
Success cases
The magic card hunter https://itunes.apple.com/cn/app/id6649640688
Beauty Fried golden flower http://app.xiaomi.com/detail/41461
Texas Hold ’ em poker http://www.yiihua.com/
Metal Slug http://www.17erzhan.com/ww2/ww2index.php
Coal instant messaging http://www.mtjst.com
Yong Le golden flower http://pan.baidu.com/s/1pCmfD
17 Bullfight http://app.91.com/Soft/Detail.aspx?Platform=iPhone
…
Job huntings
AgendaThe state of pomeloScalabilityFrameworkPerformanceRealtime appliation
Scalability---Web and Game serverWeb server unlimited scalability
Game server
World record: world of tanks(bigworld), 74,536 online usersMMORPG ： 7k-8k maximum
Why?
Why does game server not scale?
Reason 1: Long connection
Response time
Web response time ： 2000ms Game 、 realtime web ： <100ms
Message direction
web: request/reponse game: request/push/broadcast, bi-direction
How to solveNode.js to rescue
Perfect for:Network-insentive applicationReal-time applicationHolding connections
Mqtt: 120,000 connections, 1 process, 10K/connectionSocket.io: 25,000 connections, 1 process, 50K/connection
Why does game server not scaleReason 2: stateful
Web app -- statelessNo coordinates, no adjacencyPartition randomly, stateless
Game server -- statefulHave coordinate, adjacencyPartition by area, stateful
How to solve No good solution in technical level
Depending on game designDon’t make one area too crowdedBalance of area
Too crowded area
Disperse player
Why does game server not scale
MESSAGES IN
1
MESSAGES OUT
1
Reason 3: Broadcast
1 1
Why does game server not scale
MESSAGES IN
2
MESSAGES OUT
4
1 2
12
Why does game server not scale
MESSAGES IN
4
MESSAGES OUT
161 4
14
11
4 4
Why does game server not scale
MESSAGES IN
100
MESSAGES OUT
100001 100
1100
11
100 100
Why does game server not scale
MESSAGES IN
1000
MESSAGES OUT
10000001 1000
11000
11
1000 1000
1 、 How to solve --- broadcastAOI --- area of interested module:
pomelo-aoi
2 、 How to solve -- broadcastSplit process, seperate load , frontend is
statelessFrontend(connector)
Backend(area)
Single Process
broadcast
Game logic
3 、 How to solve – broadcast
Game design, don’t make one place(in area) too crowded
Map sizeCharacter densityDisperse born place
Disperse born place
4 、 How to solve -- pushScheduler
Direct ： send message directlyBuffer: buffer message in array, and flush
it every 50msOther Strategies: flexible, depending on
client number
app.set(‘pushSchedulerConfig’, { scheduler: new MessageScheduler(app);});
Why does game server not scaleReason 4: Tick (game loop)
setInterval(tick, 100)
What does every tick do?Update every entity in the
scene(disappear ， move, revive)Refresh monsterDriving ai logic(monster, player)
Tick must be far less than 100ms
Problem of tick The entity number should be limited
Pay attention to update algorithm: AI etc.
GC, full gc should be limitedV8 is good at GC when memory is under
500M, even full GCMemory must be limitedTry to divide process
At last--- runtime architecture
How to solve complexityToo … complicated?
solution: framework
AgendaThe state of pomeloScalabilityFrameworkPerformanceRealtime application
Pomelo Framework
The essence of pomelo:
A distributed, scalable, realtime application framework.
Framework --- design goalAbstract of servers(processes)
Auto extend server typesAuto extend servers
Abstract of request/response and broadcast/pushZero config request Simple broadcast api
Servers communication---rpc framework
Framework -- some featuresServer scalabilityNetworkPlugins
Server scalability
Server Scalability-- dynamic extensions
pomelo add host=[host] port=[port] id=[id] serverType=[serverType]
node app.js env=production host=[host] port=[port] id=[id] serverType=[serverType]
Network
Connector
SIOConnector HybridConnector
Pomelo component Loader
MQTTConnector
Socket.io client Socket, websocketclient
Mobile client
Network--Message encode, decode
Network--original protobufOriginal protobuf
Network--pomelo protobufOur protobuf
Network -- Proto compressConfig:
Network---protobuf definitionHandler return json, auto encoded to
protobuf
Network--pomelo protobufhttps://github.com/pomelonode/pomelo-protobuf
As an independent project, supporting unity3d, iOS, android, C
Network -- Pomelo Netflow – V0.2Version 0.2, 400 online usersOut: 3645Kb/S in: 270Kb/s
Network -- Pomelo Netflow – V0.3Version 0.3, 400 online usersOut: 925KB/S, in: 217KB/S, 25% Netflow of 0.2
PluginsA mechanism to extend framework
Example:pomelo-status-plugin , online status plugin,
based on redis
Plugins example -- Reliabilitypomelo-masterha-plugin, based on
zookeeper, master HA
Servers auto reconnect Zookeeper
Master
Slave
Slave
AgendaState of pomeloScalabilityFrameworkPerformanceRealtime application
Performance --- overviewThe performance varies largely in different
games, applications
The variationApplication type: Game or realtime
applicationGame type: round or realtime fight, room or
infiniteGame design: Map size, character density,
balance of areasTest parameters: Think time, test action
Performance --- reference data Online users per process
next-gen: support 20,000 concurrent players in one area
World record – not one areaworld of tanks(bigworld), 74,536 online
users
But in real world(MMO rpg)The real online data: maximum 800 concurrent
users per area, 7,000 concurrent users per group game servers
Performance--Server configurationOpenstack virtual machine
Performance --- stress testingUse Case 1: fight
Stress on single area(area 3), increasing step by step, login every 2 seconds
Game logic: attack monstors or other players every 2~5 seconds
Performance --- stress testing
Performance--result486 onlines
Performance -- topUsing toobusy, limit cpu 80%
Performance – stress testUse case 2 – move
Stress on single area(area 3), increasing step by step, login every 2 seconds
Game logic: move in the map every 2~5 seconds
Performance – stress test800 online users
Performance --- stress testingUse Case 3: fight & move
Stress on single area(area 3), increasing step by step, login every 2 seconds
Game logic: 50% attack monstors or other players every 2~5 seconds, 50% move around
Result: 558 onlines in one area
AgendaState of pomeloScalabilityFrameworkPerformanceRealtime application
Realtime app and gameMany success stories in realtime
applicationMessage push platform in NetEaseZhongQinBangBang
Not that frequent messages as gameDo do need that realtime (less than
50ms), we can use different pushScheduler strategy
Much more scalable than game
Message push platform
Supporting more than 10,000,000 online users
Products using message push platformlove.163.commusic.163.comnews.163.com note.youdao.comgame.163.comyuehui.163.comyuedu.163.com
……………………………………
Message push platformAMQP
Receiver publish APNS publisher
Channel push, socket
Connector Connector Connector
Android client iOS client
mqtt mqtt
Redis,Device
LVS
Receiver
Receiver
ReceiverAPNS publisher
Web client
socket.io
Message push systemsHigh load, More than 10,000,000 online
usersRealtime, broadcast to 5 million users in
1minuteReliability, qos=1 in any conditionsSave power and networkSupport all the clients
iOS(APNS, mqtt)android(mqtt)browser(socket.io)windows desktop application(mqtt)
Q&A
@xiecchttps://github.com/NetEase/pomelo
