Embed Size (px)
Citation preview
Datenbank Spektrum (2013) 13:179–188DOI 10.1007/s13222-013-0142-x
S C H W E R P U N K T B E I T R AG
Cloud Data Management for Online Games:Potentials and Open Issues
Ziqiang Diao · Eike Schallehn · Shuo Wang ·Siba Mohammad
Received: 24 June 2013 / Accepted: 18 September 2013 / Published online: 3 October 2013© Springer-Verlag Berlin Heidelberg 2013
Abstract The number of players, for massively multiplayeronline role-playing games (MMORPG), typically reachesmillions of people, geographically distributed throughoutthe world. Worldwide revenues for these games increase bybillions of dollars each year. Unfortunately, their complexarchitecture makes them hard to maintain, resulting in con-siderable costs and development risks. For normal opera-tion, MMORPGs have to access huge amounts of diversedata. With increasing numbers of players, managing grow-ing volumes of data in a relational database becomes a bigchallenge, which cannot be overcome by simply adding newservers. Cloud storage systems are emerging solutions fo-cusing on providing scalability and high performance forCloud applications, social media, etc. However, Cloud stor-age systems are in general not designed for processing trans-actions or providing high levels of consistency. In this paper,we present our current work-in-progress by analyzing theexisting architecture of MMORPGs and classifying relevantdata. Based on this, we highlight the design requirements,identify the major research challenges, and propose a Cloud-
This is an extended version of the paper “Cloud Data Management forOnline Games: Potentials and Open Issues” selected for the specialDASP issue Best Workshop Papers of BTW 2013 [1].
Z. Diao (B) · E. Schallehn · S. Wang · S. MohammadInstitute of Technical and Business Information Systems,Otto-von-Guericke University Magdeburg, Universitätsplatz 2,39106 Magdeburg, Germanye-mail: [email protected]
E. Schallehne-mail: [email protected]
S. Wange-mail: [email protected]
S. Mohammade-mail: [email protected]
based model for MMORPGs that we currently implement asa testbed for further evaluation.
Keywords Multiplayer online game · Cloud datamanagement · Cassandra
1 Introduction
A massively multiplayer online game (MMOG) can supporthundreds or thousands of players from all over the world inparallel. In a virtual game world, players can choose a newidentity, establish a new social network, compete or cooper-ate with other players, or even realize dreams that cannot befulfilled in real life. It is estimated that, in 2011, the Amer-icans spent 26 million hours and 2.6 billion dollars in totalfor playing MMOGs [15]. The game industry is developingrapidly, and the global MMOG market volume has grown to13 billion dollars in 2012 [16].
What differs between MMORPGs and other MMOGs,such as first-person shooters and real-time strategy games,is that the virtual world environment of MMORPGs runseven in the case of no players. In addition to the accountinformation, the state data of objects and characters mustbe recorded on the server side in real-time, so that playerscan quit and continue the game at any time. Player behav-ior in the game should be monitored and backed up in or-der to maintain the order of the virtual world. Furthermore,MMORPGs usually have more concurrent players than anyother MMOGs, (for example, World of Warcraft has mil-lions of concurrent players), which also exacerbates the bur-den of managing data. A qualified database system for datapersistence in MMORPGs must guarantee the data consis-tency and also be efficient and scalable [2]. However, ex-isting RDBMS cannot fully satisfy all these requirements
180 Datenbank Spektrum (2013) 13:179–188
Fig. 1 Principle architecture ofMMORPGs
simultaneously [7, 11]. Therefore, with an increasing datavolume, the storage system becomes a bottleneck, and solv-ing scalability and availability issues becomes a major costfactor and a development risk.
Cloud storage systems, which have the ability to sup-port highly concurrent data accesses and huge storage, maybecome a solution. Unfortunately, contrary to conventionalDBMSs, cloud systems are generally designed for Webapplications that have different access characteristics andrequire lower or different consistency levels. Therefore,MMORPGs need to be analyzed in more detail to assess theusability of Cloud storage systems, open issues, and possiblesolutions for them.
The approach presented in this paper is work in progress.We are currently focusing on architectural issues, concep-tual work on fulfilling specific data management require-ments, and the creation of a testbed implementing these con-cepts for further evaluation. The core ideas were presented atthe DMC workshop 2013. This paper extends the workshopcontribution by
– descriptions of more details on the current state of thetestbed implementation, including lessons learned up tothis stage of the project,
– a more thorough discussion of problems and open issuesof using Cloud storage systems for MMORPGs, and
– a more fine-grained outline and discussion of our futureresearch directions, which will most of all focus on eval-uating critical aspects for this type of application, e.g.,scalability and consistency.
The rest of the paper is structured as follows. In Sect. 2,we analyze the existing architecture and data managementof MMORPGs. In Sect. 3, we classify data of MMORPGsinto four groups and discuss their requirements. In Sects. 4and 5, we propose a Cloud-based data management modelfor MMORPG, and, finally, conclude and summarize thispaper in Sect. 6.
2 Analysis of Current Data Management forMMORPGs
In order to support players’ interaction and ensure systemsecurity [2], an MMORPG typically employs a client-servermodel or a browser-server model. Usually both models havea similar architecture on the server side. Figure 1 shows asimplified architecture of MMORPGs [7].
A player sends a request to the login server, which is re-sponsible for determining its validity. The login server co-operates with an account database that stores user accountinformation. If the validation is passed, the login server en-crypts the user ID, generates a token, and then returns itto the player. The player uses this token to communicatewith the gateway server, which is an information transit sta-tion. The gateway server is generally composed of multi-ple servers, the main functions of which are to maintainthe connection state of a player, monitor the player’s re-quests, and transmit it to a zone server or a chat server. SinceMMORPGs need to satisfy connection requests from globalplayers, these servers are usually deployed in parallel in sev-eral data centers around the world. Note that only one zoneserver provides services in one session. The zone server isresponsible for maintaining a global virtual game world forplayers, executing the game rules, simulating interactionsamong characters, and synchronizing the simulation resultsto the relevant players. In order to balance the server work-load, a zone server is composed of multiple map servers andlogic servers. The zone server reads game scripts and rulesfrom a game database and accesses state data of charactersfrom a state database. A game database is hosted in all datacenters in parallel. However, each data center has its ownunique state database. A player has to execute the charac-ter migration operation before she/he transfers to anotherzone server. When a player starts an MMORPG, all char-acter state data of this player must be read from the statedatabase at once and then cached in an in-memory database.During a game, the zone server manages data in memory
Datenbank Spektrum (2013) 13:179–188 181
Table 1 Data classification and requirements analysis
Data Consistency Performance Availability Scalability Partitioning Flexible model Simplified processing Security
Account Data � � � � � � � � � � � � � � � �Game Data � � � � � � � � � � � �State Data � � � � � � � � � � � � � � � � � � � � � � �Log Data � � � � � � � � � � � � � � � � �
in real-time and writes them periodically to disk. When aplayer quits the game, date representing his/her characterstate is persisted to the state database and deleted from theactive game data in memory. The log database in this archi-tecture records all player operations and sometimes also thechat history. Data is collected for purposes of data miningand intrusion detection.
Currently, the popular MMOPRGs mainly apply a dis-tributed RDBMS to store player account data and gamestate (e.g., Second Life and World of Warcraft both runon MySQL [24], Guild Wars runs on Microsoft SQLServer [23]), which can commit complex transactions andare proven to be stable. As an example, consider MySQLCluster [17] and its characteristics. MySQL Cluster adoptsa shared-nothing architecture to ensure scalability. In orderto balance the workload among nodes, it automatically par-titions data within a table based on the primary key acrossall nodes. Each node is able to help clients to access cor-rect shards to satisfy a query or commit a transaction. Forthe purpose of guaranteeing availability, data is replicated tomultiple nodes. MySQL Cluster applies a two-phase com-mit mechanism to propagate data changes to the primaryreplica and one secondary replica synchronously, and thenmodifies other replicas asynchronously. In this case, at leastone secondary replica has the consistent and redundant data,which can be used as a fail-over server when the primaryserver fails. MySQL Cluster also writes redo logs and check-point data to the disk asynchronously, which can be used forfailure recovery. When MySQL Cluster maintains tables inmemory, it can support real-time responses. The result is thatit can also be used as an in-memory database in MMORPGs.
3 Data Management Requirements of MMORPGs
From a system’s point of view, the essence of a game isdata processing, storage, and transmission among databases,servers, and game clients. To point out data managementrequirements, for our following considerations we classifydata into four data sets. These different classes vary widelyregarding their requirements and have to be managed ac-cordingly.
Account data: This category of data includes user ac-count information, such as user ID, password, recharge
records, and account balance. This data is usually onlyused when players log in to or log out of a game foridentification and accounting purposes.
Game data: Data such as world geometry and appearance,object and NPC (Non Player Character) metadata(name, race, appearance, etc.), system logs, configu-ration, and game rules/scripts in an MMORPG are gen-erally only modified by game developers. Some signif-icant part of game data is often stored on the client sideto minimize network traffic for unchangeable data.
State data: Player character (PC) metadata, position as wellas state of characters and objects, and inventory inMMORPGs are modified constantly. Modifications ofstate data are currently executed by an in-memorydatabase in real-time and backed up to the disk-residentdatabase periodically.
Log data: Analyzing user chat history and operation logs inan MMORPG is the most objective and direct way forgame providers to evaluate the game, find out the oper-ating habits of players, explore the game developmenttrends, and even supervise the financial system of thegame world.
For these classes, we summarize data management re-quirements in Table 1 and discuss them in the following.
Support for Different Levels of Consistency in a collabora-tive game, players interact with each other. Changes of statedata must be synchronously propagated to the relevant play-ers within an acceptable period of time. For this purpose, weneed a continuous consistency model in MMORPGs [12].The state data and account data changes must be recorded inthe database. It is intolerable that players find that their lastgame records are lost when they log in to the game again.As a result, a strong or at least Read-Your-Writes consis-tency [8] is required for such data. However, strong con-sistency is not necessary for log data and game data. Forexample, the existence of a tree in the map, the synchro-nization of a bird animation, or the clothing style of a gamecharacter is allowed to be different among client sides. Logdata is generally not analyzed immediately. Hence, even-tual consistency [8] is sufficient for these two classes ofdata.
182 Datenbank Spektrum (2013) 13:179–188
Performance/Real-Time state data is modified constantlyby millions of concurrent players, which brings a signifi-cant traffic to game servers (one player sends an average of6 packets per second [25]), thereby generating thousands ofconcurrent database connections. Such changes must be ex-ecuted in real-time (within 200 ms [25]) and persisted ondisk efficiently, which have become a challenge to databaseperformance.
Availability as an Internet/Web application, an MMORPGsystem should be able to respond to each user request withina certain period of time. If lags or complete denial of ser-vices appear frequently, this will significantly decrease theacceptance of the game and will result in sinking revenues.Availability can be achieved by increasing data redundancyand setting up fail-over servers.
Scalability typically, online games start with a small ormedium number of users. If the game is successful, thesenumbers can grow extremely. To avoid problems of a sys-tem laid out for too few users or its costs when initiallylaid out for too many users, data management needs to beextremely scalable [13]. Furthermore, log data will be ap-pended continually and retained in the database staticallyfor a long time [7]. The expansion of data scale should notaffect the database performance. Hence, the database shouldhave the ability to accommodate the growth by adding newhardware [5].
Data Partitioning performing all operations on one nodecan simplify the integrity control, but that may cause a sys-tem bottleneck. Therefore, data must be partitioned intomultiple nodes in order to balance the workload, processoperations in parallel, and reduce processing costs. Currentpartitioning schemes are most often based on applicationlogic, such as partial maps (map servers). This does noteasily integrate with the requirement of scalability, i.e., re-partitioning is not trivial when new servers are added. Ac-cordingly, suitable partitioning schemes are a major researchissue.
Flexible Data Model part of the state data does not have afixed schema, for example, PCs have varying abilities, tasks,and inventory. Additionally, MMORPGs are typically bug-fixed and extended during their runtime. Therefore, it is dif-ficult to adopt the relational model to manage such data. Aflexible data model without a fixed schema is more suitable.
Consistent Data Management In MMORPGs, only up-dates to account data and part of state data must be executedin the form of transactions. In addition, transaction process-ing in online game databases is different from that in busi-ness databases. For example, in MMORPGs, there are many
transactions, but most of them are of small size. Parallel op-erations with conflicts occur rarely. Especially in the stateand log database, which are responsible for data backup,there are no write conflict among players. Write conflicts,in these two database, are usually caused by network la-tency or server failure, which can be addressed by checkingtimestamps. Using locks as in a traditional database will in-crease response time. Additionally, deadlock detection in adistributed system is not easy. Hence, a simplified data pro-cessing mechanism is required.
Security game providers have to be concerned about datasecurity because data breach may lead to economic risks orlegal disputes. For this reason, user-specific data, such asaccount data and chat logs, must be strongly protected. Fur-thermore, it must be possible to recover data after being ma-liciously modified.
Ease of Use, Composability, and Re-usability The datamanagement system should be easy to use for developers,and it should be easy to apply it to various MMORPGs.Companies developing and maintaining MMORPGs shouldbe able to re-use or easily adapt existing data manage-ment solutions to new games, similar to the idea sepa-rating the game engine from the game content currentlywidely applied. An interesting solution would be game datamanagement provided as a service to various providers ofMMORPGs, but this strongly depends on building up trust-worthy services.
4 The Case for Cloud-Based Data Management forMMORPGs
Currently, MMORPGs mainly apply RDBMSs for data per-sistence. However, an RDBMS cannot provide some datamanagement requirements well, and considerable effortshave to be spent on compensating these requirements atother levels [11]. In contrast to RDBMSs, Cloud-based stor-age systems have inherent advantages regarding their majorcharacteristics such as scalability and availability. Neverthe-less, some typical requirements are neither fully supportedby any of the two alternatives. Accordingly, we discuss keyrequirements and the suitability of the RDBMS vs. Cloud-based storage systems.
High Performance traditional RDBMSs limit the numberof columns in each table. For instance, Oracle 10g supports1000 columns maximum [27] and SQL Server 2005 sup-ports 1024 columns maximum [26]. If we create a wide ta-ble, i.e., with hundreds of columns, RDBMSs often fail tomanage it well. If we partition a wide table into several ta-bles (see Fig. 2), the high number of required join opera-tions and locking would greatly impact on the system per-formance. Different from RDBMSs, Cloud-storage systems
Datenbank Spektrum (2013) 13:179–188 183
Fig. 2 An example schema ofstate data in an RDBS
can manage all attributes together efficiently by applyingthe simple key-value data model and data redundancy. Forexample, one row in Cassandra can contain up to 2 billioncolumns [28]. Although this solution takes some extra stor-age space, it improves the system performance significantly.
Scalability for large-scale Internet/Web applications, a sin-gle or few servers cannot satisfy the user demand for dataaccess. The only solution is scaling out. Although someRDBMS (e.g. MySQL Cluster, Oracle RAC, etc.) also haveused the shared-nothing architecture, the system’s scalabil-ity is limited by their complex architecture and data model(e.g., compliance of third normal form and data integrity).Furthermore, the growth of data volume has a significantimpact on the system performance [10, 11]. Here, with theirsimplified architecture and data model, Cloud storage sys-tems were designed to provide this high level of scalabil-ity [10].
Flexible Data Model RDBMS apply the relational model,and the table structure as well as the data type is fixed. Itis good for normalizing table schema, removing data redun-dancy, and query processing, but not for adapting to a dy-namic schema and processing big data. In contrast, Cloudstorage systems typically adopt a flexible data model, suchas the key-value data model. Data in the Cloud is heteroge-neous (structured, semi-structured, and unstructured). Thereis no fixed schema for items. Each item consists of a key foridentification and a dynamic set of attributes as value.
Consistent Data Management RDBMS usually follow atransaction mechanism, thereby providing ACID guarantees(Atomicity, Consistency, Isolation, and Durability) and roll-back. As we discussed above, this is not necessary for alldata sets in MMORPGs. For these data sets, the overheadof this ill-suited consistency mechanism becomes a draw-back. On the other hand, Cloud systems are designed forWeb applications such as Blogs and SNS (Social NetworkServices), which have a high demand for concurrent writingand reading of big data, but a low demand for timeliness ofthe information. For this reason, they generally follow the
BASE principle (Basically Available, Soft state, and Even-tually consistent). This is insufficient, e.g., for account data.For game state data, requirements fall between the two endsof the scale, and we suggest to adjust consistency manage-ment for Cloud data management systems accordingly.
Availability RDBMS, nowadays, provide solutions to im-prove availability by replication and fail-over within thecontext of distributed databases. Nevertheless, these prop-erties are not trivial to implement and maintain for a givenimplementation, especially considering the large scale ofMMORPGs. Cloud data management systems were build tosupport these aspects in a self-managing way.
4.1 An Architecture for Cloud-Based MMORPGs
As outlined above, Cloud-based data management does pro-vide strong advantages for some data sets, but in its currentform does not fulfill all requirements. For these reasons, wepropose a set of cooperating and, if required, composableCloud services to improve the existing game architecture.The general idea is to manage data in online games accord-ing to requirements for this specific class. Subsets of datahaving strong requirements for system scalability or a flex-ible data model will be persisted in a Cloud file system orin a Cloud-structured data system; other data sets are pro-vided by Cloud services that are similar to existing solutions.Figure 3 shows the proposed Cloud-based architecture forMMORPGs.
Account data must be processed with high requirementsregarding consistency and security as outlined before. As aresult, each operation should be executed within a transac-tional context. Furthermore, the scale of account data is notlarge. Accordingly, we suggest managing account data withan RDBMS as a service. We do not suggest storing accountdata in an untrusted host such as in a public Cloud for secu-rity reasons. Before the public Cloud providers bring trust-worthy data security solutions, game providers should buildprivate Cloud infrastructures, which employ the Cloud ar-chitecture and are maintained and protected by game provi-ders themselves.
184 Datenbank Spektrum (2013) 13:179–188
Fig. 3 A Cloud-basedarchitecture for MMORPGs
Log data has a large scale. Once it has been written todisk, it does not need to be modified anymore. In addition,log data is usually analyzed after a long period of time, sotimeliness and consistency are less critical. Generally speak-ing, in addition to the demand for scalability, managementrequirements of log data are not difficult to meet. Therefore,there are some existing Cloud storage systems that can beused directly, such as Cassandra [18]. Cassandra provideshigh throughput for accessing data across a large number ofservers and has the feature of high fault tolerance. Cooper-ating with Hadoop MapReduce, it can process and analyzelarge data sets in parallel.
Game data processing does not pose a challenge to theexisting solutions. Note that strong consistency across datacenters is not required by game data. We can reduce the levelof data consistency to improve system availability. There-fore, unless game data is stored on the client side, we canmanage it in a Cloud file system (game data is typically inthe form of files), such as Hadoop Distributed File System(HDFS) [9].
As discussed above, managing state data in real-time be-comes a big challenge for a disk-resident RDBMS. Unfortu-nately, Cloud storage systems with shared-nothing architec-tures also fail to provide real-time processing capabilities.Although some Cloud storage systems manage data in themain memory of each node, we cannot limit all related datato one node. Hence, changes of data must be propagated toother nodes as messages, which will increase the responsetime. Furthermore, Cloud storage systems do not supporttransaction processing, which may lead to abnormal oper-ation. Therefore, we suggest to keep an in-memory databasein each zone server as done in current architectures.
This assumption given, there will still be a need to backupstate data to the disk in order to recover an in-memorydatabase from server failures. In this process, a flexible datamodel, simplified data processing, high-performance con-
current processing, and system scalability are required. Allof which are weaknesses of RDBMS. As a result, it poses thebiggest challenge to the existing architecture of MMORPGs.In contrast, Cloud-structured data systems are able to satisfymost of theses requirements. Therefore, we propose to per-sist all state data in the Cloud. Note that backup of state datais update-intensive, which is a similar application scenarioas those typically supported by Cassandra. For this reason,our current research focus is on managing state data in Cas-sandra.
5 Using Cassandra for MMORPGs
In the context of this project, we have developed an exper-imental prototype environment for MMORPGs using Cas-sandra. The main goals of this ongoing research are:
– providing a proof of concept for Cloud-based MMORPGs,– evaluating and refining technical aspects like perfor-
mance, consistency, and availability, and– estimating necessary efforts required for the develop-
ment, operation, maintenance, and re-usage of Cloud-based MMORPGs.
Based on the architecture described in Fig. 3, our proto-type consists of three layers (client, game server, and Clouddatabase). On the server side, we have used the DarkStar(RedDwarf Server) project, which was specifically designedfor MMOGs by Sun Microsystems [19]. Darkstar is anopen-source middleware that is easy to use and powerful; forexample, it supports transaction processing, CPU load bal-ancing, and scalability [19, 20]. In our prototype, we use itto keep the communication between servers and clients, butported the code to use Cassandra as a database, currentlyrunning on Cassandra 1.2.5, which is the latest stable ver-sion [18]. We use eight Ubuntu virtual machines to host our
Datenbank Spektrum (2013) 13:179–188 185
experimental prototype. Currently, Cassandra is deployed ina cluster of five machines, and game servers are deployed inanother three virtual machines. Game servers support somesimple game logics, such as receiving a player’s login re-quest, keeping the player’s session, updating game states,handling player transaction requests, as well as player in-teractions (walking and chatting). Data is persisted to Cas-sandra every five seconds in the test environment. To sim-plify system design, we use Cassandra to manage both statedata and log data. Currently, we are working on creating atestbed that can be used to simulate a complex MMORPGenvironment with many game servers and a huge number of(randomly scripted) game clients.
So far, we have found the following advantages and is-sues of Cassandra within the project. Cassandra has a decen-tralized (peer-to-peer) structure [6], i.e., each node is iden-tical and is able to initiate reads and writes independently.Data is automatically replicated to multiple nodes. There-fore, there is no network bottleneck and single point of fail-ure. The write performance requirements of state data canbe satisfied easily and in a scalable way. This is differentfrom RDBMS (e.g., MySQL Cluster) and some other Cloudstorage systems (e.g., Google Bigtable [4]), which adopt aprimary/secondary model, where the primary node may be-come a system bottleneck.
Cassandra provides a column-family-based data model,which is richer than a simple key-value data model. Eachrow in a column family consists of a row key and a dynamicset of columns. Because of this, it is possible to manage thestate data of one character in a single row and partition databased on row key across multiple nodes in the cluster (seeTable 3). Furthermore, players only access their own statedata, so there is no join operation during reading data. Asa result, the read performance can be increased. Similarly,during a game, log data of a player can also be stored in onerow (see Table 2). For example, the row key is composed ofthe player character ID and player’s login time. The columnname (key) is the timestamp of each checkpoint, and the col-umn value is the corresponding log data. Cassandra supportstwo billion columns per row in theory [28], which fulfillsthe requirement of saving log data. Additionally, Cassandrastores data according to the sequence of column names au-tomatically, so a read request can get the serialized log dataconveniently.
Cassandra applies a hashing algorithm to create an MD5hash value of every row key. Each node in the cluster is as-signed a token that represents a range of hash values. Cas-sandra distributes data to the corresponding node accordingto the hash value of the row key. Hence, data with adjacentrow keys (character ID) can be evenly distributed to differentnodes. Note that in a game usually newly registered playersare active. In this way, state data of player character can alsobe allocated to each node evenly, which helps to balance theworkload of nodes.
Cassandra adopts a shared-nothing architecture and asimplified data model as we mentioned above. As a result,it can scale out easily by adding new hardware, and reacha linearly increasing read and write throughput [22]. Thatis also the reason that we can manage state and log data inCassandra. Another advantage is that Cassandra provides aquorum-based data replication mechanism. That means thatas long as a write can receive a quorum of responses fromreplicas, it can complete successfully. In this way, Cassan-dra ensures availability and fault tolerance. Additionally, bycontrolling the number of replicas that must respond to aread request, Cassandra offers a tunable data consistency.
On the other hand, Cassandra is not designed for onlinegames, so there are still some notable open issues.
Data Consistency Cassandra employs ReadRepair func-tionality to guarantee different levels of consistency. Thismeans, all replicas will be compared when Cassandra ex-ecutes a read operation before or after returning the re-sults, depending on the specified consistency level. InMMORPGs, state data may contain hundreds of attributesand are distributed in multiple data centers. Hence, such afeature may significantly reduce the performance of fullyconsistent reads and increase the network traffic. A com-mon method for solving this problem is to limit the dataconsistency in the application layer of the system [3]. Notethat Cassandra records timestamps in each column, and usesit as a version identification. Therefore, a possible solutionis to record the timestamps of a global server (in this pa-per, it is the access server) in both, server side and columnsin the Cloud storage system. When we read state data fromthe Cloud, the timestamps recorded on the server side willbe sent with the read request. In this way, we can find outthe most recent data easily. We set all columns in a singlerow with the same timestamp, so that only one row key andone timestamp are stored for a single row on the server side.In addition, these timestamps are partitioned and managedby access servers in parallel. For these two reasons, access-ing these timestamps on the server side will not become asystem bottleneck.
Network Traffic Cloud-based MMORPGs have to face po-tential bandwidth bottlenecks due to the following reasons:in shared-nothing systems, nodes communicate with eachother via messaging. As a result, a large amount of mes-sages are transmitted over the network; different from tablesin relational databases, a column family in Cassandra doesnot have a fixed schema. For this reason, Cassandra has toreturn the key-value pairs when it responds to a read request,but not only values. This problem is particularly significantfor MMORPGs that need to backup the state data snap-shots, which contain a large amount of redundant data. For
186 Datenbank Spektrum (2013) 13:179–188
Table 2 A possible structure of logs in Cassandra
〈rowkey〉 CharacterID@Timestimp 〈Column Family〉 Log
1220@201301040603 201301040605 201301040610 201301040615 201301040620 201301040625
login. . . . . . . . . . . . . . .logout
1127@201301181511 201301181515 201301181520 201301181525
login. . . . . . . . .logout
Table 3 An example for character state data in Cassandra
〈rowkey〉 CharacterID 〈Column Family〉 Character
1127 Name . . . Race Gender Item 1 Item 1 Attack Item 1 Add Magic Item 2 Item 2 Defense
Ann . . . elf Female bow 40 20 armour 100
1220 Name . . . Race Gender Item 1 Item 1 Attack
Alex . . . human Male sword 80
Table 4 An improved column family
〈rowkey〉 CharacterID 〈Column Family〉 Character
1127 Name: Ann MaxHealth: 200 . . . Item 1: bow Item 2: armour
Female 180 . . . A40-AM20 D100
1220 Name: Alex MaxHealth: 300 . . . Item 1: sword
Male 225 . . . A80
this reason, in case of network failure or insufficient band-width, data access in a Cloud-based MMORPG is more eas-ily blocked than that in a traditional MMORPG. Here aresome possible solutions: messages should be compressedbefore transmission; both key and value correspond to val-ues in an RDBMS (see Table 4). For example, we store PCname as column name and PC gender as column value. Ifan object has several attributes, we can use special symbolsto separate these attributes and then store it in one column.For example, the value ‘A40-AM20’ in Table 4 representsthe game fact that the attack power of a sword is 40 points,while there is an additional 20 points magic damage. This ispossible, because these attributes are only accessed togetherand never used for querying. If only some of the less impor-tant attributes (e.g., position or orientation of a character)of state data are modified, the current round backup can beignored [2, 14]; only the timestamp, the row key, and themodified values are sent as a message to Cassandra.
Statistics Currently, Cassandra does not provide data basestatistics within a data dictionary as RDBMS do. Neverthe-less, this information is helpful to support the operation,maintenance, tuning of the overall system. Though Cloudstorage systems are intended to be self-managing to somedegree, these capabilities are still limited. Hence, game de-velopers have to put more efforts and spend more time to
maintain a Cloud-based MMORPG. We address related is-sues in related research [21].
Analytical Data Processing At this level, Cloud storagesystems such as Cassandra provide only very limited supportfor analytical processing, required for MMORPGs, e.g., formarketing purposes and strategic game development. Sup-port for analytical processing can be achieved in three ways.
1. Game data can be analyzed by applying Map/Reduce ontop of the persistence servers.
2. Relevant game data could be loaded into a further Cloud-based system such as Hive, comprising a separate storagelayer and Map/Reduce.
3. Relevant game data could be exported and loaded into atraditional RDBMS-based data warehouse system.
Currently, we do not focus on analytical processing or whichof these is the best approach. A decision has to be made interms of analysis requirements such scalability, performanceand so on, while still having the impact on the running gameservers in mind.
Query Capabilities Cassandra’s query language CQL al-lows for SQL-like statements to change and look up data inCassandra. However, the capabilities of CQL are not as pow-erful as SQL in RDBMS. For example, the WHERE clause
Datenbank Spektrum (2013) 13:179–188 187
for filtering rows can only filter column values if the columnis indexed. Therefore, game developers have to consider allof the possible queries when they design a column family.That will increase the difficulty of game development andlimit the expansion of game functionality in the future.
Customized Functionality specific features of MMORPGsmake it necessary to develop new functionality for Cas-sandra. Consider the following two examples: the task ofa Cloud storage system is data backup for the in-memorydatabase. Each value that has been written to the Cloud mustbe persistent until it is out of date. Therefore, if an update op-eration fails in the committing process, values that have beenrecorded in the database should be applied and for othervalues update should be retried. To comply with this rule,a column-level atomicity is sufficient (Cassandra offers anatomicity at the column family level); note that writes andqueries in games are relatively fixed. Hence, we propose toadd a stored procedure to each node to optimize the systemperformance.
Based on the analysis above, we believe that although theadvantages of using Cassandra in MMORPGs are obvious,there are still many specific problems to be solved. Never-theless, considering that functionality of Cassandra is beingconstantly updated, we intend to focus on improving Cas-sandra in our future research to manage state data as well aslog data.
6 Conclusion
In this paper, we proposed a Cloud-based data manage-ment for MMORPGs to improve system performance andscalability. We outlined the main criteria for this proposal,by classifying and describing the typical requirements ofMMORPGs. The proposed model adopts a shared-nothingarchitecture to ensure scalability and customizes a set of datamanagement solutions, so that the new system can operateefficiently. We classified data into four groups and found outthat only state and log data should be recorded in differentways in the Cloud storage system and managed. For exam-ple, the model promises a strong consistency guarantee tostate data, but only an eventual consistency guarantee to logdata. To further improve the system’s performance, we alsoproposed some solutions to reduce the number and size ofmessages. In our future work, we will design and carry out aseries of experiments on our testbed in order to evaluate per-formance and scalability of it. Evaluation of performancewhile using different databases (e.g., MySQL, HBase, andCassandra) in MMORPGs is also in our plan.
References
1. Diao Z, Schallehn E (2013) Cloud data management for onlinegames: potentials and open issues. In: BTW workshop on datamanagement in the cloud (DMC)
2. Zhang K, Kemme B, Denault A (2008) Persistence in mas-sively multiplayer online games. In: Proc of the 7th ACM SIG-COMM workshop on network and system support for games(NETGAMES), pp 53–58
3. Gropengießer F, Baumann S, Sattler K-U (2011) Cloudy trans-actions cooperative XML authoring on Amazon S3. In: Daten-banksysteme für Business, Technologie und Web (BTW), pp 307–326
4. Chang F, Dean J, Ghemawat S, Hsieh WC, Wallach DA, BurrowsM, Chandra T, Fikes A, Gruber RE (2006) Bigtable: a distributedstorage system for structured data. In: Proc of 7th symposium onoperating system design and Implementation (OSDI), pp 205–218
5. Iimura T, Hazeyama H, Kadobayashi Y (2004) Zoned federationof game servers: a peer-to-peer approach to scalable multi-playeronline games. In: Proc of the 3rd workshop on network and systemsupport for games (NETGAMES 2004), pp 116–120
6. Lakshman A (2010) Cassandra—a decentralized structured stor-age system. Oper Syst Rev 44:35–40
7. White W, Koch C, Gupta N, Gehrke J, Demers A (2007) Databaseresearch opportunities in computer games. SIGMOD Rec 36:7–13
8. Vogels W (2008) Eventually consistent. ACM Queue 6:14–199. Shvachko K, Kuang H, Radia S, Chansler R (2010) The Hadoop
distributed file system. In: IEEE 26th symposium on mass storagesystems and technologies (MSST), pp 1–10
10. Franke C, Morin S, Chebotko A, Abraham J, Brazier P (2011)Distributed semantic web data management in HBase and MySQLcluster. In: IEEE international conf on cloud computing (CLOUD2011), pp 105–112
11. Cattell R (2010) Scalable SQL and NoSQL Data Stores. SIGMODRec. 39:12–27
12. Li FWB, Li LWF, Lau RWH (2004) Supporting continuous consis-tency in 12th Multiplayer online games,. In: ACM Multimedia’04,pp 388–391
13. Gupta N, Demers A, Gehrke J (2008) SEMMO: a scalable enginefor massively multiplayer online games. In: ACM SIGMOD conf2008, pp 1234–1238
14. Zhang K, Kemme B (2011) Transaction models for massivelymultiplayer online games. In: 30th IEEE symposium on reliabledistributed systems (SRDS 2011), pp 31–40
15. Newzoo (2011). MMO trend report, http://www.newzoo.com/trend-reports/mmo-trend-report/. Retrieved 2013-06-17. 2011
16. Newzoo (2012) MMO games market report. http://www.newzoo.com/infographics/the-global-mmo-market-sizing-and-seizing-opportunities/. Retrieved 2013-06-17. 2012
17. Oracle (2013) MySQL cluster overview. http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-overview.html. Retrieved 2013-06-17
18. Apache C (2013). Retrieved 2013-06-17. http://cassandra.apache.org/
19. Blackman T, Waldo J (2009) Scalable data storage in project dark-star. Technical report. Retrieved 2013-06-17
20. Burns B (2007) Darkstar: the Java game server, pp 5–6. O’Reilly,Sebastopol
21. Mohammad S, Schallehn E, Breß S (2013) Clustering the cloud:a model for (self-)tuning of cloud data management systems. In:Proc of the 3rd international conf on cloud computing and servicesscience (CLOSER)
22. Dory T, Mejías B, Van Roy P (2011) Comparative elasticity andscalability measurements of cloud databases. In: Proc of the 2ndACM symposium on cloud computing (SoCC), vol 11
188 Datenbank Spektrum (2013) 13:179–188
23. DBMS2 (2007) The database technology of guild wars.http://www.dbms2.com/2007/06/09/the-database-technology-of-guild-wars/. Retrieved 2013-08-02
24. DBMS2 (2009) MMO games are still screwed up in their databasetechnology. http://www.dbms2.com/2009/06/14/mmo-rpggames-database-technology/. Retrieved 2013-08-02
25. Chen K-T, Huang P, Lei C-L (2006) Game traffic analysis: anMMORPG perspective. Comput Netw 50:3002–3023
26. Microsoft, Maximum capacity specifications for SQL server 2005.http://msdn.microsoft.com/en-us/library/ms143432(v=sql.90).aspx,Retrieved 2013-08-02 (2007)
27. Oracle (2013) Logical database limits. http://docs.oracle.com/cd/B19306_01/server.102/b14237/limits003.htm. Retrieved 2013-08-02
28. Aparche (2013) CassandraLimitations, http://wiki.apache.org/cassandra/CassandraLimitations. Retrieved 2013-08-02
