83
Building Customer Support and Loyalty Jeremy Ehrhardt @JeremyEhrhardt Lead Tools Engineer, [email protected]

GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Embed Size (px)

DESCRIPTION

KIXEYE relies on its players coming back day after day, week after week and, remarkably in the social space, year after year. To maintain this kind of loyalty among our player base, CS agents needed the power to understand an individual player's gameplay experience, see their interactions with other players, and fix anything that gets in the way of players having fun. Jeremy will present postmortems from a customer support software perspective for three of KIXEYE's games: Battle Pirates, War Commander and VEGA Conflict. Our customer support software is called Monocle, and you'll learn about the architecture, our agile approach to rapid development, reliability lessons we've learned in production, and how you can build your own Monocle using free, open source components. Attendees will learn how and why KIXEYE built its customer support system as a web app, the challenges and rewards of building a uniform support API across multiple games, and how designing games with support scenarios in mind yields benefits across the whole product. This session is intended for tech leads, producers, product people and customer support/advocacy managers who are designing and supporting online games. Improve monetization and retention by keeping your players happy. Some knowledge of web application architecture and social gaming will be helpful.

Citation preview

Page 1: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Building Customer Support and Loyalty

Jeremy Ehrhardt @JeremyEhrhardt

Lead Tools Engineer, 📨 [email protected]

Page 2: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

●Real gameplay for social platforms●Deep, engaging games●Rapid iteration with big changes●Long relationship with players

Page 3: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

●No hard paywalls●Regular game events●Strong social competition

● “We monetize revenge.” — Will Harbin, CEO

Page 4: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

●20x higher ARPDAU ($0.80) than other popular social games ($0.04)●30 min+ gameplay sessions, 90+ per day●KIXEYE has been profitable for 4 years

Page 5: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Our games

Page 6: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates

Page 7: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates

Page 8: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates

Page 9: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates

●MMORTS with tower defense elements●KIXEYE.com, Facebook●3 years since launch●Real-time PvP, PvE combat●Immense world map●Extensively customizable bases and fleets

Page 10: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

War Commander

Page 11: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

War Commander

Page 12: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

War Commander

Page 13: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

War Commander

●MMORTS●KIXEYE.com, Facebook●Nearly 3 years since launch●Real-time PvP, PvE combat●Tactical control of dozens of units●Strategic gameplay on the world map

Page 14: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Vega Conflict

Page 15: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Vega Conflict

Page 16: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Vega Conflict

Page 17: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Vega Conflict

Page 18: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Vega Conflict

●MMORTS●KIXEYE.com, Facebook●1 year since launch●Real-time PvP, PvE combat●Strategic view of entire planetary systems●Server-side logic

Page 19: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Backyard Monsters: Unleashed

Page 20: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Backyard Monsters: Unleashed

Page 21: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Backyard Monsters: Unleashed

●MMO tower defense strategy●KIXEYE’s first mobile game●iOS (iPad, iPhone, iPod Touch)●Launched Q4 2013●4th year of the Backyard Monsters series

Page 22: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Support and retention●Easier to keep good players than get new ones●Customer support (CS): last line of defense

● About 60 CS agents at KIXEYE

●Players who write tickets care

Page 23: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Support and retention

“Please look at my payment history and see that I am a serious player. … I do not want to quit this game. I love it and the friends I have made. Please fix the problems.”

“I tried all of your suggestions and am still not able to play my most favorite game, any other help you can give please send it.”

Page 24: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Support and retention

●Ticket volume for 2013:● 1,093,347 total● 337,763 from whales

●Ticket volume so far, 2014 Q1:● 238,397 total● 66,336 from whales

Page 25: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle

●One-stop tool for changing game data●Provides services on top of games●Player-centric interface●Loosely linked to CS ticketing system●Completely separate from game code

Page 26: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle

●Holy Trinity of CS functions:● Search for a player● Grant hard currency● Ban a player

Page 27: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle

●Holy Trinity of CS functions:● Search for a player● Grant hard currency● Ban a player

Page 28: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: player search

Page 29: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: player search

Page 30: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: player details

Page 31: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: player details

Page 32: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: player details

Page 33: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: player details

Page 34: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: player details

Page 35: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle

●Holy Trinity of CS functions:● Search for a player● Grant hard currency● Ban a player

Page 36: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: gold

Page 37: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: gold

Page 38: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle

●Holy Trinity of CS functions:● Search for a player● Grant hard currency● Ban a player

Page 39: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: bans

Page 40: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: admin log

Page 41: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Supportedbrowsers

Monocle backend servers

Monocle frontend

Internal HTTPS API

Users😇 in-house CS😀 external CS😎 developers

Public HTTPS API

Users 👮 Fair Play

😃 analysts

Automated tools from other KIXEYE

departments

👮 Fair Play😏 designers

😬 producers😆 QA

😎 developers😱 automated QA

Monocle architecture

Page 42: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Supportedbrowsers

Monocle backend servers

Monocle frontend

Internal HTTPS API

Users😇 in-house CS😀 external CS😎 developers

Public HTTPS API

Users 👮 Fair Play

😃 analysts

Automated tools from other KIXEYE

departments

👮 Fair Play😏 designers

😬 producers😆 QA

😎 developers😱 automated QA

Monocle architecture

Page 43: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Supportedbrowsers

Monocle backend servers

Monocle frontend

Internal HTTPS API

Users😇 in-house CS😀 external CS😎 developers

Public HTTPS API

Users 👮 Fair Play

😃 analysts

Automated tools from other KIXEYE

departments

👮 Fair Play😏 designers

😬 producers😆 QA

😎 developers😱 automated QA

Monocle architecture

Page 44: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Supportedbrowsers

Monocle backend servers

Monocle frontend

Internal HTTPS API

Users😇 in-house CS😀 external CS😎 developers

Public HTTPS API

Users 👮 Fair Play

😃 analysts

Automated tools from other KIXEYE

departments

👮 Fair Play😏 designers

😬 producers😆 QA

😎 developers😱 automated QA

Monocle architecture

Page 45: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle frontend

●Open source tech● AngularJS● Bootstrap● Lo-Dash

Bootstrap

Page 46: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle frontend

●Open source tech● AngularJS● Bootstrap● Lo-Dash

Bootstrap

Page 47: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Lessons Learned

●Why go with a webapp?● Cross-platform● Easy to publish● Easy to update

●Stick to Chrome and Firefox● Others lag in features, standards, tools

Page 48: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle backend

●Gateway between frontend and games●Storage for logs about players●Arbiter of user permissions

Page 49: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle backend

●Open source tech● Linux (Ubuntu LTS)● Apache● MySQL● Python with Flask

Page 50: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle backend

●Open source tech● Linux (Ubuntu LTS)● Apache● MySQL● Python with Flask

Page 51: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle architecture

Monocle backend servers

LDAP

WC Bases…

WC Primary

BP Bases…

BP Primary

Direct MySQL DB access for first-gen games

VC BYM:U

Monorail HTTP APIs for second-gen games

User permissions

Admin logsBan queue

Internal MySQL DB

Page 52: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Lessons Learned

●Direct DB access is bad● Violates information hiding principle● Game teams need to iterate quickly● Multiple tech stacks at KIXEYE

Page 53: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monorail

Page 54: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monorail

●Specified APIs are good● Contract between Monocle and game● Purpose-built for non-game use● Testable from both sides

Page 55: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monorail

●Specified APIs are good● Contract between Monocle and game● Purpose-built for non-game use● Testable from both sides

Page 56: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monorail

●RESTful HTTP API● JSON wire format● Mandatory TLS encryption● HTTP error codes● HTTP basic auth

Page 57: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monorail

●Standard data model across games● Players● Gold● Bans● Bases● Resources● …

Page 58: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: resource editor

Page 59: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Monocle: resource editor

Page 60: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Lessons Learned

●Use strings from games● “Siege Mortar D75-B” instead of “1299000”● “Battle Barge 1” instead of “hull 50”

●Use art assets too!

Page 61: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Lessons Learned

●Use strings from games● “Siege Mortar D75-B” instead of “1299000”● “Battle Barge 1” instead of “hull 50”

●Use art assets too!

Page 62: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates inventory: in-game

Page 63: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates inventory: then

Page 64: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates inventory: now

Page 65: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates inventory: now

Page 66: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates inventory: now

Page 67: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Battle Pirates inventory: now

Page 68: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Lessons Learned

●Explore player connections● Battles● Alliances● Friends● IP addresses● Gifts

Page 69: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Player connections: battles

Page 70: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Player connections: battles

Page 71: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Player connections: battles

Page 72: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Player connections: battles

Page 73: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Player connections: IPs

Page 74: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Player connections: IPs

Page 75: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Player connections: IPs

Page 76: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Player connections: IPs

Page 77: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Lessons Learned

●Get support hooks (Monorail) into games as early as possible

Page 78: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Backyard Monsters: Unleashed

Page 79: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Backyard Monsters: Unleashed

Page 80: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Backyard Monsters: Unleashed

Page 81: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Support and retention (reprise)

“The problem was fixed the next day.”

“you helped getting my evalks working again”

“can you get more people like Cassi to work for ya she helped me out nice with my issue i had with a shockwave retrofit”

“keep up the good work for your games!”

Page 82: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Support and retention (reprise)

“Hey I just want to tell you Thank You for the 50 gold, little things like that mean a lot to the people who play the game and we do notice the effort you have all been doing to improve the game, also I would like to compliment you on your quick responses and fairness when issues do occur. You have always treated me fairly and I appreciate it.”

Page 83: GDC 2014 - Jeremy Ehrhardt, KIXEYE - Building Customer Support and Loyalty

Support and retention (reprise)

“I have been playing this game for a long time now and a couple of things have happened recently that I had to submit tickets for.

I will put my hands up, in the past I have been critical of and frustrated with Kixeye support. However, on these two occasions Kixeye dealt with both matters quickly and resolved both issues to my satisfaction.

So instead of the usual threads of whingeing, complaints and moaning, I would like to say thank you Kixeye - on both occasions job well done”