© 2019 SPLUNK INC.

© 2019 SPLUNK INC.

Winning in StarCraft 2 An Analysis of Skill Using SPL and MLTK

Shawn Routhier & Brandon GanemOct. 23, 2019

During the course of this presentation, we may make forward‐looking statements regarding future events or plans of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results may differ materially. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, it may not contain current or accurate information. We do not assume any obligation to update any forward‐looking statements made herein.

In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionalities described or to include any such feature or functionality in a future release.

Splunk, Splunk>, Turn Data Into Doing, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved.

Forward-LookingStatements

© 2019 SPLUNK INC.

© 2019 SPLUNK INC.

Winning in StarCraft 2

Sr Security Engineer | Blizzard EntertainmentShawn Routhier

Security Engineer III | GitHubBrandon Ganem

Use this if there will be two speakers for your session.

© 2019 SPLUNK INC.

Shawn Routhier

Blizzard Entertainment for 4 years• Sr. Security Engineer

Defcon25 Black Badge (Uber) Winner – Telephreak

NPC In World of Warcraft• “Shawn” in Nagrand (Outland)

Security Guy for 10+ years… and counting…• Booz Allen Hamilton, MIT – Lincoln Laboratory, Blizzard Entertainment

B.S. in Network Security (2009)

For fun I Splunk the WoW Economy

I play Protoss

© 2019 SPLUNK INC.

Brandon GanemGitHub• Security Engineer III• https://github.com/brandonganem

Splunk Trust from 2017 to present

Splunking since v4.1 (2011)• MIT – Lincoln Laboratory, Aplura, Dell, GitHub

Zerg since 1999• Highest League - Master

B.S. Computer Science – Wentworth Institute of Technology

beatus on answers/slack

I play Zerg

© 2019 SPLUNK INC.

What is StarCraft?And Why We Chose It For A Talk!

© 2019 SPLUNK INC.

StarCraftStarCraft & Brood War (1998); SC:Remastered (2017)

StarCraft 2• Wings of Liberty (2010), Heart of the Swarm (2013), Legacy of the Void (2015)

Real-Time Strategy (RTS)• Player input is read and actioned immediately (real-time)• Micro: Ability to control units effectively to reduce waste• Macro: Ability to gather resources and maintain production of units, buildings, and upgrades

Three opposing races:• Terran: Future human descendants from a resource-depleted Earth • Protoss: Sapient humanoid species of advanced technology and psionic ability• Zerg: Xenomorphic superorganisms assimilating other races genetics to their own

The 26th century galactic struggle for dominance in a distant part of the Milky Way

© 2019 SPLUNK INC.

© 2019 SPLUNK INC.

How Do You Play?Objective• Kill the opponent’s units

before they kill yours!

How?• Gather resources• Maintain an economy• Build an army• Crush your foe!

Explained in 20 seconds or less!

© 2019 SPLUNK INC.

How StarCraft Ranks Players

Player Ranking System – “Leagues”• Progress within a League by winning• Leagues progress from Bronze to Grandmaster

Performance measured by Match Making Rating (MMR)• MMR’s goal is to match ranked players for a 50% win ratio

When I’m Grand Master, I will play faster…

200

4%

4%

23%

23%

23%

23%

Target Players per League per Region

© 2019 SPLUNK INC.

Data IngestTurning Binary Data Into A Splunk Parsable Structure

© 2019 SPLUNK INC.

Ingesting aBinary FileIt’s as easy as “NO_BINARY_CHECK = True”!

.SC2REPLAY• Proprietary binary format

How do you Splunk a binary file?

© 2019 SPLUNK INC.

Extracting Data From .SC2REPLAY

We stand on the shoulders of the community.

Blizzard Supported API for communicating with the game client: https://github.com/Blizzard/s2client-proto• Nice, but we don’t need the full client

Blizzard Supported SDK for the s2protocol: https://github.com/Blizzard/s2protocol• This does a ton of the legwork for parsing the binary data structures.

GGGTracker: https://github.com/ccbrown/gggtracker• Community supported replay parsing site: https://gggtracker.com/ • Lots of great examples on how to use s2protocol && MIT Licensed!

Not really as easy as “NO_BINARY_CHECK = True”… Sorry

© 2019 SPLUNK INC.

Wrote Some Code

~1300 lines of python• Again, standing on the shoulders of giants• Incredible wealth of data within the replay files

Compressed (.SC2Replay) == ~50KB

After “uncompressing” == ~60MB

Bridging gaps and exporting in JSON

© 2019 SPLUNK INC.

Some Code & Data ChallengesRecursion is fun … Until it’s infinite• When a Unit data type is found, the get_unit function is called• This works great, until a Unit is killed by a Unit…. And so on.

Complex data structure• Feature creep • We kept finding interesting pieces of data to parse out of the replay.

Player ID weirdness• An “event” can be owned by somebody other than the player it references.• Comes into play during matches with observers

_time based on date of the replay• Had to calculate the in-game time based on 16 frames per second

…Each causing a refactor of code and rebuild of our dataset

© 2019 SPLUNK INC.

How Did We Collect Game Replays

Tournament replay packs released by Blizzard and curated by the community– https://lotv.spawningtool.com/replaypacks/

User submitted replays– https://lotv.spawningtool.com/replays

Hint: the community

© 2019 SPLUNK INC.

Player Distribution Sample by LeaguePrivate indicates a professional event 1v1 lobby

© 2019 SPLUNK INC.

Challenges With Sample Size // Bias

Original sample included several 2v2, 3v3, 4v4 matches• Identified and “| delete” those from Splunk

Difficulty finding bulk sources of replays for anything except pro matches

Difficulty finding Korean Professional replays• Local Tournaments are not usually publicly shared

There may be some player trends we miss in the pro scene– Lack of representation of professional Terran players– Qualifiers WCS EU (2019) had: 7 Terran,12 Protoss, & 13 Zerg

Samples show bias against slow drivers in the left lane

© 2019 SPLUNK INC.

AttributesUsable Metrics We Pulled From The Replays

© 2019 SPLUNK INC.

Key Datapoints (Observations)Player Stats• Actuations Per Minute (APM), Effective Actions Per Minute (EAPM)• Macro actions, Micro actions• Camera actions*, Control Groups• League Rank

Base Metrics• Worker Saturation, Gas Timing, Idle Workers, Supply/Resource Blockage & Resolution• Expansion Rate

Match Progress• Resource Accrual & Expenditure, Military Value, Units & Structures Killed/Lost

Match Outcome (Victory/Loss)

Observers cost 25 minerals & 75 gas

© 2019 SPLUNK INC.

Selecting Appropriate Attributes

Two key event_type:• ReplayMeta (overall match)• PlayerStatsEvent (frame by frame stats)

Resource current/used/killed

Food made/used; “Supply”

Friendly Fire

This is where mastery of the subject matter is important

© 2019 SPLUNK INC.

Testing Attributes

Tests numerical fields against “classfield”

Co-occurrence• The ratio of results with field present

Accuracy• Ratio of accurate predictions against

classfield

| analyzefields is OP; needs a nerf next patch

© 2019 SPLUNK INC.

Scoring The Model

Ensure a consistent testing methodology• | sample with a consistent seed

| multireport provided by MLTK GUI• “Open in Search”

Scoring measurements

Seed=1337 was essential to our success

© 2019 SPLUNK INC.

We Needed to Scale Our SPLOur SPL was getting out of hand• Most lines in one search was 624• Took 20+ minutes to modify when we needed new data

We created a few macros!• Using “| appendcols” to include new features to the dataset

Updates took as long as a Bronze vs Bronze match

© 2019 SPLUNK INC.

pstats_delta_minute($field$)More macro than in a Bronze match

© 2019 SPLUNK INC.

Supply BlockEveryone gets supply blocked at 14/15 supply

Excluding first 1.5 minutes of a match

YOU MUST CONSTRUCT ADDITIONAL PYLONS!

© 2019 SPLUNK INC.

Resource Collection & Resource PoolAlso captures expansion-rate and resource saturation

0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00

0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00

© 2019 SPLUNK INC.

SPL SidebarSC2 engine runs at 16 fps (tick-rate)• PlayerStatsEvent is summarized every 10 seconds

Artificially normalize time across all events and retain game-state timing:• | eval _time = now() + (10 * floor(frame / 160))

Data normalization is extremely important!

© 2019 SPLUNK INC.

Current Army Value

8.5% win-rate if at a negative army value delta all game

Total Minerals & Vespene cost of your army hivemind

© 2019 SPLUNK INC.

Calculated Effective MicroMicro: Ability to control units effectively to reduce waste

event_types Used:• BasicCommandEvent, TargetPointCommandEvent, TargetUnitCommandEvent, DataCommandEvent

Yin

© 2019 SPLUNK INC.

Calculated Effective MacroMacro: Ability to gather resources and maintain production of units, buildings, and upgrades

event_types Used:• UnitBornEvent, UnitInitEvent, UpgradeCompleteEvent

Yang

© 2019 SPLUNK INC.

Calculated Camera Movement• SC2 maps are played in an x,y coordinate plane• Distance was calculated with streamstats & eval:

An interesting but less than useful attribute

0:00 0:20 0:40 1:00 1:20 1:40 2:00 2:20 2:40 3:00 3:20

© 2019 SPLUNK INC.

Actuations Per Minute (APM)

Highest correlation to Winrate• | analyzefields @ 0.642

APM can be inflated

A high APM is the result of skill, not an indicator of it

Gotta go fast!

© 2019 SPLUNK INC.

Inflated APM?Kermit == Bronze 600 APM player

© 2019 SPLUNK INC.

APM by LeagueNormal distribution across leagues

© 2019 SPLUNK INC.

“It is rarely a mysterious technique that drives us to the top, but rather a profound mastery of what may well be a basic skill set.”Josh Witzkin, The Art of Learning (ISBN: 0743277457)

© 2019 SPLUNK INC.

Predicting A WinnerThe Effectiveness Of Our Selected Attributes

© 2019 SPLUNK INC.

Our Findings

A Skilled Player Will:•Resolve supply blocks quickly•Tend to both economy and army•Utilize the map•Know when to commit to a fight•Naturally have a higher APM

Backed by DATA!

Joona “Serral” Sotala2018 WCS Champion

© 2019 SPLUNK INC.

Evaluating Our Predictive CapabilityCan we see the future?

© 2019 SPLUNK INC.

Predicting A Winner As Early As PossibleWe can kind of see the future!

| fit GrandientBoosingClassifier “winner” from 1m* …

© 2019 SPLUNK INC.

Fun SlidesShawn vs Brandon5 Matches

© 2019 SPLUNK INC.

© 2019 SPLUNK INC.

The Ultimate Show Matches…of ultimate destiny!

© 2019 SPLUNK INC.

Q&AShawn Routhier | ProtossBrandon Ganem | Zerg

RATE THIS SESSIONGo to the .conf19 mobile app to

© 2019 SPLUNK INC.

You!

Thank