Designing the Call of Cthulhu app with

Google App EnginePresented by Chris Bunch at UCSB

CS 189A / 172 : Capstone Project / Software EngineeringFebruary 10, 2010

http://cs.ucsb.edu/~cgb

Motivation

•I am what you would call a “nerd”

•Plays nerdy board games

•Would like to upload stats from games and analyze them

•Could do other cool things as well

Specifically

Arkham Horror

•Lots of variables that can be analyzed

•Co-op game: all v. the board

•Mix and match expansions

•Many different winning / losing conditions

•But fundamentally boils down to:

•Save the world from total destruction

App Requirements

•Must be able to upload game data

•Should be able to predict a win or loss given some preliminary info

•Can experiment with prediction algorithms

•Should be able to summarize game data

App Requirements•Allows users to log in to upload their

data

•Allows users to share data with each other

•Handles users in a sane fashion

•Harder than it sounds

•Display a top score list for users to view

•Game provides scoring metric

Enter the Cloud

•Why go the cloud route?

•Simple answer: You likely don’t have your own hardware to host this app on, so just use somebody else’s

•But cloud computing is confusing and not well defined yet, so...

Defining Cloud Computing

•Three layers, each building on the last

• Infrastructure:

•Programmer gets a virtual machine and / or reliable storage

•Most control of environment

•Must scale manually

•Amazon EC2, Eucalyptus

Defining Cloud Computing

•Three layers, each building on the last

•Platform:

•Programmer gets a scalable API

•Less control of environment

•Scaling done automatically

•Google App Engine, Microsoft Azure

Defining Cloud Computing

•Three layers, each building on the last

•Software:

•Programmer gets to “skin” an existing app

•No control of environment

•Scaling done automatically

•SalesForce, GMail

My Choice•Clearly I want the platform!

•Platform:

•Scalable API = great!

•Less control of environment = ok

•Programability depends on APIs offered

•Have personal exp. w/ Google App Engine

What is Google App Engine?

•Scalable web framework first offered in 2008

•Users program apps in:

•Python

•Java

•Unofficially: JRuby, Groovy, JavaScript

Sandboxed Environment

•Requests limited to 30 seconds

•Only white-listed libraries can be used

•Anything non-trivial is quota-ed

•No reading / writing to the filesystem

•Web communication over HTTP(S) only

Datastore API

•A giant hash table

•Primitives:

•Get, put, delete - same as regular HT

•Query, count - new operations

•Transactions supported (row-level only)

Datastore API

•GQL queries offered - subset of SQL Select statement, no JOINs

•Tip: Queries are the most expensive operation, use sparingly

•If possible, perform it asynchronously

Memcache API

•Equivalent to memcached

•Provides access to low latency LRU cache

•Roughly 100x faster than Datastore

•But unreliable - remember it’s a cache!

•Exposes get, set, delete operations

•Exposes atomic increment / decrement

URL Fetch API•Equivalent to curl

•Used to grab web content from other sites

•Only allows for HTTP(S) data to be read

•Can also be done asynchronously

•Typically grabs images or mashup data

Mail API

•Equivalent to sendmail

•Can send or receive mail asynchronously

•Can also add attachments

•Not all types allowed - no .doc, .xls, or .ppt in Python SDK

XMPP API

•Equivalent to jabber

•Send and receive instant messages to

•appname.appspotchat.com

•App can’t join group chat yet

•Use as a notifier or alternate UI to app

Images API

•Equivalent to python-imaging

•Exposes image rotation, flipping, cropping, and histogram

•Image is stored internally as a blob, then retrieved and manipulated

Users API

•Equivalent to Google Accounts

•Provides for simple authentication

•Allows app to force login or admin on certain pages

•Also exposes URL to optionally login to

Blobstore API

•Allows for large file uploads (<50 MB)

•Users can upload videos, datasets, or large music files for later retrieval

•Upload must be done via form post

Cron API

•Equivalent to cron

•User specifies how often a web page should be accessed (e.g., daily, every Monday)

•Task is then run in the background

•Useful for running expensive data analysis

Task Queue API

•Spawns a new thread in the background

•But still must visit your app

•Configuration file specifies queues

•Useful for short to medium length tasks

To summarize:

•Many, many APIs available for what we need to do!

•Use Datastore to store / retrieve user data

•Use Users to authenticate users

•Gets around a lot of boilerplate authentication code

To summarize:

•Use Mail to e-mail users if their score is beat

•Use XMPP to allow users to add new data

•Use Cron to run compute-intensive code in the background

To summarize:

•Think about paid functionality:

•Maybe use Blobstore to allow users to upload videos explaining gameplay

•Handle greater traffic if site becomes popular

Let’s Get to Business

•Data Modeling

•Upload Stats Page

•Game Stats Page

•Game Prediction Page

Data Modeling•class Game(db.Model):

•player = db.UserProperty()

•goo = db.StringProperty()

•comments = dbStringProperty(multiline=True)

•date = db.DateTimeProperty(auto_now_add=True)

Data Modeling

•num_of_players = db.IntegerProperty()

•won_game = db.BooleanProperty()

•doom_counters = db.IntegerProperty()

•terror_level = db.IntegerProperty()

•expansions_used = dbListProperty(str)

Uploading Stats• class AddGames(webapp.RequestHandler):

• def get(self):

• user = users.get_current_user()

• if user: # means they’re logged in

• # write html form for getting data

• else:

• self.redirect(users.create_login_url(self.request.uri))

Uploading Stats•def post(self):

•new_goo = self.request.get(‘goo’)

•# validate goo, make sure data type is right

•game = Game()

•game.goo = new_goo

•game.put()

Game Stats

•games = db.GqlQuery (“SELECT * FROM Game ORDER BY date desc”)

•for game in games:

•# find the goo w/ highest player win percentage

•# can also just display the info here

Game Prediction

•def get(self):

•# user specifies the game they will play

•def post(self):

•# calculate how likely they are to win

•# right now we only care about the goo

Routing

•application = webapp.WSGIApplication([

•(‘/upload’, AddGames),

•(‘/stats’, GameStats),

•(‘/predict’, PredictGame)])

Thankfully•Most requirements have very little

state shared between URLs (all saved in DB)

•Thus each class is one of two forms:

•get / post: Grab data from user and do something with it

•get: Just display or synthesize some data

Rendering Web Pages

•Two main options available:

•self.response.out.write(“Hello World!”)

•self.response.out.write(template.render(‘index.html’, {‘my_title: ‘baz’, ‘data’: some_variable}))

•Use the second for any non-trivial app

Rendering Web Pages

•Now in your index.html:

•<title> {{ my_title }} </title>

•<div> {{ data }} </div>

Tips for Your Programs

•Separate out your code into files as needed

•app.yaml specifies this:

•handlers:

• - url: /one

• script: one.py

• - url: /two/(.*)/(.*)

• script: /two/\2/\1.py

Tips for Your Programs

•Make pages with sensitive data ‘secure’

•All access done over HTTPS

•Optional - HTTP and HTTPS work

•Never - HTTPS redirects to HTTP

•Always - HTTP redirects to HTTPS

Tips for Your Programs

•Specify static files in app.yaml

•- url: /images

• static_dir: static/images

•Can also tag ‘secure’ if desired

Tips for Your Programs

•Be creative!

•The webapp framework supports:

•get, post, head, options, put, delete, trace

•Map user pages to ~name?

Summary

•Google App Engine is more than flexible enough to handle your web app desires

•And it’s only getting better!

•More than enough possibilities to create a web app: you just need a good idea!