Writing Apps the Google-y Way

WRITING APPS THE GOOGLE-Y WAY

Pamela Fox, YOW! Australia 2010

Who am I?

twitter.com/pamelafox

pamelafox@gmail.com

pamelafox.org

you get the idea...

Who am I?

USC Google

Amazon, Flickr, Maps

Google Maps API Google Wave API

Spreadsheets, Blogger, Youtube, Picasa, Gadgets, App Engine

Who am I?

Java pYthon

What is App Engine?

“Google App Engine enables you to build and host web apps on the same systems that power Google applications.”

http://code.google.com/appengine

What is a “web app”?

Static vs. Dynamic

Anonymous vs. Users

Intranet vs. Internet

~2 billionHundreds - Thousands

What is a “web app”?

Some Google web apps

Some Google App Engine web apps

www.gifttag.comwww.buddypoke.com

Google apps on App Engine

panoramio.com pubsubhubbub.appspot.com

How does App Engine work?

1. You upload application code & resources to Google.

2. Google serves your application from scalable infrastructure.

3. You pay for only the resources that Google used in serving the application.

Example app: YOW!*

App Engine architecture

user or task

App Engine architecture

App Engine architecture

LIMIT!

The tricky bits

LIMIT!

Datastore

Entity

PropertiesKey

Entity

Entity

Entity

Entity

Path Kind Name/ID

Example: Speaker Entities

Key Path

Kind ID First Name

Last Name

Speaker1

- Speaker 1 Rod Johnson

Key Path

Kind ID First Name

Last Name

Middle Name Suffix

Speaker1 - Speaker

2 Guy Steele L Jr.

Modeling Speaker Entities

class Speaker(db.model):

firstname = db.StringProperty(required=True)

lastname = db.StringProperty(required=True)

middlename = db.StringProperty()

namesuffix = db.StringProperty()

website = db.StringProperty()

keynote = db.BooleanProperty(default=False)

Saving Speaker Entities

ron = Speaker(firstname="Ron", lastname="Johnson")

guy = Speaker(firstname="Guy", lastname="Steele",

middlename="L", namesuffix="Jr.")

ron.put()

guy.put()

Updating Speaker Entities

ron = Speaker.get_by_id(1)

guy = Speaker.get_by_id(2)

ron.website = "http://www.ronjohnson.com"

ron.keynote = True

guy.website = "http://www.guysteele.com"

guy.keynote = True

db.put(ron, guy)

LIMIT!

How Updates Happen

commitjournal apply entities

apply indexes

A B

Queries & Indexes

Query Index

Index

Index

Index

Query

Query

Query

Query

Query

Queries & Indexes

SELECT * from Speaker ORDER BY lastname

key lastname

Speaker3 Fox

Speaker4 Hohpe

Speaker1 Johnson

Speaker2 Steele

LIMIT!

Queries & Indexes

SELECT * from Speaker ORDER by middlename

key middlename

Speaker2 L

Queries & Indexes

SELECT * from Speaker WHERE keynote = True

key keynote

Speaker1 True

Speaker2 True

Speaker3 False

Speaker4 False

Queries & Indexes

SELECT * from Speaker WHERE keynote = False

key keynote

Speaker1 True

Speaker2 True

Speaker3 False

Speaker4 False

Queries

allspeakers = Speaker.all().order('lastname)

for speaker in allspeakers:

print speaker.firstname + '' + speaker.lastname + '' + speaker.website

keynotespeakers = Speaker.all().filter('keynote = ', True)

notspecialspeakers = Speaker.all().filter('keynote = ', False)

LIMIT!

Custom Indexes

SELECT * from Speaker ORDER BY lastname, keynote

key lastname keynote

Speaker3 Fox false

Speaker4 Hohpe false

Speaker1 Johnson true

Speaker2 Steele true

speakers = Speaker.all().order('lastname')

.order('keynote')

Custom Indexes

SELECT * from Speaker WHERE lastname > 'Johnson' and keynote = true

key lastname keynote

Speaker3 Fox false

Speaker4 Hohpe false

Speaker1 Johnson true

Speaker2 Steele true

speakers = Speaker.all().order('lastname')

.filter('keynote =', True)

Impossible Indexes

SELECT * from Speaker WHERE lastname < 'Steele' and firstname > 'Gregory'

key lastname firstname

Speaker3 Fox Pamela

Speaker4 Hohpe Gregory

Speaker1 Johnson Ron

Speaker2 Steele Guy

...not in subsequent rows!

Impossible Indexes

SELECT * from Speaker WHERE lastname > 'Fox' ORDER BY firstname

key lastname firstname

Speaker3 Fox Pamela

Speaker4 Hohpe Gregory

Speaker1 Johnson Ron

Speaker2 Steele Guy

...not in the correct order!

Queries with Offset

SELECT * from Speaker LIMIT 2 OFFSET 2

key lastname

Speaker3 Fox

Speaker4 Hohpe

Speaker1 Johnson

Speaker2 Steele

speakers = Speaker.all().fetch(2, 2)

1

2

...slow! LIMIT!

Queries with Cursors

query = db.Query(Speaker)

speakers = q.fetch(1000)

cursor = q.cursor()

memcache.set('speaker_cursor', cursor)

...

last_cursor = memcache.get('speaker_cursor')

q.with_cursor(last_cursor)

speakers = q.fetch(1000)

More Properties

class Talk(db.Model):

title = db.StringProperty(required=True)

abstract = db.TextProperty(required=True)

speaker = db.ReferenceProperty(Speaker)

tags = db.StringListProperty()

pamela = Speaker.all().filter('firstname = ', 'Pamela').get()

talk = Talk('Writing Apps the Googley Way', 'Bla bla bla',

pamela, ['App Engine', 'Python'])

talk.put()

talk = Talk('Wonders of the Onesie', 'Bluh bluh bluh',

pamela, ['Pajamas', 'Onesies'])

talk.put()

Back-References

pamela = Speaker.all().filter('firstname = ', 'Pamela').get()

for talk in pamela.talk_set:

print talk.title

key speaker

Talk6 Speaker2

Talk1 Speaker3

Talk2 Speaker3

Talk5 Speaker4

SELECT * from Talk WHERE speaker = Speaker3

Searching List Properties

talks = Talk.all().filter('tags = ', 'python').fetch(10)

SELECT * from Talk WHERE tags = 'Python'

key lastname

Talk1 App Engine

Talk2 Pajamas

Talk1 Python

Talk2 Onesies

LIMIT!

Entity Groups

pamela = Speaker.all().filter('firstname = ', 'Pamela').get()

talk1 = Talk('Writing Apps the Googley Way', 'Bla bla bla',

pamela, ['App Engine', 'Python'],

parent=pamela)

talk2 = Talk('Wonders of the Onesie', 'Bluh bluh bluh',

pamela, ['Pajamas', 'Onesies'],

parent=pamela)

db.put(talk1, talk2)

def update_talks():

talk1.title = 'Writing Apps the Microsoft Way'

talk2.title = 'Wonders of the Windows'

db.put(talk1, talk2)

db.run_in_transaction(update_talks)

Common Features

Counters

1 2 3 4 5people have done something.

RageTube: Global Stats

RageTube: Global Stats

SongStat

yaycountviewcount

title artist

Key

Path KindName(song)

naycount

mehcount

RageTube: Global Stats

viewcount viewcount viewcount

datastore memcache

RageTube: Global Stats

class Song(db.Model): viewcount = db.IntegerProperty(default=0) title = db.StringProperty() artist = db.StringProperty()

def get_viewcount(self): viewcount = self.viewcount cached_viewcount = memcache.get('viewcount-' + self.key().name(), self.key().kind()) if cached_viewcount: viewcount += cached_viewcount return viewcount

@classmethod def flush_viewcount(cls, name): song = cls.get_by_key_name(name) value = memcache.get('viewcount-' + name, cls.kind()) memcache.decr('viewcount-' + name, value, cls.kind()) song.viewcount += value song.put()

@classmethod def incr_viewcount(cls, name, interval=5, value=1): memcache.incr('viewcount-' + name, value, cls.kind()) interval_num = get_interval_number(datetime.now(), interval) task_name = '-'.join([cls.kind(), name.replace(' ', '-'), 'viewcount', str(interval), str(interval_num)]) deferred.defer(cls.flush_viewcount, name, _name=task_name) LIMIT!

Ratings

Rated by 500 users.

App Gallery: Ratings

App Gallery: Ratings

Comment Application

total_ratings

sum_ratings

avg_ratingrated_inde

x

comment_count

rating

App Gallery: Ratings

def UpdateAppCommentData(self, rating, operation): def UpdateCommentData(self, rating, operation): self.comment_count += 1 * operation self.sum_ratings += rating * operation self.total_ratings += 1 * operation self.avg_rating = int(round(self.sum_ratings / self.total_ratings)) self.rated_index = '%d:%d:%d' % (self.avg_rating, self.total_ratings, self.index) self.put()

db.run_in_transaction(UpdateCommentData, self, rating, operation)

app.UpdateAppCommentData(rating, db_models.Comment.ADD)comment = db_models.Comment()comment.application = appcomment.rating = ratingcomment.put()

query.order('-avg_rating').order('-rated_index')

Geospatial Queries

City-Go-Round: Agencies

citygoround.org

https://github.com/walkscore/City-Go-Round

City-Go-Round: Geo Queries

AgencyGeoModel

location (GeoPt)

location_geocells (StringListProper

ty)

City-Go-Round: Geo Queries

def fetch_agencies_near(lat, long, bbox_side_in_miles): query = Agency.all() bbox = bbox_centered_at(lat, long, bbox_side_in_miles) return Agency.bounding_box_fetch(query, bbox, max_results = 50)

def bounding_box_fetch(query, bbox, max_results=1000,): results = [] query_geocells = geocell.best_bbox_search_cells(bbox)

for entity in query.filter('location_geocells IN', query_geocells): if len(results) == max_results: break if (entity.location.lat >= bbox.south and entity.location.lat <= bbox.north and entity.location.lon >= bbox.west and entity.location.lon <= bbox.east): results.append(entity) return results

City-Go-Round: Apps

citygoround.org

City-Go-Round: Geo Queries

TransitAppLocation

GeoModel

location (GeoPt)

location_geocells (StringListProper

ty)

TransitApp

City-Go-Round: Geo Queries

class TransitAppLocation(GeoModel): transit_app = db.ReferenceProperty(TransitApp) def fetch_transit_app_locations_near(lat, longi): query = TransitAppLocation.all() bbox = bbox_centered_at(lat, long, bbox_side_in_miles) return TransitAppLocation.bounding_box_fetch(query, bounding_box, max_results = 500)

def fetch_transit_apps_near(lat, long):

transit_app_locations = TransitAppLocation.fetch_transit_app_locations_near(lat, long)]

transit_apps = [transit_app_location.transit_app for transit_app_location in transit_app_locations] return transit_apps

Full Text Search

pizza Search

ThingyIt's like pizza, but in the cloud.

Other ThingyThis will make you smell as delicious as pizza.

Disclosed.ca: Search

Disclosed.ca: Search

Contract

agency_name vendor_name

description comments

uri

Disclosed.ca: Search

from search.core import SearchIndexProperty, porter_stemmer

class Contract(db.Model): uri = db.StringProperty(required=True) agency_name = db.StringProperty(required=True) vendor_name = db.StringProperty(required=True) description = db.StringProperty() comments = db.TextProperty() search_index = SearchIndexProperty(('agency_name', 'vendor_name', 'description', 'comments'), indexer=porter_stemmer)

results = Contract.search_index.search(sheep').fetch(20)

Disclosed.ca: Search

Contract

agency_name vendor_name

description comments

uri

search_index(StringListProper

ty)

SearchIndex

Disclosed.ca: Search

key search_index

ContractSearch1 charter

ContractSearch1 june

ContractSearch1 sheep

ContractSearch2 sheep

ContractSearch1 wood

SELECT FROM ContractSearch WHERE search_index = "sheep"

More Learning

http://ae-book.appspot.com

http://code.google.com/appengine

http://blog.notdot.net/

AppEngine: Now & Later

"Run your web apps on Google's infrastructure.Easy to build, easy to maintain, easy to scale."

Roadmap:•Background processes > 30s•MapReduce•Bulk Import/Export•Channel API

App Engine for Business:•SQL• Other stuff..

Thanks for coming!

*I am never very good at conclusions, so this slide is a subtle notice to all of you that I am done talking now.