Embed Size (px)
DESCRIPTION
Talk by Pamela Fox at YOW 2010 about App Engine, the datastore, and how to implement common features for your webapp.
Citation preview
WRITING APPS THE GOOGLE-Y WAY
Pamela Fox, YOW! Australia 2010
Who am I?
USC Google
Amazon, Flickr, Maps
Google Maps API Google Wave API
Spreadsheets, Blogger, Youtube, Picasa, Gadgets, App Engine
Who am I?
Java pYthon
What is App Engine?
“Google App Engine enables you to build and host web apps on the same systems that power Google applications.”
http://code.google.com/appengine
What is a “web app”?
Static vs. Dynamic
Anonymous vs. Users
Intranet vs. Internet
~2 billionHundreds - Thousands
What is a “web app”?
Some Google web apps
Some Google App Engine web apps
www.gifttag.comwww.buddypoke.com
Google apps on App Engine
panoramio.com pubsubhubbub.appspot.com
How does App Engine work?
1. You upload application code & resources to Google.
2. Google serves your application from scalable infrastructure.
3. You pay for only the resources that Google used in serving the application.
Example app: YOW!*
App Engine architecture
user or task
App Engine architecture
App Engine architecture
LIMIT!
The tricky bits
LIMIT!
Datastore
Entity
PropertiesKey
Entity
Entity
Entity
Entity
Path Kind Name/ID
Example: Speaker Entities
Key Path
Kind ID First Name
Last Name
Speaker1
- Speaker 1 Rod Johnson
Key Path
Kind ID First Name
Last Name
Middle Name Suffix
Speaker1 - Speaker
2 Guy Steele L Jr.
Modeling Speaker Entities
class Speaker(db.model):
firstname = db.StringProperty(required=True)
lastname = db.StringProperty(required=True)
middlename = db.StringProperty()
namesuffix = db.StringProperty()
website = db.StringProperty()
keynote = db.BooleanProperty(default=False)
Saving Speaker Entities
ron = Speaker(firstname="Ron", lastname="Johnson")
guy = Speaker(firstname="Guy", lastname="Steele",
middlename="L", namesuffix="Jr.")
ron.put()
guy.put()
Updating Speaker Entities
ron = Speaker.get_by_id(1)
guy = Speaker.get_by_id(2)
ron.website = "http://www.ronjohnson.com"
ron.keynote = True
guy.website = "http://www.guysteele.com"
guy.keynote = True
db.put(ron, guy)
LIMIT!
How Updates Happen
commitjournal apply entities
apply indexes
A B
Queries & Indexes
Query Index
Index
Index
Index
Query
Query
Query
Query
Query
Queries & Indexes
SELECT * from Speaker ORDER BY lastname
key lastname
Speaker3 Fox
Speaker4 Hohpe
Speaker1 Johnson
Speaker2 Steele
LIMIT!
Queries & Indexes
SELECT * from Speaker ORDER by middlename
key middlename
Speaker2 L
Queries & Indexes
SELECT * from Speaker WHERE keynote = True
key keynote
Speaker1 True
Speaker2 True
Speaker3 False
Speaker4 False
Queries & Indexes
SELECT * from Speaker WHERE keynote = False
key keynote
Speaker1 True
Speaker2 True
Speaker3 False
Speaker4 False
Queries
allspeakers = Speaker.all().order('lastname)
for speaker in allspeakers:
print speaker.firstname + '' + speaker.lastname + '' + speaker.website
keynotespeakers = Speaker.all().filter('keynote = ', True)
notspecialspeakers = Speaker.all().filter('keynote = ', False)
LIMIT!
Custom Indexes
SELECT * from Speaker ORDER BY lastname, keynote
key lastname keynote
Speaker3 Fox false
Speaker4 Hohpe false
Speaker1 Johnson true
Speaker2 Steele true
speakers = Speaker.all().order('lastname')
.order('keynote')
Custom Indexes
SELECT * from Speaker WHERE lastname > 'Johnson' and keynote = true
key lastname keynote
Speaker3 Fox false
Speaker4 Hohpe false
Speaker1 Johnson true
Speaker2 Steele true
speakers = Speaker.all().order('lastname')
.filter('keynote =', True)
Impossible Indexes
SELECT * from Speaker WHERE lastname < 'Steele' and firstname > 'Gregory'
key lastname firstname
Speaker3 Fox Pamela
Speaker4 Hohpe Gregory
Speaker1 Johnson Ron
Speaker2 Steele Guy
...not in subsequent rows!
Impossible Indexes
SELECT * from Speaker WHERE lastname > 'Fox' ORDER BY firstname
key lastname firstname
Speaker3 Fox Pamela
Speaker4 Hohpe Gregory
Speaker1 Johnson Ron
Speaker2 Steele Guy
...not in the correct order!
Queries with Offset
SELECT * from Speaker LIMIT 2 OFFSET 2
key lastname
Speaker3 Fox
Speaker4 Hohpe
Speaker1 Johnson
Speaker2 Steele
speakers = Speaker.all().fetch(2, 2)
1
2
...slow! LIMIT!
Queries with Cursors
query = db.Query(Speaker)
speakers = q.fetch(1000)
cursor = q.cursor()
memcache.set('speaker_cursor', cursor)
...
last_cursor = memcache.get('speaker_cursor')
q.with_cursor(last_cursor)
speakers = q.fetch(1000)
More Properties
class Talk(db.Model):
title = db.StringProperty(required=True)
abstract = db.TextProperty(required=True)
speaker = db.ReferenceProperty(Speaker)
tags = db.StringListProperty()
pamela = Speaker.all().filter('firstname = ', 'Pamela').get()
talk = Talk('Writing Apps the Googley Way', 'Bla bla bla',
pamela, ['App Engine', 'Python'])
talk.put()
talk = Talk('Wonders of the Onesie', 'Bluh bluh bluh',
pamela, ['Pajamas', 'Onesies'])
talk.put()
Back-References
pamela = Speaker.all().filter('firstname = ', 'Pamela').get()
for talk in pamela.talk_set:
print talk.title
key speaker
Talk6 Speaker2
Talk1 Speaker3
Talk2 Speaker3
Talk5 Speaker4
SELECT * from Talk WHERE speaker = Speaker3
Searching List Properties
talks = Talk.all().filter('tags = ', 'python').fetch(10)
SELECT * from Talk WHERE tags = 'Python'
key lastname
Talk1 App Engine
Talk2 Pajamas
Talk1 Python
Talk2 Onesies
LIMIT!
Entity Groups
pamela = Speaker.all().filter('firstname = ', 'Pamela').get()
talk1 = Talk('Writing Apps the Googley Way', 'Bla bla bla',
pamela, ['App Engine', 'Python'],
parent=pamela)
talk2 = Talk('Wonders of the Onesie', 'Bluh bluh bluh',
pamela, ['Pajamas', 'Onesies'],
parent=pamela)
db.put(talk1, talk2)
def update_talks():
talk1.title = 'Writing Apps the Microsoft Way'
talk2.title = 'Wonders of the Windows'
db.put(talk1, talk2)
db.run_in_transaction(update_talks)
Common Features
Counters
1 2 3 4 5people have done something.
RageTube: Global Stats
RageTube: Global Stats
SongStat
yaycountviewcount
title artist
Key
Path KindName(song)
naycount
mehcount
RageTube: Global Stats
viewcount viewcount viewcount
datastore memcache
RageTube: Global Stats
class Song(db.Model): viewcount = db.IntegerProperty(default=0) title = db.StringProperty() artist = db.StringProperty()
def get_viewcount(self): viewcount = self.viewcount cached_viewcount = memcache.get('viewcount-' + self.key().name(), self.key().kind()) if cached_viewcount: viewcount += cached_viewcount return viewcount
@classmethod def flush_viewcount(cls, name): song = cls.get_by_key_name(name) value = memcache.get('viewcount-' + name, cls.kind()) memcache.decr('viewcount-' + name, value, cls.kind()) song.viewcount += value song.put()
@classmethod def incr_viewcount(cls, name, interval=5, value=1): memcache.incr('viewcount-' + name, value, cls.kind()) interval_num = get_interval_number(datetime.now(), interval) task_name = '-'.join([cls.kind(), name.replace(' ', '-'), 'viewcount', str(interval), str(interval_num)]) deferred.defer(cls.flush_viewcount, name, _name=task_name) LIMIT!
Ratings
Rated by 500 users.
App Gallery: Ratings
App Gallery: Ratings
Comment Application
total_ratings
sum_ratings
avg_ratingrated_inde
x
comment_count
rating
App Gallery: Ratings
def UpdateAppCommentData(self, rating, operation): def UpdateCommentData(self, rating, operation): self.comment_count += 1 * operation self.sum_ratings += rating * operation self.total_ratings += 1 * operation self.avg_rating = int(round(self.sum_ratings / self.total_ratings)) self.rated_index = '%d:%d:%d' % (self.avg_rating, self.total_ratings, self.index) self.put()
db.run_in_transaction(UpdateCommentData, self, rating, operation)
app.UpdateAppCommentData(rating, db_models.Comment.ADD)comment = db_models.Comment()comment.application = appcomment.rating = ratingcomment.put()
query.order('-avg_rating').order('-rated_index')
Geospatial Queries
City-Go-Round: Agencies
citygoround.org
https://github.com/walkscore/City-Go-Round
City-Go-Round: Geo Queries
AgencyGeoModel
location (GeoPt)
location_geocells (StringListProper
ty)
City-Go-Round: Geo Queries
def fetch_agencies_near(lat, long, bbox_side_in_miles): query = Agency.all() bbox = bbox_centered_at(lat, long, bbox_side_in_miles) return Agency.bounding_box_fetch(query, bbox, max_results = 50)
def bounding_box_fetch(query, bbox, max_results=1000,): results = [] query_geocells = geocell.best_bbox_search_cells(bbox)
for entity in query.filter('location_geocells IN', query_geocells): if len(results) == max_results: break if (entity.location.lat >= bbox.south and entity.location.lat <= bbox.north and entity.location.lon >= bbox.west and entity.location.lon <= bbox.east): results.append(entity) return results
City-Go-Round: Apps
citygoround.org
City-Go-Round: Geo Queries
TransitAppLocation
GeoModel
location (GeoPt)
location_geocells (StringListProper
ty)
TransitApp
City-Go-Round: Geo Queries
class TransitAppLocation(GeoModel): transit_app = db.ReferenceProperty(TransitApp) def fetch_transit_app_locations_near(lat, longi): query = TransitAppLocation.all() bbox = bbox_centered_at(lat, long, bbox_side_in_miles) return TransitAppLocation.bounding_box_fetch(query, bounding_box, max_results = 500)
def fetch_transit_apps_near(lat, long):
transit_app_locations = TransitAppLocation.fetch_transit_app_locations_near(lat, long)]
transit_apps = [transit_app_location.transit_app for transit_app_location in transit_app_locations] return transit_apps
Full Text Search
pizza Search
ThingyIt's like pizza, but in the cloud.
Other ThingyThis will make you smell as delicious as pizza.
Disclosed.ca: Search
Disclosed.ca: Search
Contract
agency_name vendor_name
description comments
uri
Disclosed.ca: Search
from search.core import SearchIndexProperty, porter_stemmer
class Contract(db.Model): uri = db.StringProperty(required=True) agency_name = db.StringProperty(required=True) vendor_name = db.StringProperty(required=True) description = db.StringProperty() comments = db.TextProperty() search_index = SearchIndexProperty(('agency_name', 'vendor_name', 'description', 'comments'), indexer=porter_stemmer)
results = Contract.search_index.search(sheep').fetch(20)
Disclosed.ca: Search
Contract
agency_name vendor_name
description comments
uri
search_index(StringListProper
ty)
SearchIndex
Disclosed.ca: Search
key search_index
ContractSearch1 charter
ContractSearch1 june
ContractSearch1 sheep
ContractSearch2 sheep
ContractSearch1 wood
SELECT FROM ContractSearch WHERE search_index = "sheep"
More Learning
http://ae-book.appspot.com
http://code.google.com/appengine
http://blog.notdot.net/
AppEngine: Now & Later
"Run your web apps on Google's infrastructure.Easy to build, easy to maintain, easy to scale."
Roadmap:•Background processes > 30s•MapReduce•Bulk Import/Export•Channel API
App Engine for Business:•SQL• Other stuff..
Thanks for coming!
*I am never very good at conclusions, so this slide is a subtle notice to all of you that I am done talking now.
