Upload
chris-bunch
View
1.333
Download
1
Embed Size (px)
DESCRIPTION
These are my slides from my talk at LA.rb, covering research at UCSB on the AppScale project. This is a condensed version of the talk I gave at SBonRails - see that talk for about twice as much material on these topics.
Citation preview
The AppScale ProjectPresented by Chris Bunch
(on behalf of the AppScale team)April 14, 2011 @ LA.rb
Overview
• Google App Engine
• AppScale - now with 50% Ruby!
• Neptune - A Ruby DSL for the cloud
Google App Engine
• A web framework introduced in 2008
• Python and Java supported
• Offers a Platform-as-a-Service: Use Google’s APIs to achieve scale
• Upload your app to Google
Quotas
Data Model
• Not relational - semi-structured schema
• Compared to models in Rails
• Exposes a get / put / delete / query interface
Storing Data
• Datastore API - Persistent storage
• Memcache API - Transient storage
• User can set expiration times
• Blobstore API - Store large files
• need to enable billing to use it
Be Social!
• Mail API - Send and receive e-mail
• XMPP API - Send and receive IMs
• Channel API - Creating persistent connections via XMPP
• Use for chat rooms, games, etc.
Background Tasks
• Cron API - Access a URL periodically
• Descriptive language: “every 5 minutes”, “every 1st Sun of Jan, Mar, Dec”, etc.
• Uses a separate cron.yaml file
• TaskQueue API - Within your app, fire off tasks to be done later
Dealing with Users
• Users API: Uses Google Accounts
• Don’t write that ‘forgot password’ page ever again!
• Authorization: via app.yaml:
• anyone, must login, or admin only
When Services Fail
• Originally: failures throw exceptions
• Just catch them all!
• Capabilities API: Check if a service is available
• Datastore, Memcache, and so on
Deploying Your App
• Develop locally on SDK
• Stub implementations of most APIs
• Then deploy to Google
How to Scale
• Limitations on the programming model:
• No filesystem interaction
• 30 second limit per web request
• Language libraries must be on whitelist
• Sandboxed execution
Enter AppScale
• App Engine is easy to use
• but we really want to tinker with the internals!
• Need an open platform to experiment on
• test API implementations
• add new APIs
Enter AppScale
• Lots of NoSQL DBs out there
• Hard to compare DBs
• Configuration and deployment can be complex
• Need one-button deployment
Deploying Your App
• run-instances: Start AppScale
• describe-instances: View cloud metadata
• upload-app: Deploy an App Engine app
• remove-app: Un-deploy an App Engine app
• terminate-instances: Stop AppScale
Deployment Models
• Cloud deployment: Amazon EC2 or Eucalyptus (the open source implementation of the EC2 APIs)
• Just specify how many machines you need
• Non-cloud deployment via Xen or KVM
AppController
• The brains of the outfit
• Runs on every node
• Handles configuration and deployment of all services (including other AppControllers)
• Written in Ruby
Load balancer
• Routes users to their app via nginx
• haproxy makes sure app servers are live
• Can’t assume the user has DNS:
• Thus we wrote the AppLoadBalancer
• Rails app that routes users to apps
• Performs authentication as well
App Server
• We modified the App Engine SDK
• Easier for Python (source included)
• Harder for Java (had to decompile)
• Removed non-scalable API implementations
• Goal: Use open source whenever possible
A Common Feature Request
Database Options
• Open source / open APIs / proprietary
• Master / slave v. peer-to-peer
• Differences in query languages
• Data model (key/val, semi-structured)
• In-memory or persistent
• Data consistency model
• Interfaces - REST / Thrift / libraries
Neptune
• Need a simple way to run compute-intensive jobs
• We have the code from the ‘net
• We have the resources - the cloud
• But the average user does not have the know how
• Our solution: create a domain specific language for configuring cloud apps
• Based on Ruby
Syntax
• It’s as easy as:
neptune :type => “mpi”,
:code => “MpiNQueens”,
:nodes_to_use => 8,
:output => “/mpi/output-1.txt”
Neptune Supports:
• Message Passing Interface (MPI)
• MapReduce
• Unified Parallel C (UPC)
• X10
• Erlang
Extensibility
• Experts can add support for other computational jobs
• Biochemists can run simulations via DFSP and dwSSA
• Embarassingly parallel Monte Carlo simulations
Wrapping It Up
• Thanks to the AppScale team, especially:
• Co-lead Navraj Chohan and advisor Chandra Krintz
• Check us out on the web:
• http://appscale.cs.ucsb.edu
• http://neptune-lang.org