View
1.269
Download
0
Category
Tags:
Preview:
DESCRIPTION
La computadora HAL 9000 es uno de los personajes centrales de "2001 Odisea del Espacio". Considerando que esta historia fue escrita en 1968, podemos decir que la tecnología ha tenido un tremendo avance desde entonces. Y nuestra habilidad para crear software también ha mejorado un poco. Pero todavía estamos muy lejos de poder construir algo como HAL 9000. En esta conferencia echaremos un vistazo a las principales tendencias en desarrollo de software hacia los próximos años.
Citation preview
From concept to cloud: A look at modern software
developmentChris Richardson
Author of POJOs in ActionFounder of the original CloudFoundry.com
crichardson@vmware.com@crichardson
plainoldobjects.com
From concept to cloud: A look at modern software
development
Chris RichardsonAuthor of POJOs in Action,
Founder of the original CloudFoundry.com crichardson@vmware.com, @crichardson
plainoldobjects.com
About Chris
(About Chris)
About Chris()
About Chris
About Chris
http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/
vmc push About-Chris
Developer Advocate for
Signup at http://cloudfoundry.com
Promo code: sgce2012
Dreams of my childhood
Published in 1968
The vision
HAL 9000
.... is an artificial intelligence
became operational on
12 January 1992.....
So how are we doing?
The reality
Floor cleaning robot
The reality
http://en.wikipedia.org/wiki/IBM_Watson cluster of ninety IBM Power 750 servers with a total of 2880 POWER7 processor
cores and 16 Terabytes of RAM.
We still have a long way to go before we can
build HAL
But despite that....
Today, a small team of people can easily build an application that’s used by millions of people world-wide
http://highscalability.com/blog/2012/5/7/startups-are-creating-a-new-system-of-the-world-for-it.html
Endpoints everywhere
The web
Cloud Computing
http://en.wikipedia.org/wiki/Cloud_computing
How did we get here?
Many of the fundamentals of computing were
invented before many of us were born
Computer hardware
1948 - First computer
http://en.wikipedia.org/wiki/ENIAC
1940
1950
1960
1970
1980
1990
2000
201030 tons167 m2
150 KW
40 divs/second
1956 - Hard drive
http://boingboing.net/2010/06/24/ibm-hard-disk-drive.html
3.7 Mbit
Weighed over a ton
Designed to fit through a door
1971 - microprocessor
http://en.wikipedia.org/wiki/Microprocessor
4 bit
108-740 kHz
2,300 transistors
The power of Moore’s law
http://www.gotw.ca/publications/concurrency-ddj.htm
Hard disk capacity
In my career1982 RM 380Z4 Mhz 8-bit Z8032K RAM100K floppy
2012MacBookPro2.5 Ghz Quad core 64 bit Intel i7256K/core + 8M cache,16G RAM512G SSD drive
Machrone’s law
“...the price of the computer you want to buy
will always be $5000*...”
* Today it’s $3000
Wirth’s law
“Software is getting slower more rapidly than hardware becomes
faster.”
Programming languages
1958 - Lisp
http://en.wikipedia.org/wiki/Lisp_(programming_language)
1940
1950
1960
1970
1980
1990
2000
2010
garbage collection dynamic typing
self-hosting compiler tree data structures
(defun factorial (n) (if (<= n 1) 1 (* n (factorial (- n 1)))))
1960 - Algol 60
http://en.wikipedia.org/wiki/Algol_60
1940
1950
1960
1970
1980
1990
2000
2010 Many languages
including Java are derived from Algol
procedure Absmax(a) Size:(n, m) Result:(y) Subscripts:(i, k); value n, m; array a; integer n, m, i, k; real y;begin integer p, q; y := 0; i := k := 1; for p:=1 step 1 until n do for q:=1 step 1 until m do if abs(a[p, q]) > y then begin y := abs(a[p, q]); i := p; k := q endend Absmax
1967 - Simula
http://en.wikipedia.org/wiki/Simula
1940
1950
1960
1970
1980
1990
2000
2010
class-based object-oriented
programming
So what have programming language designers been
doing since then?
1995 - Java
Not innovative BUT
Brought garbage collection, object-oriented programming, exception handling, safety ... to mainstream
developers
Gosling “Java is a blue collar language”
google “Gosling the feel of Java”
1940
1950
1960
1970
1980
1990
2000
2010
Tools and techniques
1980s - Cool IDEs for Lisp and Smalltalk
Class hierarchy
Editor
Repl
Debugger
ClassList
1940
1950
1960
1970
1980
1990
2000
2010
1990s• CVS - Lockless version management
• DVCS - distributed version control
• Refactoring
• Agile/XP starts to become popular
• Mosaic web browser
• Web crawlers
http://www2.jpl.nasa.gov/sl9/1940
1950
1960
1970
1980
1990
2000
2010
2000s - online developer community
• Explosion of open-source
• Binary artifact repositories - 41K maven.org
• Github.com social coding - 1M projects!
1940
1950
1960
1970
1980
1990
2000
2010
2000s - DevOps, NoOps, Continuous Deployment, etc
• Developers responsible for testing, operations, ...
• Continuous deployment: update production many times a day
• Chaos monkeys - constantly verify that the system can tolerate failurehttp://highscalability.com/blog/2012/5/7/startups-are-creating-a-new-system-of-the-world-for-it.html
1940
1950
1960
1970
1980
1990
2000
2010
William Gibson
“The future is already here – it's just not evenly distributed.”
Ideas take a long time to diffuse through the software
community
Where is here?
Polyglot applications NoSQL databases Cloud Computing
Until ~2004 Java was the one language to
rule them all
Some developers became frustrated with
the perceived complexity of Java
Ruby on Rails became popular
• Web application framework for Ruby
• Significantly more productive
• Simplicity of Convention over Configuration
• Motivated the Java community to improve:
• e.g. Spring became even easier to use
But...
…Highlighted problems with Java
• Tedious collection processing
• Painful object construction code
• Tedious XML processing
• Types: verboseness and complexity of generics
• Limited support for DSLs
Reopened the debate about programming languages
Alternative languages
• Ruby
• Groovy
• Clojure
• Scala
• JavaScript/NodeJS
Closuresscala> val numbers = Array(1,2,3,4,5) numbers: Array[Int] = Array(1, 2, 3, 4, 5)
scala> val oddNumbers = numbers.filter((x => x % 2 == 1)) oddNumbers: Array[Int] = Array(1, 3, 5)
scala> val doubled = numbers.map(_ * 2)doubled: Array[Int] = Array(2, 4, 6, 8, 10)
XML processing
scala> val xml = <foo><bar>baz</bar></foo>xml: scala.xml.Elem = <foo><bar>baz</bar></foo>
scala> (xml \ "bar").textres9: String = “baz”
Meta-programming
Computer programs that modify themselves
Grails persistence methodsclass Customer { String name}
Customer c = new Customer("John Doe") if (!c.save()) fail "validation failed: ${c.errors}" Customer c2 = Customer.get(c.id)
def customers = Customer.findAllByName(“Fred”)
Sinatra DSL example
require 'sinatra'
get '/hi' do "Hello World!"end
A Ruby-based Domain Specific Language for web programming
So what should we do with these languages?
Tomcat
Traditional web application architecture
Browser
WAR
MySQL Database
developtest
deployscale
ShippingService
BillingService
InventoryService
StoreFrontUI
Simple to
ApacheCustomer
Management
But there are problems
• Inadequate for modern, real-time web applications (need NodeJS)
• Obstacle to frequent deployments
• Slows down IDE and dev/test cycle
• Obstacle to scaling development
• Requires long term commitment to technology stack
Modular, polyglot application architecture
RabbitMQ
NodeJS
Inventory Service Shipping Service
Billing Service Inventory Database
Order Database
Standalone“headless” Spring/Java applications
Spring/Scala web application
Customer Database
Desktop Browser Native Mobile application HTML5 mobile application
StoreUI StoreUI StoreUI
StoreUIJavascriptAsynchronous,
scalable communication
CustomerManagement
Rails
Real world examples
http://highscalability.com/amazon-architecture
Between 100-150 services are accessed to build a page.
http://techblog.netflix.com/
http://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdf
http://queue.acm.org/detail.cfm?id=1394128
There are drawbacks• Complexity: Architectural, development,
deployment
• Deciding when to use it
• In the beginning: you don’t need it and it will slow you down
• When you do need it: refactoring existing code is painful
But there are many benefits
• Scales development: focused two pizza devops teams
• Deploy services independently
• Scale services independently
• Improves fault isolation
• Enforces well defined interfaces between components
• Eliminates long-term commitment to a single technology stack
Modular, polyglot applications
If services are small...
• Regularly rewrite using a better technology stack
• Pick the best developers rather than best <pick a language> developers ⇒ polyglot culture
• Adapt system to changing requirements and better technology without a total rewrite
Moreover: you are not the same you ...• 50 to 70 billion of your cells die each day
• Cell lifetimes:
• hours - some white blood cells
• days - stomach lining cells
• years - bone cells
• lifetime - brain cells
• Yet you (the system) remains intact
http://dreamsongs.com/Files/WhitherSoftware.pdf
http://dreamsongs.com/Files/DesignBeyondHumanAbilitiesSimp.pdf
Can we build software systems with these
characteristics?
Too much technical debt ⇒ component death?
Polyglot applications NoSQL databases Cloud Computing
Relational databases have been the place to
store your data
Limitations of relational databases
• Scaling
• Despite Moore’s law vertical scaling is limited/expensive
• Horizontal scaling is limited
• Distribution
• Updating schema
• O/R impedance mismatch
• Handling semi-structured data
Solution: Use NoSQLBenefits
• Higher performance
• Higher scalability
• Richer data-model
• Schema-less
Drawbacks
• Limited transactions
• Relaxed consistency
• Unconstrained data
NoSQL databases
http://nosql-database.org/ lists 122+ NoSQL databases
Data Model Examples
Key-value Redis, Voldemort
Document MongoDB, CouchDb
Extensible columns/Column-oriented
Cassandra, Hbase, SimpleDB, DynamoDB
Graph Neo4j
• Advanced key-value store
• Very fast
• Optional persistence
Redis
K1
K2
K3
V1
V2
V2
MongoDB: fast, scalable, document orientedServer
Database: Food To Go
Collection: Restaurants
{ "_id" : ObjectId("4bddc2f49d1505567c6220a0") "name": "Ajanta", "serviceArea": ["94619", "99999"], "openingHours": [
{ "dayOfWeek": 1, "open": 1130, "close": 1430 },{ "dayOfWeek": 2, "open": 1130, "close": 1430 }, …
]}
BSON = binary JSON
Sequence of bytes on disk è fast i/o
Datacenter 2
Cassandra cluster
Datacenter 1
Cassandra cluster
Cassandra: very scalable
Node 1
Node 2
Node 3
Node 4
Node 1
Node 2
Node 3
Node 4
Application Application
The future is polyglot persistence
IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg
e.g. Netflix• RDBMS• SimpleDB• Cassandra• Hadoop/Hbase
Polyglot applications NoSQL databases Cloud Computing
Let’s imagine that you want to deploy an app...
• Do you know how much hardware to buy?
• Can you afford it?
• How long does it take to approve, buy and install?
• Who is going to set it up?
• What happens if the traffic increases 10x?
• Can we afford a test lab?
Cloud computing empowers us to deal
with these challenges...
Cloud computing defined
IT delivered as a service
Over the internet
Self-service
Pay per use
SaaS
PaaS
IaaS
The three layers of cloud computing
Wednesday October 18, 2006....
.... spoke at the Oakland JUG
AWS evangelist
Sign up and deploy your application a few minutes later
• Login using your existing Amazon account
• Select the web
Benefits of IaaS
• Agility
• Pay per use
• Elasticity
• Focus on deploying your application
But you need to configure and maintain the operating systems, the app servers and the
databases etc.
SaaS
PaaS
IaaS
Need to move up the stack
PaaS
= Application deployment
and management
Service provisioning
+
Simple, Open, Flexible,
Scalable
The Open Platform as a Service
Deploy and scale applications in seconds, without locking yourself into a single cloud
Applica'on Service Interface
Data Services
Other Services
Msg Services
Easy polyglot application deployment and service provisioning
vFabric Postgres
vFabric RabbitMQTM
Additional partners services …
OSS community
Private Clouds
PublicClouds
MicroClouds
CloudFoundry.COM - Multi-tenant PaaS operated by VMware
Runtimes & FrameworksServices
vCenter / vSphere
CloudFoundry.COM (beta)
Infrastructure
Micro Cloud FoundryTM – Industry first
downloadable PaaS
Runtimes & FrameworksServices
Your Laptop/PC
Micro Cloud Foundry
Single VM instance of Cloud Foundry
that runs on a developer’s MAC or PC
CloudFoundry.ORG - Community open-source project
CloudFoundry.ORG
DownloadCode
Setup Environment
Deploy Behind FirewallBOSH
Apache2 license
Your Infrastructure
Sinatra + Redis
require 'sinatra'require 'redis'
configure do $r = Redis.new(:host => '127.0.0.1', :port => '6379') if !$r end
get '/' do "Hello World! " + $r.incr("hitcounter").to_send
http://sgce2012.cloudfoundry.com/
Connect to Redis
Increment hit counter
Consuming 3rd party cloud services
Thousands of 3rd party services
http://www.programmableweb.com/apis/directory
Number of APIs increasing exponentially
• Predominantly REST
• Predominantly JSON
• > billion API calls/day: Twitter, Google, Facebook, Netflix, Accuweather, ...
• Increasing number of API-only companies
http://www.slideshare.net/jmusser/j-musser-apishotnotgluecon2012
Cloud service trends
Diverse
SendGrid
GeoNamesTwilio
Tropo
Mashery
Yahoo
Amazon
Face.comPaypal
Cybersource Aviary.com
WorldAddressesAlchemyAPI
Amazon Simple Storage Service (S3)
• Bucket
• has a globally unique name, e.g. cloudtools.org.photos
• contains objects
• Objects - 1 byte – 5 Tbytes
• Pay per use: storage + bandwidth
• Accessed via RESTful API
• PUT – create bucket or object
• GET – retrieve object
• DELETE – delete an object 1T objects June 2012
face.com - face detection and recognition
http://api.face.com/faces /detect.json?api_key=&api_secret=&urls=<url>&detector=Aggressive&attributes=all
Twilio - Telephony and SMS as a service
• REST API
• Allocate phone numbers
• Make and receive phone calls
• Send and receive SMS messages
• Pay per use:
• Phone number – per month
• Phone calls - per-minute
• SMS – per SMS sent or received
• Example: Salesforce – SMS-based voting for 19,000 conference attendees
Expose your APIs and
build an ecosystem
Where are we going?
Going beyond today’s pop culture
But love is blind and lovers cannot seeThe pretty follies that themselves commit;For if they could, Cupid himself would blush To see me thus transformed to a boy.
William Shakespeare, The Mechant of Venice
Gartner hype curve
http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp#
Concurrency
Clock speeds have plateaued
http://www.gotw.ca/publications/concurrency-ddj.htm
Multi core is the future
• Dual core A5 for iPhone 4S
• Quad core Intel i7 for laptops
• 10 core Intel Xeon Processor
• 3072 core NVidia Tesla Kepler
Sometimes we can use request-level parallelism and let the container worry about it
BUT not always...
But concurrent programming is difficult
The problem=
Mutable state that’s shared between multiple
threads (using locks)
Not thread safepublic class InventoryTracker { private int inStock; private int sold = 0; public InventoryTracker(int initialInventory) { this.inStock = initialInventory; } public int getInStock() { return inStock; } public int sell(int units) { inStock -= units; sold += units; return inStock; }}
Thread safepublic class InventoryTracker { private int inStock; private int sold = 0; public InventoryTracker(int initialInventory) { this.inStock = initialInventory; } public synchronized int getInStock() { return inStock; } public synchronized int sell(int units) { inStock -= units; sold += units; return inStock; }}
• Single threaded runtime ⇒ no concurrency
• Runs an event loop that waits for i/o events
• i/o event ⇒ callback function
• Updates state
• Publishes events ⇒ triggers more callbacks)
• Initiates i/o, registering callbacks
• Returns
NodeJS - Event driven
Mutable state that’s shared between multiple threads
NodeJS examplevar http = require('http');var fs = require('fs');var url = require('url');
var inStock = 100;var sold = 0;
http.createServer(function (req, res) { var quantity = parseInt(url.parse(req.url, true).query.quantity); inStock -= quantity; sold += quantity; var message = "inStock=" + inStock + ", sold=" + sold; res.end(message);}).listen(1337, "127.0.0.1");
console.log('Server running at http://127.0.0.1:1337/');
HTTP request handler
http://127.0.0.1:1337?quantity=1
• Actor = object that asynchronously exchanges immutable messages with other actors
• Actor processes one message at a time
• Sends messages
• Updates its private mutable state
• Creates new actors
• No shared state
Scala Actors
Mutable state that’s shared between multiple threads
Scala examplecase class Sell(quantity: Int)case class Sale(inStock : Int, sold : Int)
class InventoryTracker(initialInventory : Int) extends Actor { var inStock = initialInventory var sold = 0 def receive = { case Sell(quantity) => inStock -= quantity sold += quantity sender ! Sale(inStock, sold) }}
Update state
Message types
Send reply
case class Order(quantity: Int)
class StoreFront(inventoryTracker: ActorRef) extends Actor { var numberOfOrders = 0
def receive = { case Order(quantity) => numberOfOrders += 1 inventoryTracker ! Sell(quantity) case Sale(inStock, sold) => println("numberOfOrders=" + numberOfOrders + ",inStock=" +
inStock + ", sold=" + sold) }}
Send message
Handle reply
• Eliminates mutable state
BUT
• I/O and state management require Monads
• Monads are extremely difficult to learn
Pure functional programming
Mutable state that’s shared between multiple threads
Clojure - a modern LISP• Practical functional programming language
• Runs on the JVM
• Core data structures are immutable
BUT
• Ref = object that holds a mutable reference to a value
• Refs can only be updated within a transaction
• Software Transaction Memory
• Atomic, Consistent, Isolated
• No explicit locksMutable state that’s
shared between multiple threads
Clojure STM example
(def it (make-inventory-tracker 100))
(sale it 25)
(defn sale [tracker quantity] (dosync (let [new-in-stock (alter (:in-stock tracker) - quantity) new-sold (alter (:sold tracker) + quantity) ] [new-in-stock new-sold] )))
(defrecord InventoryTracker [in-stock sold])
(defn make-inventory-tracker [initial-inventory] (InventoryTracker. (ref initial-inventory) (ref 0)))
dosync=
a transaction
Update refs
Create refs
Final thoughtsSoftware development is an old
... yet immature profession
Multi-core programming is a challenge
BUT
The cloud enables anyone with a good idea to create an application that touches the lives of millions of people
Thank you!
Chris Richardson
Author of POJOs in ActionFounder of the original CloudFoundry.com
crichardson@vmware.com@crichardson
plainoldobjects.com
Signup @ cloudfoundry.com Promo code: sgce2012
Recommended