Big Data TutorialQCon São Paulo 2013
Everything old is new again
Everything old is technically
feasible
The ability to summon 100’s or 1000’s of
machines with an API call is what brings parallel
computing to everyone...
combined with virtually limitless cloud storage,
Big Data is now accessible to everyone, not just big companies.
Tweet @jedberg with feedback!
Jeremy Edberg
Tweet @jedberg with feedback!
What is reddit?
Tweet @jedberg with feedback!
Netflix is the world’s leading Internet television network with nearly 38 million members in 40
countries enjoying more than one billion hours of TV shows and movies per month, including original
series. For one low monthly price, Netflix members can watch as much as they want, anytime,
anywhere, on nearly any Internet-connected screen.Source: http://ir.netflix.com
What is Netflix?
Tweet @jedberg with feedback!
Why Big Data, how is it useful and what can it do for you?
SQL and NoSQL -- What's the difference, what are the pros and cons, how do you move from one
to the other?
Practical steps to keep your Big Data systems reliable.
NoSQL technologies such as HBase/HDFS, BigTable, MongoDB, S3, Redis, Cassandra, Hadoop, Pig, Hive,
Flume and more.
What You Will Learn
Tweet @jedberg with feedback!
This is your workshop
• We’ll be together for 3+ hours
• You (or your employer) paid a lot of money to be here
• Let’s make it worth your while!
Tweet @jedberg with feedback!
Let’s make this awesome together
• Ask questions
• Let me know if you want me to move on or go into more detail
• Keep it interactive!
Tweet @jedberg with feedback!
Schedule
Introduction to Big Data and its uses
Survey of Big Data Technology
Real-Time Data Systems
Demo: Cassandra in Action -- Building and using a data model
Building reliable Big Data systems
Wrap up, conclusions, questions
Tweet @jedberg with feedback!
What is Big Data?
• The tools and processes of managing and utilizing large datasets.
• (with virtualized resources)
• Structured and Unstructured data
(I’ll ask this again at the end)
Tweet @jedberg with feedback!
Simple vs. Complex
Tweet @jedberg with feedback!
Flu outbreak
Tweet @jedberg with feedback!
Data Wants to be Free
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Data is the most important asset your business will
have.
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Privacy
• That sharing comes at a cost, and that’s privacy.
• Some people value privacy vs utility, and some don’t.
• Teenagers don’t seem to value privacy at all.
Tweet @jedberg with feedback!
So how can Big Data help me?
Tweet @jedberg with feedback!
Security
Tweet @jedberg with feedback!
Security
Tweet @jedberg with feedback!
How Big Data transformed the dairy
industry
Tweet @jedberg with feedback!
How India’s “Satyamev Jayate” uses Big Data to power their TV show.
Tweet @jedberg with feedback!
Trend Analysis
Tweet @jedberg with feedback!
Trend Analysis
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Actionable Metrics
Tweet @jedberg with feedback!
Other Metrics
• Pennies earned
• Pageviews
• Votes / comments / links
Tweet @jedberg with feedback!
How Big Data can make your business more successful.
• Use big data to do real time analysis to deliver better experiences for your customers
• Sometimes information is more valuable when it is shared.
• We are floating in good answers, but the good questions are scarce.
• Keep your data clean on the way in.
• Where does big data create value in your company?
Tweet @jedberg with feedback!
What's possible -- and what's difficult -- for companies that adopt Big Data approaches to
storage and analysis.
• Data gravity. As you data gets bigger you need to move your application closer to it.
• Moving from Sql to NoSql
Tweet @jedberg with feedback!
DataWhat does Netflix do with it all?
Tweet @jedberg with feedback!
We store it!
• Cache (memcached)
• Cassandra
• RDS (MySql)
Tweet @jedberg with feedback!
RDS (Relational Database Service)
Tweet @jedberg with feedback!
Cassandra
Tweet @jedberg with feedback!
Overview
3
!""#$%&'()* +(##,%-(./*
0$1,*
Data collection pipeline
Data processing pipeline
234*
Overview Data collection pipeline Data collection pipeline Data collection pipeline
Text
Data Collection Pipeline
3
Data processing pipeline Data processing pipeline
TextTextData Processing Pipeline
Tweet @jedberg with feedback!
Chuckwa/Honu messages / min
63 billion
messages a day
Tweet @jedberg with feedback!
Hiveselect videoID, count(*) as cfrom events where dateint>=20120611 and dateint<=20120617 and event="Watched" and result="SUCCESS" group by videoid order by count desc limit 5;
Tweet @jedberg with feedback!
A/B Testing
Tweet @jedberg with feedback!
A/B Testing
Online Data Offline Data
Test Cell allocationTest MetadataStart/End dateUI Directives
Test trackingRetention
Fraction ViewedPages Viewed
Tweet @jedberg with feedback!
Atlas
Tweet @jedberg with feedback!
AWS Usage (Ice)Dollar amounts have been carefully removed
Tweet @jedberg with feedback!
Chronos
Tweet @jedberg with feedback!
Netflix Dataoven
Data WarehouseOver 2 Petabytes
Ursula
Aegisthus
Data Pipelines
From cloud Services
~100 BillionEvents/day
From C*Terabytes ofDimension
data
Hadoop Clusters – AWS EMR
1300 nodes 800 nodes Multiple 150 nodes
Over 2 Petabytes
Hadoop Clusters – AWS EMR
RDS
Metadata
Gateways
Tools
Tweet @jedberg with feedback!
Genie: Goals
• Open up the data engineering infrastructure• Self-service for SLA/production
jobs
• Abstraction/management of back-end resources• Hadoop/Hive/Pig as a Service• Eliminate “gateway” bottlenecks
Tweet @jedberg with feedback!
Genie: Set of Services
• Job Execution• REST-ful API to run Hadoop,
Hive and Pig jobs
• Abstracting out cluster details from clients
•Horizontal scalability via auto-scaling groups on the cloud
Tweet @jedberg with feedback!
Genie: Set of Services
• Resource Configuration/Management
• Management of cluster status
• Repository of configurations (for cluster, hive, pig)
• Mapping of jobs to clusters
Tweet @jedberg with feedback!
Data Gravity
• Coined by Dave McCrory
• First described here: http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds/
Tweet @jedberg with feedback!
What is Data Gravity?
Source: nationalgeographic.com
Tweet @jedberg with feedback!
Data Gravity and you
• The bigger your dataset, the harder it is to move from anywhere to anywhere
• Also, how do you move that data without affecting your running application?
Tweet @jedberg with feedback!
reddit’s data gravity problem
• We had a lot of data that was ever-growing
• We were so resource constrained we couldn’t move it without hurting our application
Tweet @jedberg with feedback!
Netflix’s data gravity problem
• Needed the data in the datacenter
• We were “Roman Riding” for a long time
Source: http://horseandman.com
Tweet @jedberg with feedback!
Questions?
Tweet @jedberg with feedback!
Schedule
Introduction to Big Data and its uses
Survey of Big Data Technology
Real-Time Data Systems
Demo: Cassandra in Action -- Building and using a data model
Building reliable Big Data systems
Wrap up, conclusions, questions
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
SQL vs. NoSQL
• NoSql is generally unstructured and the data storage is schemaless
• Eventually consistent systems
• Horizontally scalable
Tweet @jedberg with feedback!
SQL vs. NoSQL
• SQL systems have structured data and fixed schemas
• ACID compliant (I’d rather put my $$ here than in an eventually consistent system!)
• Generally have to scale up, not so good at out
Tweet @jedberg with feedback!
CAP Theorem
• Consistent
• Available
• Partition-resistant
Tweet @jedberg with feedback!
Key/Value vs. Document Store
• Key/Value is just like the hash table data structure you are used to
• Great for use with object oriented languages
• redis, Cassandra, S3
Tweet @jedberg with feedback!
Key/Value vs. Document Store
• Stores whole documents with certain properties, often in JSON or XML
• Good for large chunks of data, like things scraped from the web
• MongoDB, CouchDB
Tweet @jedberg with feedback!
JSON• JavaScript Object Notation
• Originally a subject of JSON, now a standard cross platform document format
• Lots of parsers in many languages
• Very similar to XML, less verbose
Tweet @jedberg with feedback!
{"firstName": "John","lastName": "Smith","age": 25,"address": {
"streetAddress": "21 2nd Street","city": "New York","state": "NY","postalCode": "10021"
},"phoneNumber": [
{"type": "home","number": "212 555-1234"
},{
"type": "fax","number": "646 555-4567"
}]
}
Tweet @jedberg with feedback!
The Technologies
Tweet @jedberg with feedback!
Cassandra
Tweet @jedberg with feedback!
Cassandra Architecture
Tweet @jedberg with feedback!
Cassandra Architecture
Tweet @jedberg with feedback!
How it works• Replication factor
• Quorum reads / writes
• Bloom Filter for fast negative lookups
• Immutable files for fast writes
• Seed nodes
• Multi-region
• Gossip protocol
Tweet @jedberg with feedback!
Cassandra Benefits
• Fast writes
• Fast negative lookups
• Easy incremental scalability
• Distributed -- No SPoF
Tweet @jedberg with feedback!
Things Netflix stores in Cassandra
• Track service level call
• Instrument low level HTTP client
• Calls graph (who is calling who)
• Request processing vs Perceived latency
• Payload marshalling/unmarshalling- duration, size, etc
• Service Results- Status, Error code, Exception, etc
Tweet @jedberg with feedback!
Things Netflix stores in Cassandra
• Video Quality
• Network issues
• Usage History
• Playback Errors
Tweet @jedberg with feedback!
Why Cassandra?
• Availability over consistency
• Writes over reads
• We know Java
• Open source + support
Tweet @jedberg with feedback!
astyanax
• Netflix Cassandra Java client
• High level abstractions for Cassandra
• https://github.com/Netflix/astyanax
Tweet @jedberg with feedback!
Hadoop
Image from searchworks.org
Tweet @jedberg with feedback!
http://highlyscalable.wordpress.com/2012/02/01/mapreduce-patterns/
Tweet @jedberg with feedback!
HBase
• Open source clone of Google’s BigTable (a sparse, distributed multi-dimensional sorted map)
• Integrated with Hadoop for easy reads and writes.
• Distributed
Tweet @jedberg with feedback!
Overview
3
!""#$%&'()* +(##,%-(./*
0$1,*
Data collection pipeline
Data processing pipeline
234*
Overview Data collection pipeline Data collection pipeline Data collection pipeline
Text
Data Collection Pipeline
3
Data processing pipeline Data processing pipeline
TextTextData Processing Pipeline
Tweet @jedberg with feedback!
Hiveselect videoID, count(*) as cfrom events where dateint>=20120611 and dateint<=20120617 and event="Watched" and result="SUCCESS" group by videoid order by count desc limit 5;
Tweet @jedberg with feedback!
Hive
INSERT OVERWRITE TABLE user_active SELECT user.* FROM user WHERE user.active = 1;
Tweet @jedberg with feedback!
PigA = load 'passwd' using PigStorage(':'); B = foreach A generate $0 as id;dump B; store B into ‘id.out’;
Tweet @jedberg with feedback!
Oozie<property> <name>cassandra.thrift.address</name> <value>${cassandraHost}</value></property><property> <name>cassandra.thrift.port</name> <value>${cassandraPort}</value></property><property> <name>cassandra.partitioner.class</name> <value>org.apache.cassandra.dht.RandomPartitioner</value></property><property> <name>cassandra.consistencylevel.read</name> <value>${cassandraReadConsistencyLevel}</value></property><property> <name>cassandra.consistencylevel.write</name> <value>${cassandraWriteConsistencyLevel}</value></property><property> <name>cassandra.range.batch.size</name> <value>${cassandraRangeBatchSize}</value></property>
Tweet @jedberg with feedback!
Other “NoSQL” solutions
• Memcache
• Redis
• CouchDB
• MongoDB
• DynamoDB
• Voldemort
• Riak
• Zookeeper
• S3
• Postgres
Tweet @jedberg with feedback!
I love memcacheI make heavy use of memcached
Tweet @jedberg with feedback!
A
BCC B
3
2
A1
Tweet @jedberg with feedback!
A
BCC B
3
2
A1
D
Tweet @jedberg with feedback!
A
BCC B
3
2
A1
D
+
EVCache
Tweet @jedberg with feedback!
Redis
• Stores the entire database in RAM
• Support complex data structures
• Writes to disk periodically
• Fast and predictable with small datasets
Tweet @jedberg with feedback!
Redis data structures
• strings
• hashes
• lists
• sets and sorted sets
Tweet @jedberg with feedback!
Redis use cases
• As a drop in replacement for memcache
• Ephemeral data that you’re ok with losing
• Performance falls off a cliff when the dataset gets bigger than RAM
Tweet @jedberg with feedback!
CouchDB
• Document oriented database
• Stores json like objects with deep queries
• Not easy to scale horizontally
• Uses JS mapreduce functions call “views” for data access
Tweet @jedberg with feedback!
CouchDB use cases
• You have a large dataset and you want easy access to attributes
• Prototyping or just starting out
Tweet @jedberg with feedback!
MongoDB
• Document store, similar to CouchDB
• JSON like objects that are easy to work with
• Javascript query language
Tweet @jedberg with feedback!
MongoDB use cases
• Similar to CouchDB
• Less scalable than CouchDB
• Biases towards speed over durability
Tweet @jedberg with feedback!
Voldemort
• Open source clone of Amazon’s Dynamo database (not DynamoDB)
• Consistent key hashing for fast lookups and easy horizontal scaling
• Built in versioning
Tweet @jedberg with feedback!
Voldemort use cases
• Places where eventual consistency are ok
• Like an Amazon shopping cart for example!
• Or Linkedin!
• Sometimes multiple different answers can come back, it is up to the client to figure out the right answer
Tweet @jedberg with feedback!
Riak
• Like Voldemort (Amazon’s Dynamo paper)
• Uses a gossip protocol like Cassandra
• Query in Erlang or Javascript
Tweet @jedberg with feedback!
Riak use cases
• Similar to Voldemort, where eventual consistency is ok
Tweet @jedberg with feedback!
Zookeeper
• Specialized key/value store
• Presents like a file system
• Distributed for reliability and fast reads
• At the expense of slow writes with more nodes
Tweet @jedberg with feedback!
Zookeeper use cases
• System configuration
Tweet @jedberg with feedback!
Postgres
Tweet @jedberg with feedback!
Sample Schemalink_thing int id timestamp date int ups int downs bool deleted bool spam
link_data int thing_id string name string value char kind
Tweet @jedberg with feedback!
The thing layer
• Postgres is used like a key/value store
• Thing table has denormalized data
• Data table has arbitrary keys
• Lots of indexes tuned for our specific queries
• Thing and data tables are on the same box, but don’t have to be
Tweet @jedberg with feedback!
Moving from Postgres to Cassandra
• We were lucky -- we already used key/value
• But it wasn’t completely straightforward
• Some things are a lot easier relationaly
• Like taking counts of things
Tweet @jedberg with feedback!
Tips to moving successfully
• No normalizaion
• Your app will have to do a lot of what your database used to do
• De-normalize
Tweet @jedberg with feedback!
Schedule
Introduction to Big Data and its uses
Survey of Big Data Technology
Real-Time Data Systems
Demo: Cassandra in Action -- Building and using a data model
Building reliable Big Data systems
Wrap up, conclusions, questions
Tweet @jedberg with feedback!
Hadoop -- Past its prime?
• Was pioneered by Google, then an open source clone came along
• Google has mostly moved on to more real-time systems
Tweet @jedberg with feedback!
Google Projects
• Dremel a.k.a. BigQuery
• Percolator
• Pregel
Tweet @jedberg with feedback!
Other real-time projects
• Storm -- Twitter
• Turbine -- Netflix
• Redshift -- Amazon
Tweet @jedberg with feedback!
NoSQL + SQL + Hadoop
• The latest trend in Big Data
• Putting a layer of SQL on top of a distributed data store
• Finally splitting the query layer from the data layer!
Tweet @jedberg with feedback!
Schedule
Introduction to Big Data and its uses
Survey of Big Data Technology
Real-Time Data Systems
Demo: Cassandra in Action -- Building and using a data model
Building reliable Big Data systems
Wrap up, conclusions, questions
Tweet @jedberg with feedback!
Building a Data Model
• What questions you want to ask your data?
• Don’t try and normalize anything
• Instead of changing a value keep a record of what happened
Tweet @jedberg with feedback!
Let’s build a telemetry system!
• This is a slightly modified real-world example of something we built to support the Netflix open connect project
Tweet @jedberg with feedback!
Background
• Caches all over the world
• Named like ORD1, LAX1, SJC2, etc.
• We need to collect about 20 metrics for each cache on a regular basis
Tweet @jedberg with feedback!
The questions
• Get last 3 runs for SJC2 and show the collected data
• What caches did we see on the last run and what are their details?
Tweet @jedberg with feedback!
The tables
collected_propertiescollected_propertiescollected_propertiescollected_propertiescollected_propertiescollected_propertiescollected_properties
Keys HealthyHealthy other load upup
collection_cache_by_timescollection_cache_by_timescollection_cache_by_timescollection_cache_by_timescollection_cache_by_timescollection_cache_by_timescollection_cache_by_times
Keys cache1 cache2 cache3 cache4 cache5 ...
collections_by_cachecollections_by_cachecollections_by_cachecollections_by_cachecollections_by_cachecollections_by_cachecollections_by_cache
Keys 1 2 3 4 5 ...
Tweet @jedberg with feedback!
Python Code Walkthrough
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Files129837 95014
43534 10020
345345 90069
980345 10001
1098445 59390
9084309 32901
43534 98898Queue
Data Loader
Data Loader
Data Loader
Data Loader
Queue
Data Processor
Data Processor
Data Processor
Data Processor
DB
DB
Tweet @jedberg with feedback!
Schedule
Introduction to Big Data and its uses
Survey of Big Data Technology
Real-Time Data Systems
Demo: Cassandra in Action -- Building and using a data model
Building reliable Big Data systems
Wrap up, conclusions, questions
Tweet @jedberg with feedback!
Building a Reliable Data Store
Tweet @jedberg with feedback!
If it won’t scale, it'll fail.-- paradrox
Tweet @jedberg with feedback!
1 > 2 > 3 Going from two to three is hard
Tweet @jedberg with feedback!
1 > 2 > 3 Going from one to two is harder
Tweet @jedberg with feedback!
1 > 2 > 3If possible, plan for 3 or more from the beginning.
Tweet @jedberg with feedback!
Going multi-zone
Tweet @jedberg with feedback!
Benefits of Amazon’s Zones
• Loosely connected
• Low latency between zones
• 99.95% uptime guarantee per region
Tweet @jedberg with feedback!
Going Multi-region
Tweet @jedberg with feedback!
Leveraging Multi-region
• 100% uptime is theoretically possible.
• You have to replicate your data
• This will cost money
Tweet @jedberg with feedback!
Reliability and $$
Tweet @jedberg with feedback!
Alert Systems
alerting
api
api
COREEvent
Gateway
Paging Service
AmazonSES
CORE Agent
Other Team’s Agent
CORE Agent
Atlas
Tweet @jedberg with feedback!
Automate all the things!
Tweet @jedberg with feedback!
Automate all the things!
• Application startup
• Configuration
• Code deployment
• System deployment
Tweet @jedberg with feedback!
Automation
• Standard base image
• Tools to manage all the systems
• Automated code deployment
Tweet @jedberg with feedback!
Netflix has moved the granularity from the
instance to the cluster
Tweet @jedberg with feedback!
!"#$%&'()*'+,-')./!0)/120)3456)
7'8)1,$')%()*,#-%+'(9):/;)
<#'()*=$=)
/'(#%>=?,@=A%>)
1$('=&,>B):/;)
*CD)
E%1)F%BB,>B)
GH'>!%>>'-$)!*I)J%K'#)
!*I)D=>=B'&'>$)=>L)
1$''(,>B)
!%>$'>$)M>-%L,>B)
!%>#"&'()M?'-$(%>,-#)
:71)!?%"L)1'(+,-'#)
!*I)MLB')F%-=A%>#)
J(%N#')
/?=9)
7=$-O)
The Netflix SOA
Tweet @jedberg with feedback!
The Netflix way
• Everything is “built for three”
• Fully automated build tools to test and make packages
• Fully automated machine image bakery
• Fully automated image deployment
Tweet @jedberg with feedback!
The Monkey Theory
• Simulate things that go wrong
• Find things that are different
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
The simian army• Chaos -- Kills random instances
• Chaos Gorilla -- Kills zones
• Chaos Kong -- Kills regions
• Latency -- Degrades network and injects faults
• Conformity -- Looks for outliers
• Circus -- Kills and launches instances to maintain zone balance
• Doctor -- Fixes unhealthy resources
• Janitor -- Cleans up unused resources
• Howler -- Yells about bad things like Amazon limit violations
• Security -- Finds security issues and expiring certificates
Tweet @jedberg with feedback!
Circuit BreakersBe liberal in what you accept, strict in what you send
Tweet @jedberg with feedback!
Incident Reviews
• What went wrong?
• How could we have detected it sooner?
• How could we have prevented it?
• How can we prevent this class of problem in the future?
• How can we improve our behavior for next time?
Ask the key questions:
Tweet @jedberg with feedback!
Cassandra Architecture
Tweet @jedberg with feedback!
Cassandra Architecture
Tweet @jedberg with feedback!
Database Resiliency with Shardingwith Sharding
Tweet @jedberg with feedback!
Horizontal vs. Vertical
Tweet @jedberg with feedback!
Sharding• reddit split writes across four master databases
• Links/Accounts/Subreddits, Comments, Votes and Misc
• Each has at least one slave in another zone
• Avoid reading from the master if possible
• Wrote their own database access layer, called the “thing” layer
Tweet @jedberg with feedback!
Queues are your friend• Votes
• Comments
• Thumbnail scraper
• Precomputed queries
• Spam
• processing
• corrections
Tweet @jedberg with feedback!
Pain Points
Higher and more varied network latency
Workaround: Fewer network calls, ask for more data at a time.
Tweet @jedberg with feedback!
Pain Points
EBS sometimes slowed down a bit
Workaround: Use caching and replication with read slaves to avoid relying on a single disk, or better yet, avoid the need for EBS altogether.
Tweet @jedberg with feedback!
Pain Points
Instances go away sometimes or become so slow that you want to make them go away.
Workaround: Avoid single points of failure and make sure your servers have automated configuration.
Tweet @jedberg with feedback!
Protip
The environment in a public cloud is inherently more variant (co-tenants, abusive or heavy users, etc)
Make sure your code is written to handle this -- state should be kept somewhere shared and redundant, not on the instance.
Tweet @jedberg with feedback!
Protip
Security was not the first thought when a lot of the cloud systems were designed
Make it your first thought though. A little planning goes a long way. Use security groups judiciously and keep those keys safe!
Tweet @jedberg with feedback!
Protip
Keep track of those limits!
To prevent someone from consuming too much, all resources have per account limits. Keep track of them and get them raised ahead of when you need them. Make sure to catch the exceptions too.
Tweet @jedberg with feedback!
Cause chaos
Tweet @jedberg with feedback!
Best Practices
• Keep data in multiple Availability Zones
• Avoid keeping state on a single instance
• Take frequent snapshots of EBS disks
• No secret keys on the instance
• Different functions in different Security Groups
Tweet @jedberg with feedback!
Autoscaling
Traffic Peak
Text1
2
Tweet @jedberg with feedback!
What about private clouds?
• Some of the problems you don’t have: noisy neighbors, lack of physical access
• Problem you do have: You have to pay for your spare capacity instead of someone else
Tweet @jedberg with feedback!
A taxonomy of Big Data and next-
generation storage solutions
• Noisy neighbors are a problem.
• Efficiency is necessary and getting better
Tweet @jedberg with feedback!
Schedule
Introduction to Big Data and its uses
Survey of Big Data Technology
Real-Time Data Systems
Demo: Cassandra in Action -- Building and using a data model
Building reliable Big Data systems
Wrap up, conclusions, questions
Tweet @jedberg with feedback!
What is Big Data?
• The tools and processes of managing and utilizing large datasets.
• (with virtualized resources)
• Structured and Unstructured data
(What’s missing?)
Tweet @jedberg with feedback!
This is where the slide on what you should have learned would
go.
I’m more interested in what you actually learned.
Tweet @jedberg with feedback!
More Netflix details
• http://techblog.netflix.com/2010/12/four-reasons-we-choose-amazons-cloud-as.html
• http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html
• http://techblog.netflix.com/2011/03/cloud-connect-keynote-complexity-and.html
Tweet @jedberg with feedback!
Just a quick reminder...(Some of) Netflix is open source:
https://github.com/netflix
Including astyanax:
https://github.com/Netflix/astyanax
reddit is open source too:
https://github.com/reddit
patches are now being accepted!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Netflix is hiring
http://jobs.netflix.com/jobs.html
Tweet @jedberg with feedback!
Please don’t forget to vote!
Voting is how we know what to present to you next time. :)
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Email: jedberg@{gmail,netflix}.com
Twitter: @jedberg
Web: www.jedberg.net
Facebook: facebook.com/jedberg
Linkedin: www.linkedin.com/in/jedberg
You can contact me here for questions: