Embed Size (px)
DESCRIPTION
Configuring, deploying, and managing Big Data infrastructure, Hadoop in particular, is time consuming and expensive. Infochimps’ Ironfan is an open source systems configuration suite for the cloud, quickly and easily orchestrating an entire Big Data stack including data ingestion, scraping, storage, computation, and monitoring. With Ironfan, you can spin up clusters when you need them and turn them off when you don’t, enabling you to spend your time, money, and engineering focus on finding insights and creating value, not getting your machines ready. These are the slides from the SXSWi workshop, where individuals learned how to go from a single development machine to a full-stack cloud deployment.
Citation preview
DevOps: Empowering Developers with Infrastructure
SXSW 2013 – Tuesday, March 12
Go here: http:// infochim.ps/15INnv8
Nathan Eliot - @temujin9Ryan Miller - @rmiller107
Amanda McGuckin-Hager - @shoogieTim Gasper - @timgasper
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 1
Agendahttp://infochim.ps/15INnv81. Intros - Housekeeping
(15 min – 15 total)
2. Initial Setup
(30 min – 45 total)
3. Debug Initial Set Up
(30-45 min – 1:15 total)
4. Standing Up a Simple Cluster
(30-60 min – 2:15 total)
5. Hadoop!
(30-60 min – 3:15 total)
6. General Q&A
(30-60 min – 4:00 total)
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 2
Key Ironfan Contributors
• Flip Kromer, @mrflip– CTO of Infochimps
• Nathaniel Eliot, @temujin9– Ops Engineer of Infochimps
• Chris Howe– System Architect at Civitas Learning
Infochimps Enterprise Cloud for Big Data
Custom Applications(Java, Python, etc.)
Business Intelligence(Cognos, BOBJ, Microstrategy)
Packaged Apps(ERP, CRM, etc.)
CUSTOMER APPLICATIONS
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 4
Why We Love Chef
• Infrastructure as Code– Version Control– Shareable– Testable– Recapitulable
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 5
Why We Love Chef
MySQL Nginx SOLR
My Application3/12/2013 #ironfan #devops #sxsw #bigdata #chef 6
Why We Love Chef
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 7
Why We Don’t Love Chef
• Anything is possible• Nothing is simple• There’s not much
repetition (not DRY)
Why We Don’t Love Chef
Too much is hard-coded at development/upload time!
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 9
Why We Don’t Love Chef
How do we make @server_ips dynamic?
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 10
Why We Wrote Ironfan
• Simplify, unify, and standardize our usage of the Chef toolset
• Build further abstractions on top of Chef
• Give us superpowers that Chef doesn’t have yethttp://github.com/infochimps-labs/ironfan
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 11
What Does Ironfan Do
Ironfan
Chef
Simple helpers in the silverware cookbookabstract common Chef patternsand keep things DRY.
What Does Ironfan Do
Dynamic service discovery:
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 13
What Does Ironfan Do
A simple DSL for defining clusters of machines.
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 14
Big Data for Chimps
May 2013
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 15
As we walk through Ironfan…
3/12/2013 #ironfan #devops #sxsw #bigdata #chef 16
• Shortlink: http://infochim.ps/15INnv8
FYI• We are hiring! (we have offices in Austin
& SF)– [email protected]– infochimps.com/careers
• Learn more about our enterprise product:– [email protected]
