Upload
puppet-labs
View
5.004
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Attend this talk and learn how Puppet Labs handles community contributions and the FOSS tools we’ve published to automate much of the process. Puppet Labs handles thousands of contributions from hundreds of contributors and we’ve integrated Github, TravisCI and Trello to manage all of it. Come see how we do it and what we’ve built! Jeff McCune Software Developer, Puppet Labs Recovering systems engineer turned software engineer, responsible for helping the Puppet community contribute their code to our open source projects.
Citation preview
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 1/29
Automating Community ContributionsJeff McCune
Twitter: @0xEFFEmail: [email protected]
http://puppetlabs.com
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 2/29
We'll Cover:Process
Challenging 2012 process
Improved 2013 process
Automation & Tooling
Github (patches)
Travis (tests)
Trello (work items)
Puppet Webhooks (automation)
Questions
Please ask questions during or after the talk, there will be time.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 3/29
Merge more patches with automationDedicated people work well
Context switching is a challenge
Automation
Reduce context switchingSingle pane of glass
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 4/29
2012 ProcessPlatform Team
Bug Fixes
New Features
Security Incidents
Community Pull Requests
Two types of work
Planned - Bugs and Features
Unplanned - Pull Requests and Security
Pull requests often came last in the priority list
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 5/29
2012 Process OutcomeLarge and growing backlog
No response for months
Sprints to "catch up"
Public complaints from key community members
Stressed out team
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 6/29
2012 Results
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 7/29
2013 ProcessCommunity Team
Community Pull Requests
Goal: 2 PM next business day response
One type of work, not two
Reduced context switching
Platform Team
Bug Fixes
New Features
Security Incidents
Reduced context switching
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 8/29
2013 ProcessCommunity Team responsibilities:
Community Pull Requests
Platform Team responsibilities:
Bug Fixes
New Features
Security Incidents
One type of work:
Unplanned - Pull Requests and Security
Pull requests are consistently prioritized.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 9/29
2013 ResultsWe're merging ~ 15 community patches per week
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 10/29
Process ConclusionDedicated People
Minimize context switching
Decreased backlog, faster response time.
Backlog down to 25 from 80
New pull requests responded to within two days instead of months.
Automating the processWith a solid process in place, we focused on automating the tedious aspects.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 11/29
Work in Flight
Trello
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 12/29
Automation and ToolingGithub (patches)
Travis (tests)
Trello (work items)
Three separate tools, lots of switching back and forth.
Puppet Webhooks puts it all in once place.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 13/29
Why more Tooling?Too many sources of information are tedious and error prone to review.
Issue Tracker (Should I work on this now?)
Jenkins and Travis (Does this patch break stuff?)
Github (What are people saying about this patch?)
CLA (Has the contributor signed the license agreement?)
Trello (What should we be working on?)
All of this information is important. It's also spread out on four or more different sites. Itwas often overlooked.
A solution is to move all of the information into one place.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 14/29
All information in TrelloOur goal is to get all of the information we need to review a pull request into Trello.
New pull request creates a card.
Code comments
Contributor bio
CLA Information
select(contributor)
Completed this week
Puppet, Facter, Hiera, stdlib on one board
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 15/29
Gepetto BotGepetto Bot is a service account with a profile on Github, Trello, and Heroku. Thepuppet-webhooks Heroku app logs into these services as Gepetto Bot.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 16/29
Consolidated InformationGepetto Bot gathers contributor bio using the Github API:
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 17/29
Board ColumnsCards move left to right. Gepetto Bot creates them in the Response Needed column.
Response Needed
Accepted
Doing
Waiting on Contributor
Waiting > 1 week
Going through CI
Finished this Iteration
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 18/29
How it worksWebhooks
Github executes a callback when events happen. The callback is a very simple HTTPPOST with a JSON body containing data about the event.
Think push notifications for the web.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 19/29
How it worksThe Puppet Webhooks code running in Heroku receives the HTTP POST and:
1. Github posts JSON to our Heroku app2. Persist the JSON to a Delayed Job queue3. Start a Delayed Job worker process using workless gem4. Data is posted to Trello in a matter of seconds
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 20/29
Why use Delayed Job?Heroku provides:
Small PostgreSQL databases for free
750 hours of process time for free
31 x 24 = 744 hours for the web process
6 "extra" hours for free
Automatic retries with back-off
The workless gem allows us to only run DJ worker processes when there's actuallywork to be done.
Note: While this is all free of charge, a credit card is required to use the Heroku APIwith workless.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 21/29
Automatic Process ManagementThe web process runs persistently.
bundle exec thin start p $PORT e $RACK_ENV
The worker process runs only when there's work to be done.
bundle exec rake jobs:worksilent
Workless automatically adjusts the worker dyno:
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 22/29
What is automated?New Pull Request => New Trello Card
Code Comment => Trello Comment
Collect Contributor Bio
Card due Date
Weekly Summary of completed cards
Unimplemented event handlersPull Request is closed
New code is pushed
Travis CI tests complete
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 23/29
Weekly summary emailSent to puppet-dev mailing list weekly
Automated summary of completed Trello cards
Scans card comments and extracts summary: matches
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 24/29
How the weekly summary worksGithub Gist as a persistent data store!
1. App reads a Liquid template from a raw github URL2. Collect all completed Trello cards3. Fill in the template with the card data4. Post the resulting Markdown back to the Gist
No persistent state in the app or the database. Liquid template is easily updated.
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 25/29
Summary TemplateHeader:
## Puppet Dev Community Summary
* Finished Card Total: **{{ cards.size }}**{% for section in sections %} * {{ section[0] }}: {{ section[1].size }}{% endfor %}
Sections:
{% for section in sections %}## {{ section[0] }}
Cards:
{% for card in section[1] %}#### [{{card.title}}]({{card.url}})
{{card.message }}
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 26/29
Post back to Githubheroku run bundle exec rake jobs:summary
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 27/29
Tooling Wrap UpAutomate the Tedious Stuff
Try the Puppet Webhooks App
github.com/puppetlabs/puppet-webhooks
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 28/29
What we coveredProcess
Context Switches = (ಥ_ಥ)
Dedicated People = (◕‿◕)
Automation & Tooling
Github, Travis, Trello
Puppet Webhooks automates all three
Key Message
Merge more patches with automation
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 29/29
Thank You! Questions?
Jeff McCune0xEFF