Upload
fabian
View
42
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Jenkins Scalability Summit. Oct 2013. Logistics. Where is …? Network. Agenda Part 1: Story time. Learn from our collective experience Identify high-priority problems What is hurting serious users? Concrete details are good. Agenda Part 2: Discuss & Design. - PowerPoint PPT Presentation
Citation preview
©2013 CloudBees, Inc. All Rights Reserved 1©2013 CloudBees, Inc. All Rights Reserved
Jenkins Scalability SummitOct 2013
©2013 CloudBees, Inc. All Rights Reserved 2
Logistics• Where is …?• Network
©2013 CloudBees, Inc. All Rights Reserved 3
Agenda Part 1: Story time• Learn from our collective experience• Identify high-priority problems• What is hurting serious users?• Concrete details are good
©2013 CloudBees, Inc. All Rights Reserved 4
Agenda Part 2: Discuss & Design• Collectively pick a few topics• Split into 2 tracks
• See if we can start shaping up solutions
• Please keep shared notes– Details!
©2013 CloudBees, Inc. All Rights Reserved 5
Possible topics includes …• CPU/memory/disk/network/… consumption• Stability/diagnosability of slaves• Access control of builds/UI• Organizing jobs and build records• Master to master communication• Workflow / choreography• User interface• Stability of Jenkins releases• Plugin compatibilities
©2013 CloudBees, Inc. All Rights Reserved 6
Goals• No one goes listen-only mode• Produce notes to show to the broader
community• Figure out how to do this better next
year
©2013 CloudBees, Inc. All Rights Reserved 7©2013 CloudBees, Inc. All Rights Reserved
What We’ve Done&
What We Can
©2013 CloudBees, Inc. All Rights Reserved 8
Thread Reduction• Per-slave consumption– SSH (down to 1)– Channel (down to 1)
• Executor thread on demand
©2013 CloudBees, Inc. All Rights Reserved 9
Thread Reduction: TODO• NIO• Asynchronous job execution–More about this in workflow
©2013 CloudBees, Inc. All Rights Reserved 10
Memory reduction• Lazy loading of build records• Database plugin
©2013 CloudBees, Inc. All Rights Reserved 11
Design Choices That Constrain Us• Compatibility– Adding is doable, changing is often hard
• Heterogeneousness in data model• Thread-driven execution model
©2013 CloudBees, Inc. All Rights Reserved 12©2013 CloudBees, Inc. All Rights Reserved
CloudBees Scalability Efforts
©2013 CloudBees, Inc. All Rights Reserved 13
Multi-master / Meta Jenkins• Master-to-master channel• Extensions on top of it– “Cloud” impl to lease slaves– Push security realm– Push update center–…
©2013 CloudBees, Inc. All Rights Reserved 14
Multi-master• Where we think we can go with this–Move jobs around– Sharding with UI mashup
• Where this wouldn’t take us– Loss of inflight builds– True horizontal scaling
©2013 CloudBees, Inc. All Rights Reserved 15
Workflow• Brand new job type• Kill multiple birds in one stone• Inspired by buildflow & jenkow
©2013 CloudBees, Inc. All Rights Reserved 16
Ingredients• No single thread to execute the whole
thing– Analogous to NIO– Check pointing
• BPMN-like workflow execution model– Surface syntax independent
• Groovy DSL–With continuation-passing style execution
model• Post-execution visualization– But no pre-execution visualization
©2013 CloudBees, Inc. All Rights Reserved 17
Ingredients• Unified properties, build variables, and
environments• Open up direct access to file store• Hands-free process forking• Interop with existing job types