Constant Contact SF's Road to CD

Presented by :

Road to CD at Constant ContactIncreasing Velocity

1

Agenda

• Life Before Continuous Delivery • Discovering and Implementing Continuous

Delivery • Refining the Flow • Working with Continuous Integration/Delivery • Evolving Continuous Delivery

2

Quick Overview of Engineering at Constant Contact

3

Engineering

• About 15-18 engineering teams each with an average size of 8-12 engineers

• Typical ratio of 1 Quality Engineer per 2-3 Software Engineers

• Teams function as either Feature teams, Platform teams or a combination of both

4

Engineering

• Distribution includes: • Waltham, MA (Head Office) • Tribeca, NY • Battery Park, NY • San Francisco, CA • Boca Raton, FL • Loveland, CO

5

Life Before CD

6

Life Before CD

• Weekly deploys • Build pipelines owned and operated by

Engineering Services • All deploys had to go through a request and

queue process • Progressing through 5 different environments

• Development, Functional, Load, Stage, Production

7

Life Before CD

• Engineers coding large units of work in the Development Environment

• Code was then deployed to the Functional environment

• Testing implemented and executed by the Quality Engineers in the Functional testing Environment

• Integration testing may happen in the Functional or Load Testing Environment

• Once all tested code progressed to the Stage Environment

8

9

Backlog In DevelopmentReady for Dev Ready for

ValidationIn

ValidationReady for

Deploy Deployed

10

Development LoadFunctional Stage Production

30% Stable

80% Stable

90% Stable

99% Stable

99% Stable

Life Before CD

• Visibility on what was being deployed was very general

• We knew work was going out but troubleshooting issues tended to be tricky due to large commits

• Kanban SDLC • Multitasking was a big thing

11

Discovering and Implementing Continuous Delivery

12

The Big Question

13

Howdo we start?

Where

What is everyone else doing?

• We first looked at how other teams had approached CD

14

New York, NY Office

• Pre-emptive CD

15

Waltham, MA Office

• Integrated CD

16

We set goals!

• Development teams own their applications • Build and deploy through to production

• Deliver quality code minimising defects through improved test coverage

• Increase velocity by 20% by the end of 2013, another 50% by the end of 2014

• Direct to production deploys by the end of 2014

17

We formed a CD Team

• Main body of governance for CD • Core development and assistance of integrations • Goto team for anything CD • Consisted of:

• Product Owner • Dev Manager • Architect • QE

• Engineering was utilised form the team adopting CD

18

Planning

• Key to success in implementing CD • Where are you deploying? • What are the steps to a successful deploy? • What are the effects of one event to another? • How will engineers engage with CD? • How will this affect existing product work? • What are the milestones you want to deliver to

a team? • Comes even before thinking of tooling

19

The Approach

• Convert to a single-branch model • Hard version all artefacts (produced and

consumed) • Harden test coverage • Set up the skeleton build jobs

• Simple Checkout -> Build -> Unit Test -> Publish -> Deploy Pipeline

• Deploy to 1 environment only then backfill

20

The Approach

• We didn’t want to manage multiple environments any more just 1 environment!

• Implement the practice of Feature Switches • Coach the Product owners how to write User

Stories

21

Working with Continuous Delivery

22

Technology Stack

• Jenkins • Open Stack (New) • LDAP • Puppet • GitHub • Artifactory (New)

23

Single Branch Model

24

Release

Feature 1

Feature 2

Single Branch Model

25

Release

Feature 1

Feature 2

Build Validation

Single Branch Model

26

Release

Feature 1

Feature 2

Pull Request Validation

Build Validation

Integration Testing

• Whole environment configured to execute Integration testing

• Separate Jenkins Server • Separate OpenStack Group • Owned by the QE team

27

28


CD Deploy Manual Deploy

Manual Deploy

Manual Deploy

Manual Deploy

29


CD Deploy

Backfill

Manual Deploy

Manual Deploy

Manual Deploy

1 OS VM

30


CD Deploy

Backfill

Manual Deploy

Manual Deploy

Backfill

1 OS VM 2 OS VMs

31


CD Deploy

Backfill

Push Button Deploy

Backfill Backfill

30% Stable

80% Stable

90% Stable

99% Stable

99% Stable

1 OS VM 2 OS VMs

4 OS VMs

32


CD Deploy

Backfill

Push Button Deploy

Backfill Backfill

30% Stable

80% Stable

90% Stable

99% Stable

99% Stable

1 OS VM 2 OS VMs

4 OS VMs

6 OS VMs

Refining the Flow

33

Automating Servers

• Ensure that domains are properly configured • Setting up and pulling down instances in a

coordinated fashion • Attaching to the correct environment

34

Monitoring & Alerts

• Jenkins Job health plugin • Healthcheck dashboard • Splunk Logging • Nagios

35

Results of First Deliverable

• We had direct to Load deploys happening with backfill

• Test coverage up • New SDLC with QEs engaging at the beginning

of the flow, not waiting for engineers to complete work

• New way to view the health of the system through monitoring

• Base skeleton workflow that would be used to build new pipelines

36

Discoveries

• There is no main entry point when starting CD • Planning is crucial • Your SDLC will most likely change • Monitoring is key to knowing your progress • Educating the users of the system is also

paramount • Everyone is involved with CD

37

Discoveries

• CD is evolutionary • It will become part of your projects, not a

separate project • It will help your velocity just not right away • Don’t be afraid to innovate

38

Working with CD

39

SDLC

• Scaled Agile Framework • Initially taken from v1.0 • More recently v3.0

• Had to change • Epics -> Features -> Stories (-> Tasks)

40

SDLC

• Needed better tracking of what was happening now

• Velocity increased • A new development programme had to be put

together • Moved from Kanban to Scrum

41

SDLC

• Introduced 3 flows in Jira: • Epic Flow • Feature Flow • Scrum/Story Flow

42

Evolving CD

43

2015

• Examine the gaps and address them • Better integration with GitHub, Jenkins and Jira • Improve reporting and Release Notes • Try and automate the whole process from

changing a Jira Ticket status to building out a whole environment

44