SaaS for the Enterprise

Automated, cross-‐pla1orm so2ware delivery

Eric Sproul Circonus !OmniTI Surge, September 2015

SaaS for the Enterprise

SaaS for the Enterprise | Eric Sproul | Surge 2015

Eric Sproul

@eirescot

Release Engineering at Circonus

• Con5nuous {Integra5on,Delivery}

• Packaging

• Deployment Orchestra5on

/me

2

What We'll Cover• Brief overview of Circonus, our challenges and philosophy

• Our automa5on and release processes

• How OmniOS packaging works

• How we deliver on-‐premise bits


API-‐driven, scalable monitoring and data analysis

Circonus Overview

• Comes in two flavors: public SaaS, private/on-‐prem SaaS

• Distributed architecture

• PlaLorms: OmniOS, RHEL/CentOS

4


Public Private

• Self sign-‐up

• Nothing to install

• Circonus runs the One True Copy

• 6 billion data points/day, and growing

• Operate your own full copy

• Data stored on-‐premise

• Economies of scale

• Policy compliance

SaaS All The Things!

5


~20 loosely-‐coupled components

Distributed Architecture

• Web servers, caches

• Fault detec5on, no5fica5on

• Message queues

• REST APIs

• Proprietary distributed 5me-‐series data store

6


~300 individual so=ware packagesStarted as just public SaaS

• 6 language run5mes

• C, Perl, Java, Node.js, Erlang, LuaJIT

• >90% are dependencies for the main components

• Need to deliver everything across OSes, constantly

• Right tool for each job

• For so_ware you ship to someone else, it's da_

• But here we are

Big Stack, Big Challenges

7


OS differences represent risk, both to dev and ops

Keep Your {S**t,Stuff,So2ware} To Yourself

KYSTY

• Differing versions, paths complicate development and deployment

• Seek minimal dependence on host OS so_ware (libc, zlib, openssl)

• Custom install path (e.g., /opt/circonus)

• Control update cadence

8


Package Everything

• Capture state consistent with public SaaS

• Use na5ve plaLorm tools

• Achieve cross-‐plaLorm consistency, repeatability

hEps://flic.kr/p/56V

nBv

https://flic.kr/p/56VnBv



























































• Brief overview of Circonus, our challenges and philosophy





We use Jenkins CI

Con5nuous Integra5on

• Component applica5ons built per commit or on regular schedule

• Run test suites

• Kick off downstream package build jobs

• Build system based on OmniOS userlandhhps://github.com/omni5-‐labs/omnios-‐build

• One script can produce either IPS or RPM package

11

https://github.com/omniti-labs/omnios-build


Con5nuous Integra5on

12

reconnoiter-‐omnios

<commit>

reconnoiter-‐el6

broker-‐omnios-‐package broker-‐el6-‐package

Build/test

Package


Typical dev -‐> stage -‐> prod cycle

Public SaaS

Con5nuous Delivery

• We deploy from master

• Developers responsible for pushing their own changes

• SaaS web/API is a checkout in stage & prod due to volume, pace of change; remainder is packages

• "Fail forward"

13


EnTrely OS packages, weekly cadence

On-‐Premise Releases

• Friday

• roll web package, encompassing all changes that week

• update ga5ng package, capturing all package changes for the week

• Monday

• assuming no issues, run promote script for each plaLorm

14






IPS

Reference pla1orm for Circonus

OmniOS Packaging

• Image Packaging System, aka "pkg(5)"

• Created by Sun for OpenSolaris, replaces SVR4

• Now used by OmniOS, OpenIndiana, Oracle Solaris

• Transac5onal, metadata-‐driven and integrated with ZFS

• Network-‐based, flexible search grammar

• Changes-‐only updates

16


FMRIs in IPS

17

pkg://circonus/web/[email protected],5.11-0.151006:20150918T040731Z

circonus Publisher (an en5ty that creates packages)

service Name (basename)

0.1.1442504613,5.11-0.151006:20150918T040731Z Version

pkg Scheme

web Category (can be arbitrarily deep)

Fault-‐Managed Resource IdenRfier


Designed for easy parsing

FMRI Version Breakdown

• Strictly numeric comparison, split on punctua5on

• Comparison is le_ to right

18

0.1.1442504613,5.11-0.151006:20150918T040731Z

0.1.1442504613 Component Version ("the so_ware's version")5.11 Build Version (OS release, aka `uname -‐r`)0.151006 Branch Version (distro-‐specific meaning)20150918T040731Z Timestamp (ISO 8601)


Describes a specific version of a package

Package Manifest

• Collec5on of ac5ons that deliver files, dirs, links, dependencies, etc. via ahributes

• Ahributes are key-‐value pairs

• Fully describes state of package upon installa5on

• Programma5c transforms

19


Package Manifest

20

set name=pkg.fmri value=pkg://circonus/web/[email protected],5.11-0.151006:20150918T040731Zset name=pkg.summary value="Front-end web services"set name=pkg.descr value="Superset of code needed for web-related roles"set name=pkg.human-version value="4bd369c from Thu Sep 17 15:43:33 2015 UTC"set name=publisher [email protected] group=bin mode=0755 owner=root path=www/lib/Circonus/Account...file 09a7c5d0a7b948c98e6085351e04ab0cef966d0e chash=55e2136c03fc47969ed4ac025538b93da5cd2739 group=bin mode=0644 owner=root path=www/lib/Circonus/Account.pm pkg.csize=14599 pkg.size=67965...link path=www/htdocs/s/tz/zoneinfo/GMT_Offset/UTC.json target=../UTC.json...depend fmri=web/service-support type=require


Dependency Types

• Require

• the referenced package provides essen5al func5onality

• including a version sets a "floor" (minimum version)

• Op5onal

• non-‐essen5al, but if installed, must meet version constraint, if any (same as require)

• Incorporate

• like op5onal, but sets "ceiling" as well as "floor" to the given degree of precision

21


Versions in Dependencies

require

op5onal

any version of foo

library/foo

foo >= 2

library/foo@2

foo >= 2.1

library/[email protected]

22


Versions in Dependencies

23

incorporate

foo 2.x, not 1.x or 3.x

library/foo@2

foo 2.1.x, not 2.0 or 2.2


exactly foo 2.1.2



circonus-‐incorporaTon: gaTng package

Hint: they are preVy cool

Incorpora5ons in Prac5ce

• Installed before other things

• Establishes allowed versions of dependents prior to their installa5on

• Dependent updates disallowed un5l incorpora5on itself is updated

• Incorpora5on update moves all dependents forward in unison

24


Incorpora5on Manifest

25

set name=pkg.fmri value=pkg://circonus/field/[email protected],5.11-0.151006:20150908T145421Zset name=pkg.human-version value="dea2423:Tue Sep 8 14:54:11 2015 UTC"set name=pkg.summary value="Circonus release incorporation"set name=pkg.descr value="The circonus-incorporation package constrains the versions of Circonus and dependent third-party software to a supported configuration."...depend fmri=pkg://circonus/field/[email protected],5.11-0.151006 type=incorporatedepend fmri=pkg://circonus/platform/[email protected],5.11-0.151006 type=incorporatedepend fmri=pkg://circonus/web/[email protected],5.11-0.151006 type=incorporatedepend fmri=pkg://circonus/web/[email protected],5.11-0.151006 type=incorporate

Delivers no filesystem content

Just sets dependencies on allowed versions


Incorpora5on Update

26

# pkg update -nv pkg://circonus/field/circonus-incorporation Packages to update: 5 Estimated space available: 1.71 TBEstimated space to be consumed: 72.03 MB Create boot environment: NoCreate backup boot environment: Yes Services to change: 1 Rebuild boot archive: No!Changed packages:circonus field/circonus-incorporation 0.2.1440769730,5.11-0.151006:20150828T134856Z -> 0.2.1442611311,5.11-0.151006:20150918T212156Z field/fq 0.9.1,5.11-0.151006:20150709T180854Z -> 0.9.2,5.11-0.151006:20150902T152622Z platform/snowth-perl 0.1.1435086592,5.11-0.151006:20150623T191609Z -> 0.1.1441380339,5.11-0.151006:20150904T153106Z web/reconnoiter-docs 0.1.1440615780,5.11-0.151006:20150826T190835Z -> 0.1.1442325587,5.11-0.151006:20150915T140433Z web/service 0.1.1440684644,5.11-0.151006:20150828T040745Z -> 0.1.1442504613,5.11-0.151006:20150918T040731ZServices: restart_fmri: svc:/network/fq:default






Need to deliver the same updates on Linux as OmniOS

About That Cross-‐PlaLorm Thing...

• Recall that we roll packages for both OSes on each code change

• Build system translates names from FMRI to RPM-‐acceptable form

• "pkg://circonus/web/service" -‐> "circonus-‐web-‐service-‐*.rpm"

28


...about incorporaRons

If wishes were fishes... !

RPM has no noTon of a "version ceiling" !

yum versionlock can only funcTon once something is installed, difficult to manage updates

RPM Don't Care

29

hEps://flic.kr/p/3f12JL

https://flic.kr/p/3f12JL


yum is greedy, wants whatever's latest in repo

Update like it's 1999

Handling Updates for Linux

• Place matching versions into prod repo when the OmniOS incorpora5on moves forward

• Scripted promo5on procedure

• read incorpora5on details, translate names to RPM

• determine list of RPMs to move from stage to prod repo

30


Start with OmniOS

Promo5ng to Stable

• Iden5fy the version of circonus-‐incorpora5on corresponding to this week's update

• Promo5on script copies this version from staging to prod repo

• Dependent packages already in prod repo, but users not allowed to install them un5l they update

31


Linux

Promo5ng to Stable

• Refer to the same version of circonus-‐incorpora5on as in OmniOS

• Promo5on script reads the JSON document produced by the incorpora5on update, translates names to RPM style

• Finds latest RPM rev at specified version for each package

• Copy RPMs from staging to prod yum repo

32


FMRI to RPM

33

"web/service" : { "component-version" : "0.1.1442504613", "fmri" : "pkg://circonus/web/[email protected],5.11-0.151006", "human-version" : "4bd369c from Thu Sep 17 15:43:33 2015 UTC", "publisher" : "circonus", "version" : "0.1.1442504613,5.11-0.151006" },

JSON info from incorpora5on

$ rpm -qpi circonus-web-service-0.1.1442504613-1.el6.x86_64.rpm Name : circonus-web-service Relocations: / Version : 0.1.1442504613 Vendor: Circonus, Inc.Release : 1.el6 Build Date: Fri 18 Sep 2015 04:28:52 AM UTCSummary : Superset of code needed for web-related roles (4bd369c from Thu Sep 17 15:43:33 2015 UTC)Description :Superset of code needed for web-related roles (4bd369c from Thu Sep 17 15:43:33 2015 UTC)

Promoter script locates the corresponding RPM


Fix UI bug on a Tuesday

1. Tue: test fix in dev

2. Tue: push to staging, test again

3. Tue: deploy to prod (public SaaS)

4. Fri: scheduled web/service pkg build

5. Fri: web/service update included in circonus-‐incorpora5on update

6. Mon: promote incorpora5on

From development to on-‐premise availability

Example Change

34


Distributed installer/updater for on-‐premise version

(Ab)using configuraRon management

Delivering the Goods

• chef-‐solo, delivered as an OS package (what else?)

• Site-‐local configura5on provided as Chef data bag (JSON file)

• assigns product component roles to specific hosts

• exposes user tunables

35


Distributed installer/updater for on-‐premise version

(Ab)using configuraRon management

Delivering the Goods

• Via site config, every node knows where all other roles are

• Templates config files for all services running on current node

• Knows how to start services and recover those not in the correct state

• Performs package updates to stay in sync with latest from public SaaS

36

QuesRons?

www.circonus.com

Stop by the booth and say hello

http://www.circonus.com

Internet

SaaS for the Enterprise