DevOps, CI, APIs, Oh My! - Texas Linux Fest 2012

DEV/OPS, CONTINUOUS DEPLOYMENT & APIS, OH MY!

Matt Tesauro, Texas Linux Fest – San Antonio, TX, August 2012

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

WHO AM I?Matt Tesauro – Cloud Application Security Guy + OWASP

2

[email protected]

[email protected]

Racker since October 2011

Rackspace’s Cloud Product Group

Work with developers and QE

OWASP International Foundation Board

Member and Treasurer

Project Leader of OWASP Live CD &

OWASP WTE projects


RACKSPACE® HOSTINGThe Service Leader in Cloud Computing

3

172,000+CUSTOMERS

4,000+ RACKERS

9 GLOBAL DATA

CENTERS

120 +COUNTRIES

2008, 2010, 2011 & 2012

LEADER IN GARTNER'S MAGIC

QUADRANT FOR MANAGED HOSTING

40%FORTUNE® 100

OFTHE

WE SERVE

RAX

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4

OUR VISIONTo be recognized as one of the World’s greatest service companies.“ ”


RAX CLOUD APPROACHOpen source orchestration, management & provisioning cloud platform


PROVIDER DCCUSTOMER SITERACKSPACE LOCATIONS

6

DEDICATED PUBLIC CLOUD PRIVATE CLOUD PRIVATE CLOUD PUBLIC CLOUD

Rackspace ProvidesThe Fanatical Support

• One Control Panel across OpenStack connected clouds• One Fanatical Support Team• Our Cloud, Your Cloud, Partner Hosted OpenStack Cloud• Global Reach

THE FUTURE: FANATICAL SUPPORT ANYWHERE

6


SECURING APPS IN A DevOps WORLD


A quick Overview of DevOps

• The combination of traditional development activities with operations and testing (QA/QE)

• Collaboration, communication and integration is key

• Agile development model (sprints, scrum, …)

• Release coordination and automation

"DevOps" is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals.


CI, CD, CD, TDD and API

CI == Continuous Integration

CD == Continuous Deployment

CD == Continuous Delivery

TDD == Test Driven Development

API == Application Programming Interface


10

• Cycle time for software is getting shorter

• Continuous delivery is a goal

• Scanning windows are not viable

• First mover / first to market advantage

THE PROBLEM

10


THE PROBLEM – or at least more

• Traditional software development left little time to test

• DevOps, Agile and Continuous Delivery squeeze those windows

even more

• New languages and programming methods aren’t making

this better

• Growth of interpreted languages with loose typing

hurts static analysis efforts

• Few automated tools to test APIs especially

RESTful APIs

• Little time for any testing, manual testing is doomed


12

• Automated software testing

• Automated operational infrastructure

• Automated security testing

THE SOLUTION

12


Think like a developer

Sprints break software into little pieces…

• Break your testing into little pieces

• Use your threat model to know the crucial bits to test

Long and short running tests

• Testing time drives testing frequency

• Code for tests needs to be optimized

Smoke test versus full regression test

• Smoke test early and often

• Full regression tests on regular intervals


Maximize what you’ve got

Make the most of your frameworks• Embrace, understand and fill gaps where necessary

Make the best use of your time…

• Make tests easily repeatable

• Make tests easy to understand

• Make tests abstract and combine-able

• Ala carte tests for mixing and matching

• Think about the Unix pipe | and its power


Test Driven Development Security

Under the constraints of DevOps, Continuous Deployment

Your testing has to be nimble

Dare I say…Agile

In TDD, you know your code works

when the tests pass

In TD(S), you know your app has met

the baseline when the tests pass


A snail on fire!


17

AUTOMATING

17

• Declarative configuration language• Plain-text configuration in source control• Fully programmatic, no manual interactions


18

CHEF

18

1. Solo

2. Server

3. Hosted

4. Private Hosted NodeNode

NodeNode

Node

NodeNode

NodeNode

Node

NodeNode

NodeNode

Node

Racker

Server / Hosted / Private


19

COOKBOOKS

19

• Most major software packages have cookbooks

• You will have to write your own / customize

• Good place to spend security cycles

-Merge patches upstream for extra points.


20

GROUPING & TAGGING

20

• Tagging your servers applies the required set of recipes

• A base set of recipes is common

• Each server will have multiple tags set at bootstrap time

NodeNode

NodeNode

DB

NodeNode

NodeNode

Cache

NodeNode

NodeNode

Web

Apache

Monitoring

MySql

Memcache


21

LIMITATIONS

21

• Focus on single machines• A multi-box configuration is based on copying existing configurations

• No support for implicit application or environment configuration

• Applications include more than just servers

• Images have security issues

Web Web Web

Cloud Load Balancer

MemcachedDatabase as a

Service

Web

Cloud Files CDN


22

CHECKMATE

22

A system to build generic application configurations

Architect• Templates• Questions

Contractor• Decomposition• Orchestration

Inspector• Verification• Due Diligence


23

ARCHITECTURE

23

Message Queue

Architect

Contractor

Inspector

Checkmate Web

Message Queue

Compute

Storage

Load Balancer

Database

Hadoop

Caching

• Components communicate through a common queue

• Each provisioning component is independent


24

ARCHITECT

24

Template

Generic Provider Definitions

Architecture Questions

Scaling Factors

base:

name: wordpress large

environment-name: {tenantId}-wordpress-large

providers:

- rackspace:

- compute: &rax-cloud-servers

endpoint: https://...

- loadbalancer: &rax-lbaas


- database: &rax-dbaas


- common:

vendor: rackspace

credentials:

- token: {token}


25

ARCHITECT

25

Template



Scaling Factors

• Requests per hour?

• Budget

• High availability

• Disaster resistant

• SSL

• Backup

• CDN

…


26

ARCHITECT

26

Template



Scaling Factors

tiers:- name: web resource: &loadbalancer min-occur: 1 type: loadbalancer connection: public port: [80, 443] allow: all isolation: none resource: &webheads min-occur: 2 type: compute os: Ubuntu 11.10 memory-min: 2Gb memory-max: 4Gb configs: - wordpress-mp attributes: - role: web connection: *database











37

CONTRACTOR

37

• Takes Architect’s plan and builds it

• Task Decomposition

-Uses standard workflow patterns

• Orchestration / Ordering

• Status Reporting

• Farms out tasks to sub-contractors Our current implementation uses an open source

Python workflow engine, SpiffWorkflow.


38

INSPECTOR

38

• Takes Architect’s plan & contractor’s output

• Focuses on checking for code compliance

-Not perfection, bare minimums

• Can include multiple facets

-Security

-Scalability

-ComplianceOur current implementation includes WP Scan for

WordPress and the Nikto vulnerability scanner.


39

INSPECTOR

39

+ Server: Apache/2.2.12 (Ubuntu)+ No CGI Directories found (use '-C all' to force check all possible dirs)+ Apache/2.2.12 appears to be outdated (current is at least Apache/2.2.17). Apache 1.3.42 (final release) and 2.0.64 are also current.+ ETag header found on server, inode: 12534048, size: 317, mtime: 0x4b9436dbea280+ Allowed HTTP Methods: GET, HEAD, POST, OPTIONS + OSVDB-3268: /icons/: Directory indexing found.+ OSVDB-3233: /icons/README: Apache default file found.+ 6448 items checked: 0 error(s) and 5 item(s) reported


40

INSPECTOR

40

[!] The WordPress "http://---.com/readme.html" file exists.[!] WordPress version 3.1 identified from meta generator.

[+] Enumerating installed plugins...Checking for 2394 total plugins[+] We found 2 plugins:Name: disqus-comment-systemLocation: Name: wordpress-popular-postsLocation:

[+] There were 1 vulnerabilities identified from the plugin names:

[!] ["WordPress Plugin Disqus Comment System <= 2.68 Reflected Cross-Site Scripting (XSS)"]*


Architect• Templates• Questions

Contractor• Decomposition• Orchestration

Inspector• Verification• Due Diligence

Monitor• Trending• Thresholding

41

FUTURE WORK

41


So I was talking with a friend…

He was bemoaning the pace of change and the speed at which software was being pushed to production…

In essence, management has made the decision that getting their app out the door with possible bugs is more valuable to the business then having strong assurance that the software has few or no significant bugs.

You’ve got to up your game, get automated, agile and get on pace with your developers.

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

ANY QUESTIONS?

Technology

DevOps, CI, APIs, Oh My! - Texas Linux Fest 2012