Building for operations - Reinhardt Quelle

Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.

Building Applications for Operability

Reinhardt QuelleOperations [email protected]

rquelle@*

1

V16e


scr

© 2011 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) 3

Great stuff, but not that’s not what my team does…

© 2011-2012 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])

We host and manage SaaS Applications

4

Millions of Meetingsfor

10s of Millions of Userstotaling

Billions of Minuteseach month

7x24x365

Cisco Social

WebEx

Jabber


DevOps

Damon Edwards (DTO)John Willis (now at Dell) suggest

“CAMS” as useful mnemonic.


DevOps > Tools

•Automation


DevOps > Tools

•Automation

•Metrics


DevOps > Tools

•Automation

•Metrics

•Sharing


DevOps > Tools

•Culture

•Automation

•Metrics

•Sharing


DevOps > Tools

•Culture / Collaboration

•Automation

•Metrics

•Sharing


Collaboration

Requires:

• shared goals

•communication

•empathy

•


Yay, We’re going to do DevOps!

http://www.realclearworld.com/blog/2009/01/


When you start, its more like this:

http://www.news.com.au/world/are-barack-obama-and-vladamir-putin-holding-highlevel-talks-or-are-they-on-an-awkward-date/story-fndir2ev-1226665803294


At a more personal level: remember that first date?

http://www.hercampus.com/love/dating-hooking/9-first-date-disasters-how-deal-them


Where do we begin?


Perspective helps (a lot)

http://www.navigonlp.com/blog/stop-look-look-again/boat-land-perspective-2/


I come from an Operations background


Green = Good


Illustrating the Ops perspective to Engineering

http://cdn.twentytwowords.com/wp-content/uploads/Rhinos-Perspective-e1369734532511-634x447.jpg


A quick illustration:A real, albeit deceased, product

(RIP)


Developer’s View

Compute Node

config A

resolv.conf

ntp.conf

smb.conf

...

config BApp


Operations’ View

Compute Node

config A

configsshd

configrsyslog

configinetd /etc/sysconfig/*

/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BApp


Servers run in "clusters"

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

start/stoporchestration

config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App


Which means they may have to share data:

spare

LUN

SAN

LUN LUN LUN

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App


Which means they may have to share data:

spare

LUN

SAN

LUN LUN LUN

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Aconfig B

App

Note: there are *much* better ways to share data!


spare

LUN

SAN

LUN LUN LUN

Postpath Unified Node (PUN)

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BService X Service Y

config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

We have several front-end servers


...which also have to be redundant.

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM


spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Load Balancer

Webmail RPC ADM

And that means load balancers to direct traffic


spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

There are multiple clusters


... and geo-redundant data centers.

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

GSS


Which means data must be replicated - somehow.

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

GSS


But wait! There’s more!

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

GSS

Security Appliances

Monitoring Servers

Identity Servers

Billing Mgt Servers


But wait! There’s more!

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

Load Balancer

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

spare

LUN

SAN

LUN LUN LUN


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager


config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BPPSD

cluster manager

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Webmail RPC ADM

GSS

Security Appliances

Monitoring Servers

Identity Servers

Billing Mgt Servers

Security Appliances

Monitoring Servers

Identity Servers

Billing Mgt Servers


“Turtles all the way down”

eth0

hostname

/

NIC

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

Service Yconfig Bconfig A

hd

SAN

inst1

eth0 physical

physical hostname

/

NIC

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

Service Yconfig Bconfig A

/

eth0:1 cluster

eth1 mgt

/var..

inst2

eth2 iSCSI, NFS

service hostname

Even individual systems are not like your developer’s workstation:


Deployment Environment is Complex

• Many nodes in cluster = many “points” to monitor, including systems and services

• Dependencies on external services (DNS, Active Directory, SAN)

• Dependencies within application stack

• Sequencing of operations critical


So, lets just use PaaS!

ContainerAppA

$ cf deploy appname


Uh, Yeah.

DEA 1Ubuntu 12.04

DEA 1Ubuntu 12.04

DEA 1Ubuntu 12.04

DEA ContainerApp

A

Container

AppB

REST ServerUbuntu 12.04

RabbitMQCentOS 6.4 Cassandra

CentOS 6.4CassandraCentOS 6.4

CassandraCentOS 6.4

Cloud ControllerUbuntu 12.04

Ubuntu 12.04NATS

CF Router

RabbitMQCentOS 6.4

Internet

VHOSTA

CassandraCentOS 6.4

Keysp.A

LB/SSL

HTTPS

HTTP

HTTP/SData/Msg Bus

C/C: NATS/HTTP

Keysp.B

VHOSTB

UAM

HM

ContainerAppA'

Container

AppB'


I’m not going to discuss application architecture in detail• Designing for Always-on Services:

• We don’t ever stop serving customers (intentionally, anyway)

• Avoiding the Fallacies of Distributed Computing:

Emphasized by load and geography

• The need for loosely coupled services:

Multi-service synchronized updates are extraordinarily costly

• Embracing Failure:

At scale, something is ALWAYS breaking

• Designing for the real world

Circuit breakers, exponential fall-back, dynamic configuration

Read Web Operations (Alspaw, et al), Release It! (Nygaard)


Running Bits in ProductionWhere do we want the bits to run? Deployment Targets

How do they get there? Packaging/Distribution

How are they supposed to behave? Configuration

How do we start and stop them? Command/Control (aka Orchestration)

Are we sure they are running (or ready to run)?

Health Checks and Verification

When will we run into a wall? Metrics

Ow! A wall! Alerts

How did we end up at that wall? Logs

1

2

3

4

5

6

7

8


Note: “Monitoring” is multi-faceted

Where do we want the bits to run? Deployment Targets

How do they get there? Packaging/Distribution

How are they supposed to behave? Configuration

How do we start and stop them? Command/Control (aka Orchestration)

Are we sure they are running (or ready to run)?

Health Checks and Verification

When will we run into a wall? Metrics

Was that a wall we just hit? Alerts

What just happened there? Logs

1

2

3

4

5

6

7

8

Monitoring


Where does a package run?

Service ≠ Server• Multiple servers may run same service

• A single server may run multiple services - even several of the same kind (esp. 32bit)

1


One Service, Multiple Hosts

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

Front-End Server

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


config Bconfig A

LB


Application Compute Node

config A

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config BApp

Instance config A'config B'

AppInstance

Multiple Services, One Host


Deployment Target Choices

Where bits run:

Shared system

Virtual Machine

Dedicated server

May depend on:

Balancing Resources

Failover Strategies

Security or Isolation Changes over time

“Refactoring your Infrastructure”


Packaging/Distribution

• How is software Bundled, Transported and Unpacked/Installed

• Goal is to get a specific version of bits extracted onto a system in a predictable way with its dependencies

2


Distribution Choices

• Recommended:

• versioned and signed OS package (RPM, Deb) from controlled repository

• puppet “code” is in package, too - puppetfile is source

• Less desirable:

• direct version control repository (svn, git) checkout

• scp tarball

• Note there are different “units of deployment”

• Images, Packages, Applications


Configuration

• How is this instance supposed to behave?

• How is this server supposed to be configured?

• May be multiple instances on a given server

• Configuration is often changed over time

• Consider “static” and “dynamic” configuration

3


Standard OS Platform


configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD


+ Application


configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config Aconfig B

PPSD



configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

config Aconfig B

PPSD

Fail

uh, oh...


Application Configuration• Unique per application instance

• Does not overwrite system/environment

• Application switches:

Select Configapache2 -fnginx -cpostfix -c

Verify Configapachectl configtestnginx -tpostfix check

Reload Changed Configapachectl gracefulnginx reloadpostfix reload

Dump Config postconf


Various classes of configuration

• Static (ish)

• Environments: Dev, QA, Staging, Production

• Data Centers: SJC, DFW, NRT, AMS

• Clusters: ISJ1, AC

• Use Puppet/Chef/cfengine

• Dynamic

• Cluster membership

• Service hosts: DB, Logs, Metrics, Identity

• Use Zookeeper/Noah/Eureka/DNS - MX, GSS/Anycast


Common Practice:Configuration Management

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

configurationdaemon

config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

configurationdaemon

config Aconfig B

App

configurationmaster+

Environmental Data - Puppet Hiera - Chef Databags

Cluster Spec - * - TOSCA - CloudFormation - Heat

Machine Spec - Puppet Modules - Chef Cookbooks


Need a new Dev/QA test environment?

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

configurationdaemon

config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

configurationdaemon

config Aconfig B

App

mstr A

+Environmental Data - Puppet Hiera - Chef Databags

Cluster Spec - * - TOSCA - CloudFormation - Heat

Machine Spec - Puppet Modules - Chef Cookbooks

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

configurationdaemon

config Aconfig B

App

Compute Node

configsshd

configrsyslog


/etc/logrotate/*

/etc/snmp/*

/etc/passwd

...LVM

/etc/*

....

resolv.conf

ntp.conf

smb.conf

...

...DR:BD

configurationdaemon

config Aconfig B

App

mstr B

Environmental Data - Puppet Hiera - Chef Databags


Make applications configuration-friendly

Select Configapache2 -f /path/to/confignginx -c /path/to/configpostfix -c /path/to/config

Verify Configapachectl configtestnginx -tpostfix check

Reload Changed Configapachectl gracefulnginx reloadpostfix reload

Dump Config postconf


Make configuration cooperative and configuration management friendly:

logrotate include /etc/logrotate.d

monit include /etc/monit.d

sudo #include /etc/sudoers.d

Sensu“Deep Merge” of JSON config


Command and Control(aka “orchestration”)

• Starting and stopping services usually requires specific sequence

• By definition, cluster has multiple systems on which processes must be controlled

• Sequencing must be enforced, repeatable

• Dependencies must be managed

• Ad-hoc processes may be required on demand

4


Many tools available

• Fabric

• Capistrano

• MCollective

• RunDeck

• And many, many more


So now we’re running...

Sure about that?


Are We Alive?

• Health Checks provide specific status

• Often variable depth:

• shallow/frequent, deep/less frequent

• Many Consumers

• Cluster managers, load balancers, process monitors

• Personal favorite - “Pre-Flight checks”

• Call on demand

5


Now you can run, but...


Where are we headed?• Metrics help us understand trends and predict future...

• or give us historical context for some event.

• Collected by central tools for every device in environment

• Operations team can deal with server, environment

OS

App

Network Equipment

Monitoring

?

6


Application Metrics

•Metrics used in development for profiling often continue to be interesting in production

•Highly specific to a specific service:

number of open connections quota usage

message duration message size distribution

AD Lookup response time, uptimemessage distribution across users/folders

cache hit/miss rate MAPI operation count


Example Application Metrics

Apache $ apache2ctl status

memcached

$ echo -ne "stats\r\n" | nc -i1 localhost 11211 ...STAT curr_connections 2STAT total_connections 71072406STAT connection_structures 11STAT cmd_get 58419832STAT cmd_set 4127175STAT get_hits 53793350STAT get_misses 4626482...

squid

[calh@photocache3 ~] $ /usr/sbin/squidclient -p 80 cache_object://localhost/5min...client_http.all_median_svc_time = 0.000911 secondsclient_http.miss_median_svc_time = 0.042766 secondsclient_http.nm_median_svc_time = 0.000000 secondsclient_http.nh_median_svc_time = 0.024508 secondsclient_http.hit_median_svc_time = 0.000911 seconds...

mysql

mysql> SHOW SLAVE STATUS \G... Connect_retry: 60 Master_Log_File: db1-bin.439 Read_Master_Log_Pos: 374103193 Relay_Log_File: dbslave-relay.125 Relay_Log_Pos: 854427197 Replicate_ignore_db: Last_errno: 0 Last_error: Skip_counter: 0 Exec_master_log_pos: 587702364 Relay_log_space: 854431912...


Again, many tools available

• JMX

• NewRelic

• statsd (extremely light-weight and easy to use)

• snmp (highly standard, but high barrier to use and configuration for application layer)

• collectd + graphite


Alerts

• Actionable

• May be derived from other sources, or raised directly.

• Have to be routed to be useful - thank you, Sean!

•The application teams will get these!

7


What have you been up to?

• Logging key for debugging, tracing, compliance, etc.

• Logs should be collected off-box for multiple constituents, including:

Operations, Support, InfoSec/Compliance, Engineering

• Maximum utility when levels manageable at runtime without application restart

• Logstash, Splunk... no excuses

8


Logs vs. Stats

• Logs can be fall-back facility for monitoring of health and metrics

But this is less than desirable

- Shifts development to ops, usually late in the game

- Can be brittle

- Usually less efficient


Logging in Production

• Clustered, parallel processes require host and process IDs

• Activity flows across services - how do we correlate?

• Audit logs important - who did what when?

• Keep privacy in mind - if logs have private data, we can’t easily take them out of production.

• Passwords must never be logged

© 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])

“Wow, that’s a lot we have to build!”

© 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])

Or not.


If applications provide standard management

interfaces...

App

Logging

Metrics

Health

Process Mgt

Config


OSLogging

Metrics

Health

Process Mgt

Config

Config

Config

Log HostLogging

Stats HostMetrics

Alert ManagerHealth

Cluster/Orchestration ManagerCM

Configuration ManagerConfig

VCS

Operations

...they can “plug in” to existing infrastructure

App

Logging

Metrics

Health

Process Mgt

Config

Engineering


Operation’s Asks

• Make Operability an integral part of product requirements and designs

• Engage your operations team as soon as possible to let us help you leverage our platform and services

• Talk with us about HA architectures: we don’t have a magic wand!

© 2010 Cisco and/or its affiliates. All rights reserved. 75

Thank you!

www.cisco.com/careers

Technology

Building for operations - Reinhardt Quelle