View
235
Download
0
Embed Size (px)
Citation preview
NO IT LEFT BEHIND
Connecting the software-defined data center to
multi-modal IT
1
• Setting the Stage
• BiModal Vs. MultiModal
• Our Approach
• Summary and Lessons Learned
AGENDA
2
18 Years in Business
SETTING THE STAGE
3
Focused on Developing Solutions for the DoD, DHS and other Federal Agencies
• Wide Range of Compliance Challenges
SETTING THE STAGE
4
Diverse Staff
• 300+ Software Engineers
• 150+ Field Personnel
• 100+ Active Projects at any time
SETTING THE STAGE
5
150+ Corporate Systems
• 3rd Party and Homegrown!
SETTING THE STAGE
6
Development• Research and Development• Enhancement• Bug fixes
Life Cycle• Configuration Mgmt• Release Mgmt• Packaging
Infrastructure• Test & Integration• Documentation• Help Desk
Customers• Maintenance• Training• Pro Services
SETTING THE STAGE
7
LANDSCAPE
8
LANDSCAPE
9
LANDSCAPE
10
APPLICATIONS• Unanet, JIRA, SharePoint, Finance, HR…
FABRIC• VMWare, NSX, Laas, Paas, ...
NETWORK• Routers, Phones, VTC, …
COMPUTE• UCS, Towers, Laptops, ...
STORAGE• NetApp, EMC, OneDrive, …
LANDSCAPE
11
12
We have built infrastructure
PROGRESS
13
But building infrastructure is
not the goal.
PROGRESS
• We need to be more Agile to keep up with rapidly changing business needs
• Virtualization Alone Not Sufficient!
• Tenets
NEED FOR CHANGENO MORE ISLANDS
14
Traditional IT focused on Stability and Efficiency
BIMODAL OPTION
15
BIMODAL OPTION
DevOps IT focused on new tools and agile processes
16
PROS: • Reduced risk for ongoing core business activities
• Frees forward looking teams to not worry about the day to day
BIMODAL OPTION
17
PROS: • Reduced risk for ongoing core business activities
• Frees forward looking teams to not worry about the day to day
• Ignores the interdependence between past and future
• Near term increase in resource needs
• Significant cultural impact
• ”haves and have nots”
• Latent talent ignored
BIMODAL OPTION
CONS:
18
• Systems evolve at a natural pace
• Some applications will die off
MULTI-MODAL OPTION
Migrate Everything
19
• Responsiveness and reliability are the same thing!
MULTI-MODAL OPTION
Treat All Systems as Agile
20
• Each piece of the enterprise is at a different place on the continuum and will move forward at a different speed (crawl/walk/run)
• Dynamic on-premise environment that supports flexible, timely and relevant capabilities while adhering to a wide range of compliance challenges
MULTI-MODAL OPTION
DevOps approach spanning the entire enterprise
21
22
HOW ARE WE DOING THIS?
23
APPROACH
Collaborate and Communicate
Fail Fast, Recover Fast and learn
Always Keep Learning
Technologies Small Batch Size
Dsmorse_iss
24
• Ad hoc is bad hoc
• Solution “discovered” on production
• No one person has whole story
• Configuration drift
• Non existent Documentation
• Any manual task can introduce human error
• We shouldn’t have to call you on your on day off
• Goals:
• Repeatable, Repeatable, Repeatable, …
• Known state
• DRY (yeah I used to be a dev)
WHY DEVOPS
Dsmorse_iss
25
• Ad hoc is bad hoc
• Solution “discovered” on production
• No one person has whole story
• Configuration drift
• Non existent Documentation
• Any manual task can introduce human error
• We shouldn’t have to call you on your on day off
• Goals:
• Repeatable, Repeatable, Repeatable, …
• Known state
• DRY (yeah I used to be a dev)
WHY DEVOPS
Dsmorse_iss
28
Frequent small changes are less risky than a Large release
SMALL BATCH SIZE
Dsmorse_iss
29
SMALL BATCH SIZE
Fail Fast: Tracking MTTR
• Small Batch
• Easy to ID change
• Disciplined about ConfigManagement
• Known state before & after
• Automated scripts for backup & deploy
• Scripts also verify expected stat
Anti-fragile thinking to allow us to upgrade live systems during regular hours
Dsmorse_iss
30
• Desired state configuration
ANSIBLE-WHAT GOES WHERE
Dsmorse_iss
31
• Desired state configuration
• It is just a YML file
ANSIBLE-WHAT GOES WHERE
Dsmorse_iss
32
• Desired state configuration
• It is just a YML file
• Agentless! Uses SSH
ANSIBLE-WHAT GOES WHERE
Dsmorse_iss
33
• Desired state configuration
• It is just a YML file
• Agentless! Uses SSH
• Idempotent
Idempo-What: “Operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application.” – wikipedia
ANSIBLE-WHAT GOES WHERE
Dsmorse_iss
34
• Docker containers increase portability
DOCKER, DOCKERDOCKER
Dsmorse_iss
35
• Docker containers increase portability
• Increase density & VM utilization
DOCKER, DOCKERDOCKER
Dsmorse_iss
36
• Docker containers increase portability
• Increase density & VM utilization
• Easier Deployment
DOCKER, DOCKERDOCKER
Dsmorse_iss
37
• Docker containers increase portability
• Increase density & VM utilization
• Easier Deployment
• Immutable images removes complexity, changes the way we take care of our apps
DOCKER, DOCKERDOCKER
Dsmorse_iss
38
SYSTEM DIAGRAM
Dsmorse_iss
39
SYSTEM DIAGRAM
Dsmorse_iss
40
SYSTEM DIAGRAM
Dsmorse_iss
41
A critical part of DevOps is looking at change metrics
• Did we achieve the desired outcome?
• Should we pivot or persevere?
• What problems do we not know we have
COLLABORATE AND COMMUNICATE
Dsmorse_iss
42
A critical part of DevOps is looking at change metrics
• Did we achieve the desired outcome?
• Should we pivot or persevere?
• What problems do we not know we have
Closed systems have been replaced by Open JIRA projects
• All changes are public!
COLLABORATE AND COMMUNICATE
Dsmorse_iss
43
A critical part of DevOps is looking at change metrics
• Did we achieve the desired outcome?
• Should we pivot or persevere?
• What problems do we not know we have
Closed systems have been replaced by Open JIRA projects
• All changes are public!
HipChat was brought in to allow automated flow of relevant data into tailored rooms• Provide context to the sequence of events• Real-time discussion, feedback and change!
COLLABORATE AND COMMUNICATE
Dsmorse_iss
44
Team Development
• Ongoing cross-training and mentoring• Pair programing workstation to enhance
our ability to mentor each other
KEEP LEARNING
Dsmorse_iss
45
Team Development
• Ongoing cross-training and mentoring• Pair programing workstation to enhance
our ability to mentor each other
Git Flow• Pull Requests• Peer Review
KEEP LEARNING
Dsmorse_iss
46
Team Development
• Ongoing cross-training and mentoring• Pair programing workstation to enhance
our ability to mentor each other
Git Flow• Pull Requests• Peer Review
Blameless Retrospectives• Posted to Confluence• 100% transparency
KEEP LEARNING
Dsmorse_iss
47
• Reliable alerts on public facing systems.
• The structures we put in place to support 12 factor apps, can also be applied to our legacy systems to increase commonality
o Decrease cognitive shift
o Embrace changes on both sides
• Growing number of Anisble roles for common tasks
o Firewalls, NTP, SMNP, common tools
• Performed our first few canary rollouts
WHERE ARE WE TODAY?
Dsmorse_iss
48
More of the same!
• Scripting
• Containerization
• Monitoring
• Collaboration
• Next 6 months
Dsmorse_iss
49
More of the same!
• Scripting
• Containerization
• Monitoring
• Collaboration
New Stuff!
• vRealize/NSX for Self-Service
• vSphere integrated containers
• Docker data center
• Next 6 months
Dsmorse_iss
50
More of the same!
• Scripting
• Containerization
• Monitoring
• Collaboration
New Stuff!
• vRealize/NSX for Self-Service
This is working!
NEXT 6 MONTHS
Dsmorse_iss
51
After 9 Months
• Cleared more than 3 years of backlog!
• Migrated from manual changes to scripted changes
• Containerized workloads running on their own
• Containers as a Service
SUMMARY AND LESSONS LEARNED
Dsmorse_iss
52
After 9 Months
• Cleared more than 3 years of backlog!
• Migrated from manual changes to scripted changes
• Containerized workloads running on their own
• Containers as a Service
Lessons Learned
• Establish Metrics Early On
• Change Takes Time
• Expect and Embrace Failure
• Technology Enables, but Not a Panacea
SUMMARY AND LESSONS LEARNED
Dsmorse_iss