View
218
Download
0
Category
Preview:
Citation preview
Monitor Open stack environments from the bottom up and front to back
Roger Ruttimann VP Engineering, GroundWork OpenSource November 17, 2015
What’s ahead of us
Overview: The Impact of Virtualization on IT operations
How Open Stack fits into the Virtualization landscape
Monitoring the changed landscape of IT infrastructure
New Monitoring concepts
Selection of the right tools to fit the right process
Conclusion
2 OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Start with the facts: Open Stack is real!
It’s all OpenSource Has 5000 active members, growing daily Conferences draw close to 10,000 participants Many corporate sponsors
3
OpenStack deployments are not just happening in a far away land ... Not secret potion or evil conspiracy Not just a geek movement Serious deployments for IT Operations not just in the Silicon Valley
It’s disruptive and requires serious re-tooling for IT operations
Let’s review the challanges and what tools best fit the new realities
Box Spy
OSCM 2015 Monitor Open Stack environments from bottom up and front to back by Roger Ruttimann
Once upon a time
SysAdmin task: Add To IT infrastructure {
Deploy servers into datacenter Provisioning applications Define monitoring for each element Monitoring was done with SSH checks and port gets
} repeat Checking for up down and send email
4 OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Switch
Router
Switch
Apps
Apps
Apps
Apps
Apps
Firewall
Server Server Server
Monitor
Port get
ssh
snmp
snmp
snmp
virtualization
Then came along compute virtualization
5
Datacenter compute optimization • Server resources were virtualized to
improve efficiency which was < 30% • Products like ESX allowed resource
optimization not disturbing the provisioning process
Rollout • VM machines were provisioned • Applications installed • Monitoring defined as before New was an API to the virtualization manager (example VSphere API) to poll data
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Switch
Router
Switch
Apps
Apps
Apps
Apps
Apps
Firewall
Server Server Server
Monitor
Port get
ssh
snmp
snmp
snmp
• REST API
But it didn’t stop there
6
Software Defined Everything • Compute (Hypervisors) • Storage (SDS) • Network (SDN)
Hybrid Cloud Public/Private • Amazon WebServices • Rackspace • Azure
Change created blind spots in coverage Suddenly ssh/snmp methods don’t cover all
KVM
RHEV-M VCenter
virtualization
Switch
Router
Switch
Apps
Apps
Apps
Apps
Apps
Firewall
Server Server Server
Monitor
Port get
ssh
snmp
snmp
snmp
• REST API
Virtualized Network
Virtualized Storage
Switches Switches Switches
• REST API
• REST API
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Infrastructure landscape completely changed
7
Silo
vie
w
Hyp
ervi
sor m
anag
er v
iew
Apps VMs
Devices
Files Shared I/O Fog
Network & Storage OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Silo
vie
w
Silo
vie
w
Silo
vie
w
Silo
vie
w
Silo
vie
w
And the moment you think you’ve seen it all …
8
… DevOps is pushing the envelope even more
Linux Containers
Are the new kids in town
Application Isolation Rapid deployment Elastic Scalability
API for deployment:
Monitoring is somebody else’s job
virtualization
Switch Switch
Apps
Apps
Apps
Apps
Apps
Server Server Server
Monitor
Port get
ssh
• REST API
Virtualized Network
Virtualized Storage
Switches Switches Switches
• REST API
• REST API
Apps Apps Apps
Server
Docker Host
Apps
Server
Docker Host
Apps Apps Apps Apps
Server
Docker Host
Apps Apps Apps Apps
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Don’t panic
9
What happened over the last 5 years: • Virtualize everything • Private and Public clouds
• API centric world not just for Applications
• REST API is a standard • CORBA, SOAP & proprietary API’s are
now classified as dinosaurs
• Browser based UI / JavaScript is king • Learn to speak REST & JSON
Diverse Virtualization stacks
vSphere API
ESX
NSX
vSan
OpenStack API
Nova
Neutron
Cinder Glance
AWS API
EC2
VPC
S3
oVirt API
Linux KVM
Network
NFS
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
How does Open Stack fit in?
10
Service oriented architecture Pluggable hypervisors, network, storage to support wide range of technologies Elastic Compute Units for better virtualization efficiency Standardized API’s to all services
Sounds really good doesn’t it ?
Object Store--Swift
Image--Glance Compute/
Hypervisor--Nova
Network— Quantum-Neutron Storage--Cinder
Dashboard--Horizon
Identity--Keystone
API API API API API
API
But what’s the reality ?
Façade Service Façade Service
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Open Stack Profile Editor
11 OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Open Stack Profile Editor
12 OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
ManageIQ/ Cloud Forms
Ansible
API
Manager Stack
API
HP Helion API
VMWare VRealize
Mirantis Fuel
API
API
Provisioning/Configuration
Servers Servers
API is good but comprehensive coverage is better
13
Object Store
Image
Computer
Network
Identity
Storage
Dashboard
Open Stack API
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Container metrics
Performance data
Virtual infrastructure
Legacy network
Servers
Storage racks
Applications, Infrastructure checks
ssh/SNMP/port get
Unified View Standardized Data Collection
Unified View
14
Silo tools • Impossible to correlate • Naming mismatch • Over-monitoring • Encourages: “It’s not my problem
check your system” • No Big Picture Dashboards
Need for • Stack monitoring bottom to top • Aliasing of names • Combine best of breed collectors • Correlated metrics across all
infrastructure • Dashboard for each customer/client
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Stack monitoring for Open Stack
15
Function Source Host name Alias
Network, Storage Netflow SNMP
Hardware IPMI SNMP
Drac-server1 os-eng-h1
Identity server check-mysql check-port
server1:port os-eng-h1
Operating System
check-proc check-mem check-load
server1 os-eng-h1
OpenStack-API Hypervisor CPU/Mem metrics, Network storage
10.10.0.1 os-eng-h1
OpenStack API VM CPU/mem, Network, storage 10.10.10.123
eng-slicer-1
Operating system
check-proc check-mem check-load
eng-cent6-actg
eng-slicer-1
DockerHost Memory, CPU eng-cent6-actg eng-slicer-1
Container Memory, CPU, Procs FAC3443DA77 Load-bal-01
Application Check_https
172.28.102.51 Load-bal-01
Hyp
ervi
sor
VM
A
PP
Automation / Continuous configuration discovery
16
Adding Virtual machines, containers is automatic
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
V C
Hypervisor
So is the monitoring
Management Monitor System
Configuration collector
Data collector
Plugins
Dashboards Data
C C V V V V V C
Monitoring is like a flight recorder -- collecting all active data
Continuous operations by .. synchronization of configuration
re-configuration of plugins Collection of data (availability & performance)
Open Stack API
Icinga2 setup
OpenStack Monitoring: Tools selection
17 OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
OpenStack Linux OS Platform
Hypervisor Hardware IPMI Port
Agent
OpenStack Controller Keystone
OpenStack Storage Cinder
Unified Monitoring View
Data Management Collector Collector Collector
Fuel
Murano Application Software Catalog
Icinga2
1) Fuel, Open Stack deployment 2) Murano, Application deployment 3) Fuel Plugin, Configuration generate 4) Fuel Plugin, Configure monitoring
Docker
Node 2
Node 3
Docker
VM 1
OPEN TSDB
Icinga2
VM 1 VM 2 VM 3 VM 4
Box Spy
Box Spy
Docker OPEN TSDB
Performance Data
18
Monitoring systems
Grafana Dashboards
VCenter
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Centralized Data Collector • Monitoring send Perf Data to single
API Expandable storage cluster • Expand on demand
Dashboard to visualize and drill down • Historic, raw performance data • Group alike metrics to find outliers
Reference Architecture monitoring
Aliasing and data normalization in backend
VCenter
API
Data Collector
Hub
Collector Collector Collector
Collect
Data Integration
Normalization Aliasing
API
Integrate
API
Visualize
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Growing number of API’s require integration Hub
Benefits
20
Automation • Reduced cost of maintenance • Dynamic configuration • Streamlined operations
Personnel • System Administrator & Dev Ops
functions merge
Capacity & Resource planning • Complete Bottom (network) to Top
(application) data collection • Allocate resources at the right level
Hardware • No vendor lock-in • Transparency through API
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Lessons learned
21
Virtualization and containerization require new monitoring techniques Legacy hardware/software will be around for a while so don’t throw tools away Everything API -- needs integration Aliasing is hard but maybe easier than cross silo cooperation Use the best tool for the task Don’t over-monitor to cover a gap
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Conclusion
22
Open Source tools are the driving force for innovation
DevOps just selects the best tool to do the job
A single monitoring tool to “rule them all” doesn’t exist
Integration is a complex task – Don’t expect teams will agree on naming, process, and workflow
Automation and pragmatism will prevail as DevOps has demonstrated
Virtualize everything will continue
Automation and continuous discovery is necessary for rapid scale out
Open Stack is the Open Source Virtualization platform but Monitoring coverage needs major improvements
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
23
Thank you
Roger Ruttimann rruttimann@gwos.com
OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann
Recommended