Upload
icinga
View
1.169
Download
7
Embed Size (px)
Citation preview
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Monitoring with Icinga2Jordan Jethwa | Web Technologist, Online Experience Management
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Thank you
Icinga Development, QA, and Support§ Commitment to Open Source values§ Technology§ Release schedules
Community§ Mailing list, IRC, and Monitoring Portal users
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Online Experience Management
About us§ DevOps!§ Develop platform code that runs on Adobe Experience Manager (AEM)§ Support and maintain infrastructure§ Automation, monitoring, and metrics
Stakeholders§ Content authors across multiple worldwide business units§ Teams with integrations, etc
Priorities§ Accelerate content development activities§ Reduce technical and maintenance debt§ Provide a unified and consistent experience
3
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Online Experience Management (continued)
Adobe@Adobe§ Showcase technologies from Adobe§ Collaborate with engineering teams
§ Innovate
Environments§ Complex
§ Landscapes (dev -> prod)§ Platforms (legacy vs new, intranet, etc)§ Differing SLAs
4
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Ops day to day stack slice
5
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Icinga2 Case StudyOnline Experience Management
6
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Move to DevOps (~2 years ago)
An opportunity to evaluate, modernize, and experiment§ Methodology§ Technology stack
Scalability§ An ever growing infrastructure§ More hands in the cookie jar: Dev + Ops
§ More frequent code and infrastructure changes
Flexibility§ All the things
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Why Icinga2?
A few other evaluations§ Nagios - NOPE!§ New Relic, Datadog – Cons: Hosted, Java Agent byte-code injection
§ Sensu – Cons: Set up and maintenance
Icinga2§ Object-based configuration format (macros, assignments, conditionals, and functions!)
§ Dependencies§ Metrics and built-in graphite export§ Icinga Web 2§ Compatible with Monitoring Plugins (https://www.monitoring-plugins.org/)§ Icinga Exchange (https://exchange.icinga.org/)
§ REST API – Manage objects, retrieve information, perform actions like rescheduling checks
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Current Icinga2 stats
9
220+hosts
2100+ checks
5m intervals
~ 1sexecution time
1 server (failover)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Configuration Design
Focus on maintainability, flexibility, and automationStructure:
§ Tree-based§ One file per configuration object (mostly)
Groups:§ Not just for visualization!
Templates:§ Apply a set of generic attributes to multiple objects
Assignments – Match object attributes:§ HostGroups and ServiceGroups§ Custom variables
10
/etc/icinga2/conf.d/|—- api-users.conf|—- app.conf|—- commands/|—- downtimes/|—- groups/|—-—-—- hostGroups/|—-—-—- serviceGroups/|—- hosts/|—-—-—- platform0/|—-—-—-—-—- myhost.conf|—-—-—-—-—- myhost/|—-—-—-—-—-—-—- myhost_specific0.conf|—- notifications/|—- services/|—-—-—- ssh.conf|—-—-—- platform0_specific/|—-—-—-—-—-—-—- platform_specific0.conf|—- templates/|—-—-—- hosts/|—-—-—- notifications/|—- users.conf
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Configuration Design (continued)
Keep it simple and modularObjects:
§ Rely on inheritance§ assign and ignore§ Human readable
Host object automation:
§ Current: Script generates from CMDB but, CMDB entry is manual L§ Target: Chef
11
+-----------------------------+
|object Host "example_host" { |
| import "rhel6-5" |
| import "aem6-1" |
| |
| display_name "Example Host"|
| address = "192.168.0.22" |
| |
| vars.landcape = "prod" |
| vars.location = "eu-west-1"|
|} |
+-----------------------------+
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Guess the chart
12
0
10
20
30
40
50
60
70
80
90
100
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Millions of notifications go unread everyday
13
0
10
20
30
40
50
60
70
80
90
100
Notification fatigue
Productivity Notifications
You can make a differenceReduce notifications:
§ Notification intervals based on SLAs§ Use dependencies!§ Scheduled downtimes§ Route using user groups
Notification alternatives/additions:§ Slack (https://github.com/jjethwa/icinga2-slack-
notification)§ PagerDuty
§ Push notifications (Pushover, etc)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Performance Data
14
It’s all about metricsEnable performance data writer:
§ No
Enable Graphite Carbon writer:§ Pa
Grafana:§ Visualization – See trends, correlate data§ Dashboards – Impress execs J
Bonus round:§ Icinga Web 2 graphite module
(https://github.com/findmypast/icingaweb2-module-graphite)
# icinga2 feature enable perfdata
# icinga2 feature enable graphite
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Docker
Zero to Icinga2 in less than a minute§ x§ debian:jessie
Resources:§ Docker Hub: https://hub.docker.com/r/jordan/icinga2/§ GitHub: https://github.com/jjethwa/icinga2
Contributing a patch:§ Submit an issue describing your proposed change to the repo in question.§ I will respond to your issue promptly.§ Fork the desired repo, develop and test your code changes.§ Submit a pull request.
15
# docker pull jordan/icinga2
# sudo docker run -p 80:80 -t jordan/icinga2:latest
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Upcoming activities
§ Host object generation - Replace current CMDB script§ Plug n Play Icinga2 satellites - Gather metrics from different office networks
16