ADVANCED PROBLEM DETECTION (POKROČILÉ NASTAVENÍ … · Zabbix Certified Specialist Update Zabbix...

ADVANCED PROBLEM DETECTION

(POKROČILÉ NASTAVENÍ TRIGGERŮ)

How does Zabbix work

Basic and additional ways of detecting a problem

Possible reaction scenarios

AGENDA

ZABBIX - enterprise-level and open-source

monitoring system.

Competitive advantages of the solution:

Free of charge

«All-in-one»

Easy-to-manage

Mature, high-quality and reliable

Flexible (also applicable to problem detection)

WHAT IS ZABBIX?

ZABBIX DATA FLOW

Visualization

DATABASEDATABASE

History

ZABBIX SERVERZABBIX SERVER

Analysis

Notifications

Data collection

Every N seconds

Zabbix will evenly distribute checks

Different frequency in different time periods

Every X seconds in working time

Every Y second in weekend

At a specific time (Zabbix 3.0)

Ready for business checks

Every hour starting from 9:00 at working hours (9:00, 10:00,…, 18:00)

HOW OFTEN TO EXECUTE CHECKS?

RHow to detect problems in the

incoming data flow?

TRIGGER

Triggers!

TRIGGER

RTrigger –

problem definition

TRIGGER

Example

{server:system.cpu.load.last()} > 5

Operators

- + / * < > = <> >= <= not or and

Functions

min max avg last count date time diff regexp and much more!

Analyze everything: any metric and any host

{node1:system.cpu.load.last()} > 5 and {node2:system.cpu.load.last()} > 5 and

{nodes:tps.last()} < 5000

TRIGGER

Performance

JUNIOR LEVEL

FALSE POSITIVES

«Flapping»

Availability

{server:net.tcp.service[http].last()} = 0

JUNIOR LEVEL

TOO SENSITIVE

{server:net.tcp.service[http].last()} = 0

RToo sensitive leads to

False positives

How to avoid false positives?

HOW TO AVOID FALSE POSITIVES?

Be careful and define problems wisely!

What does it really meansystem is overloaded

application does not work

service is not available

EXAMPLES

Problem: CPU load > 5No problem: CPU load = 4.99 Resolved?

Problem: free disk space < 10%No problem: free disk space = 10.001% Resolved?

Problem: SSH check failedNo problem: SSH is up Resolved?

ANALYZE HISTORY

Performance

{server:system.cpu.load.min(10m)} > 5

Availability

{server:net.tcp.service[http].max(5m)} = 0

{server:net.tcp.service[http].max(#3)} = 0

ANALYZE HISTORY

{server:system.cpu.load.min(10m)} > 5

ANALYZE HISTORY

{server:net.tcp.service[http].max(#3)} = 0

DIFFERENT CONDITIONS FOR PROBLEM AND

RECOVERY

Before

Problem definition: {server:system.cpu.load.last()}>5

Recovery expression:

{server:system.cpu.load.last()}<=1

DIFFERENT CONDITIONS FOR DIFFERENT TRIGGER STATES

{server:system.cpu.load.last()} > 5 … {server:system.cpu.load.last()} <= 1

No false positives!

System is overloaded

Problem definition: {server:system.cpu.load.min(5m)}>3

Recovery expression: {server:system.cpu.load.max(2m)}<=1

No free disk space /

Problem definition: {server:vfs.fs.size[/,pfree].last()}<10

Recovery expression: {server:vfs.fs.size[/,pfree].min(15m)}>30

SSH is not available

Problem definition: {server:net.tcp.service[ssh].max(#3)}=0

Recovery expression: {server:net.tcp.service[ssh].min(#10)}=1

EXAMPLES

How to detect?

By comparing with the data from the same period, the period is taken from the

Average CPU load for the last hour is 2x higher than

CPU load for the same period week ago

{server:system.cpu.load.avg(1h)} > 2 * {server:system.cpu.load.avg(1h,7d)}

ANOMALIES

Comparison with the data 7 days ago

FORECAST

Trigger function timeleft

When we reach a certain threshold value

FORECAST

Trigger function forecast

When we reach a certain threshold value

4 hours

DOES HISTORY ANALYSIS AFFECT PERFORMANCE OF ZABBIX?

Yes, but not significantly.

Especially as of Zabbix 2.2.0.

DATA BASEDATA BASEZABBIX SERVERZABBIX SERVERCACHECACHE

DEPENDENCIES

CRM is not workingCRM is not working

DB is inavailableDB is inavailable

No free diskspaceNo free diskspace

Hide dependent problems

How to display problems?

SECTION „PROBLEMS“

Tag word: meaning

Customer: Alza

Customer: Globus

Datacenter: NY2

Datacenter: San Francisco

Area: Performance

Area: Availability

Area: Security

Environment: Staging

Environment: Test

User impact: None

User impact: Critical

USE OF OBTAINED VALUES

Use of useful information in tags or names

Example:

“RAID Controller Module wide port has gone to failed state. Enclosure 0, Slot 0 (Critical)”

instead

“New SNMP trap received (Critical)”

How to react to problems?

POSSIBLE REACTIONS

Event correlation

Automatized problem solving

Manual problem closing

Sending notifications to a user or a group of users

Registration of tasks in the Helpdesk system

EVENT CORRELATION ON TRIGGER LEVEL

Correlation of events at the trigger level allows you to compare individual problems reported by a single trigger.

10/Aug/2016:06:25:30 service Jira stopped “Service Jira stopped” PROBLEM

How does it work?

10/Aug/2016:06:27:32 service MySQL stopped “Service MySQL stopped” PROBLEM

How does it work?

10/Aug/2016:06:27:32 service MySQL stopped “Service MySQL stopped” RESOLVED

10/Aug/2016:06:28:11 service MySQL started

How does it work?

10/Aug/2016:06:27:32 service MySQL stopped “Service MySQL stopped”RESOLVED

10/Aug/2016:06:34:22 service Redis stopped “Service Redis stopped” PROBLEM

How does it work?

10/Aug/2016:06:34:22 service Redis stopped “Service Redis stopped” RESOLVED

10/Aug/2016:06:37:58 service Redis started

How does it work?

10/Aug/2016:06:25:30 service Jira stopped “Service Jira stopped” RESOLVED

10/Aug/2016:06:34:22 service Redis stopped “Service Redis stopped” RESOLVED

10/Aug/2016:06:37:58 service Redis started

10/Aug/2016:06:55:31 service Jira started

EVENT CORRELATIONA new problem appears

Existing problems

EVENT CORRELATION

Existing problems No correlation rules

EVENT CORRELATION

Existing problems No correlation rules

EVENT CORRELATION

Existing problems No correlation rules (close old)

ESCALATE!

Immediate reaction

Delayed reaction

Notiofication if automatic action

failed

Repeated notifications

Escalation to a new level

ESCALATE!

Critical

problem

Critical

problem Repeated Email

SMS and ticket

Service restart

SMS to manager

10 min

15 min

20 min

IN SUMMARY

Analyze history

No problem!= Solution

Use different conditions for problem definition and recovery

Pay attention to anomaly detection

Use correlation

Resolve common problems automatically

Do not hesitate to escalate!

ZABBIX SERVICES

ZABBIX DEMO

Video najdete zde: https://www.coreit.cz/monitoring-it-prostredi

ZABBIX WEBINÁŘE

Připravili jsme pro Vás další Zabbix webináře. Všechny informace najdete u

nás webu: https://www.coreit.cz/zabbix-webinare-do-konce-roku-2020/

26.11. Rozšíření funkcí Zabbixu

1.12. Poznejte Zabbix

3.12. Komunikace se Zabbixem pomocí API

8.12. Vizualizace dat v Zabbixu

PROFESIONÁLNÍ ŠKOLENÍ

Become Zabbix certified without attending the training. If you are certain of your knowledge, ZCU, ZCS and ZCP exams can be purchased separately.https://www.zabbix.com/training?language=czech#training_schedule

Aktuální termíny

připravovaných

školení

najdete zde

Pokud byste potřebovali jakékoliv informace, neváhejte se na nás obrátit:

KONTAKTUJTE NÁS

Telefon: +420 840 771 177

Web: https://www.coreit.cz

Email:info@coreit.cz

tomas.hermanek@coreit.cz

LinkedIn:https://www.linkedin.com/company/coreitcz/

https://www.linkedin.com/in/hermanekt/

Twitter:https://twitter.com/CoreITcz

https://twitter.com/hermanekt

Mobil Tomáš Heřmánek: +420 732 447 184

OTÁZKY?

DĚKUJI ZA POZORNOST!

ADVANCED PROBLEM DETECTION (POKROČILÉ NASTAVENÍ … · Zabbix Certified Specialist Update Zabbix...

Documents

Zabbix in Japan - LIAA · 3 What is Zabbix? An image of Google datacenter Zabbix is a solution for monitoring IT infrastructure

Installation Agent Zabbix sur Windows · Dossier d’installation Installation de l’agent Zabbix sur Windows Asema Zabbix V.3.4 Installation de l’agent Zabbix sur Windows Janvier

Zabbix Conference LatAm 2016 - Douglas Esteves - Zabbix at UNICAMP

Zabbix: Прошлое, настоящее и будущее (Zabbix: Past, present and the future)

Statistické nástroje a pokročilé možnosti SFX

Zabbix 2014 Conference : Implementing Zabbix in large Banking Environment

Zabbix Manual

Zabbix 3.0 Training Certified Specialist - xiaotonghz.com.cn · Zabbix 3.0 Training Certified Specialist Day 2 The Enterprise class Monitoring Solution for Everyone

Manual Zabbix 18

Tutorial Zabbix

ZABBIX HA CLUSTER SETUPS · 2019-12-27 · 9 all cluster ip’s and hostnames # VIPs for cluster: 192.168.7.87 zabbix-ha-app 192.168.7.89 zabbix-ha-db-app 192.168.7.88 zabbix-ha-fe-app

Open Source Enterprise Monitoring with Zabbix - · PDF fileOpen Source Enterprise Monitoring with Zabbix Alexei Vladishev, Founder of Zabbix

Page 1 Page 2 ZABIX About Zabbix Software About Zabbix

Zabbix Manual v1.6

Zabbix Conference LatAm 2016 - Jessian Ferreira - Wireless with Zabbix

ZABBIX Business - Softnixupdate.softnix.co.th/download/Saletools/Zabbix/ZABBIX... · 2012-06-01 · Zabbix Solution functionality is available in every copy of software. A cutting-edge

Zabbix Tuning

Pokročilé responzivní obrázky

Zabbix Router

NodeConductor Zabbix Documentation - Read the Docs · PDF fileNodeConductor Zabbix Documentation, Release 0.6.0 Zabbix service provides an interface to Zabbix monitoring system. Contents