Homebrew Your Own Metrics - An IBM Domino Administrator's Guide to SNMP (MWLUG 2014 Session SA103)

Homebrew your own Metrics: An IBM Domino Administrator's Guide to SNMP

(SA103)

Ray Bilyk, Lotus Notes AdministratorDr. Tom Bilan, Director of Technical Services

- IHI, LCE and BLDIlitch Holdings, Inc.


Ray Bilyk• Notes Administrator at Ilitch Holdings, Inc.• • Member of the Month, SocialBizUG.org• Silver Medal, National Homebrew Competition• http://www.thepridelands.com

Dr. Tom Bilan• Director of Technical Services at Ilitch Holdings, Inc.• https://www.linkedin.com/in/bilan


Outline

• Overview of presentation• Abstract• Overview of Domino monitoring capabilities• Backstory• SNMP Basics• SNMP Support in Domino• SNMP command-line• Other examples


ABSTRACT:It's important as an IBM Domino administrator to get metrics on your servers so that you know what's going on in your "Brewhouse". There are also many ways to obtain this very valuable information, like 3rd party tools and even the IBM Domino Administrator client. What many don't know is that there is also another way to capture this very valuable information. SNMP (Simple Network Management Protocol) is an Internet-standard protocol for managing devices on IP networks. In this session, you will learn more about it, what it can give you, how to set it up, and ways to 'drink in' or monitor the information. With proper configuration, administrators not familiar with IBM Domino can obtain and use key IBM Domino server metrics. Homebrew your own metrics by attending this session!


• Traditional Monitoring– Monitoring Configuration database (EVENTS4.NSF)– Statistics Reports database (STATREP.NSF)

• Domino Domain Monitoring (DDM)• Domino Administrator client• 3rd Party Tools

Existing Domino Monitoring Options

The Backstory


WelcomeTom

Bilan!

What is SNMP?

It’s not Simple (until you understand it)It’s not just for NetworksIt’s for ManagementIt’s a ProtocolIt’s an industry standard communication protocol for talking

to devices that are connected to networks. If a device says it’s manageable it likely has SNMP capabilities.


How can SNMP help you?

SNMP can be used to measure something immediately or over time so that you can make better decisions.

SNMP traps can be sent to alert you of an event (i.e. drive failure).

SNMP can be used to measure and then set thresholds that trigger alerts.


How can SNMP help you? cont

SNMP is an enabler that allows visibility into real information so you’re not guessing.SNMP can answer questions like:

– Do we need more phone lines?– Do we need more Internet bandwidth?– Why is my storage running slow?

• Is it CPU? Memory? Disk I/O? Network??????

– Am I going to run out of disk space 3 months from now?– What was going on 20 minutes ago when everyone said the

network was running slow?– Do I have a POS system that is getting hard drive errors and will

probably die 3 weeks from now? – What temperature was it in the data center at 4 am 3 weeks ago?– How many mail messages are in the queue?


Why use SNMP instead of a special tool?SNMP provides a common interface to get management

information for the thousands of different devices made by thousands of vendors.

Each product may have it’s own management program but what if you want a dashboard with many different devices on one web page?

Once you master SNMP every device looks the same. (In other words, all metric collection problems look like nails so SNMP is the metaphorical hammer)

It’s a standard and there are MANY free tools


The Lingo

MIB – Management Information Base – A vendor provided file that explains what SNMP functionality they support.

OID – Object Identifier – This is a specific variable that you can get or set on a device.

MIB Namespace – A hierarchy that explains how an OID is determined. (picture coming in a few slides)

Community – Password (RO = read-only, RW=read-write)


The Lingo Cont.

Get – Asking the device for a valueSet – Changing a value in a deviceTrap – A device sending out an alertManager – Your PCAgent – The device you’re monitoring


MIB


MIB Namespace


Data Types


What you need to use SNMPThe bare minimum to get started:

– A SNMP querying program– IP address of the device– Community name– The OID

Extras:

– Device MIBs (yes… you don’t need MIBs to query a device)

– A cool GUI– SNMP Trap receiver


Examples


Domino MIB ExamplelnWaitingMail OBJECT-TYPE --<< INT mail.waiting SYNTAX INTEGER (0..2147483647) ACCESS read-only STATUS optional DESCRIPTION "Number of mail messages waiting to be routed." ::= { lnMail 6 }

lnDeadMail OBJECT-TYPE --<< INT mail.dead SYNTAX INTEGER (0..2147483647) ACCESS read-only STATUS optional DESCRIPTION "Number of dead (undeliverable) mail messages." ::= { lnMail 1 }

SNMP MIB Example – Writeable lnRemoteReboot OBJECT-TYPE SYNTAX INTEGER { no(1), yes(2) } ACCESS read-write STATUS mandatory DESCRIPTION "Set to 'yes' to reboot the entire system. This value will be returned as 'yes' if the

system is in the process of rebooting but hasn't come down

yet." ::= { lnControl 5 }

Domino SNMP Query ExampleH:\>snmpwalk -Of -v2c -c mwlug -m ALL 172.16.0.37 lnServerCPUCount.iso.org.dod.internet.private.enterprises.lotus.notes.lnInfo.lnStats.lnServer.lnServerInfo.lnServerCPUCount.0 = INTEGER: 4

H:\>snmpwalk -On -v2c -c mwlug -m ALL 172.16.0.37 lnServerCPUCount.1.3.6.1.4.1.334.72.1.1.6.2.12.0 = INTEGER: 4

H:\>snmpwalk -v2c -c mwlug -m ALL 172.16.0.37 lnServerCPUCountNOTES-MIB::lnServerCPUCount.0 = INTEGER: 4

H:\>snmptranslate -Td -m NOTES-MIB .1.3.6.1.4.1.334.72.1.1.6.2.12.0NOTES-MIB::lnServerCPUCount.0lnServerCPUCount OBJECT-TYPE -- FROM NOTES-MIB SYNTAX INTEGER (0..2147483647) MAX-ACCESS read-only STATUS optional DESCRIPTION "The number of processors installed on this server."::= { iso(1) org(3) dod(6) internet(1) private(4) enterprises(1) lotus(334)

notes(72) lnInfo(1) lnStats(1) lnServer(6) lnServerInfo(2) lnServerCPUCount(12) 0 }

Monitoring ScriptH:\>copy con watchsnmp.bat@echo offsnmpwalk -v2c -c mwlug -m ALL 172.16.0.37 %1watchsnmp.bat %1^Z 1 file(s) copied.

H:\>watchsnmp.bat lnWaitingMailNOTES-MIB::lnWaitingMail.0 = INTEGER: 3NOTES-MIB::lnWaitingMail.0 = INTEGER: 3NOTES-MIB::lnWaitingMail.0 = INTEGER: 4NOTES-MIB::lnWaitingMail.0 = INTEGER: 4NOTES-MIB::lnWaitingMail.0 = INTEGER: 5NOTES-MIB::lnWaitingMail.0 = INTEGER: 4NOTES-MIB::lnWaitingMail.0 = INTEGER: 5NOTES-MIB::lnWaitingMail.0 = INTEGER: 4NOTES-MIB::lnWaitingMail.0 = INTEGER: 2^CTerminate batch job (Y/N)? y

Monitoring ScriptH:\>watchsnmp.bat lnserveravailabilityindexNOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 38NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 38NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 38NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 39NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 39NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 39NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 39NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 39NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 39NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 39NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 39NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 40NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 40NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 40NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 40NOTES-MIB::lnServerAvailabilityIndex.0 = INTEGER: 40^CTerminate batch job (Y/N)? y

Graphing - lnWaitingMail

SNMP Table Example


http://www-01.ibm.com/support/docview.wss?uid=swg21169283or IBM Domino Administrator 9.0.1 Social Edition Help

Look for “Configuring the Domino SNMP Agent” (OS specific steps)

Steps for setup on Windows OS1. Configure SNMP at the Windows operating system level

– Go to Start --> Settings --> Control Panel --> Add/Remove Programs.– Select "Add/Remove Windows Components" on the left.– Choose "Management and Monitoring Tools" and click "Details".– Select all under "Management..."– Reboot the operating system

2. Start Domino

Setting up Domino for SNMP

http://www-01.ibm.com/support/docview.wss?uid=swg21169283

http://www-01.ibm.com/support/docview.wss?uid=swg21169283


3. Configure the Domino LNSNMP agent to run as a service: – Run the following commands in the Domino program directory.

First stop the service if it is already running - in a DOS prompt run commands :c:\Lotus\domino\net stop lnsnmp c:\Lotus\domino\net stop snmp NOTE: If the service is not already running you may see the error:

System error 1060 has occurred. The specified service does not exist as an installed service.

c:\Lotus\domino\LNSNMP -Sc ( this command is case sensitive ) If it starts you will see :Service creation complete

Setting up Domino for SNMP (continued)


3. Continued– More commands to run…

c:\Lotus\domino\net start snmp c:\Lotus\domino\net start lnsnmp If it starts you will see:

The Lotus Domino SNMP Agent service was started successfully.(NOTE: SNMP console appears on server machine)

4. Run these commands on Domino server console load quryset load intrcpt (for SNMP traps)load collect

5. Add quryset and/or intrcpt and collect to ServerTasks (NOTES.INI)

Setting up Domino for SNMP (continued)

Barracuda Web Filter MIBBarracuda-SPYWARE DEFINITIONS ::=BEGIN IMPORTS MODULE-IDENTITY, OBJECT-TYPE, INTEGER FROM SNMPv2-SMI barracuda FROM Barracuda-REF; bspyware MODULE-IDENTITY LAST-UPDATED "201011040000Z" ORGANIZATION "Barracuda Networks, Inc." CONTACT-INFO " Barracuda Networks Inc. 3175 S. Winchester Blvd. Campbell, CA 95008 " DESCRIPTION " Barracuda Web Filter MIB. Provides: Objects: * 1.3.6.1.4.1.20632.3.1.2 -- ActiveTCPConnections * 1.3.6.1.4.1.20632.3.1.3 -- Throughput * 1.3.6.1.4.1.20632.3.1.4 -- PolicyBlocks * 1.3.6.1.4.1.20632.3.1.5 -- SpywareWebHitBlocks * 1.3.6.1.4.1.20632.3.1.6 -- SpywareDownloadBlock * 1.3.6.1.4.1.20632.3.1.7 -- VirusDownloadBlock * 1.3.6.1.4.1.20632.3.1.8 -- SpywareProtocolBlocks * 1.3.6.1.4.1.20632.3.1.9 -- HTTPTrafficAllowed


Example – Barracuda Output


Server Room Humidity


http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=PJgq2JgQuRY2RM&tbnid=7-TcfEeAVssj0M:&ved=0CAUQjRw&url=http://www.squidworks.net/2010/06/apc-netbotz-200-cacti-temp-and-humidity-graphs/&ei=H6juUZ_wCYXE8gGa-oCAAQ&bvm=bv.49478099,d.b2U&psig=AFQjCNEdprge4FI0wjgvLta1GPeTRB3b_g&ust=1374681372784113

NetApp CPU Usage


Voice Channel Usage – Multiple Sources


http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=CDJl9G9DBUAGxM&tbnid=oCypw_dovvRBcM:&ved=0CAUQjRw&url=http://www.barryodonovan.com/index.php/2007/03/31/cacti-as5300-e1-use&ei=nqfuUcWoKYqb8AGK_oBo&bvm=bv.49478099,d.b2U&psig=AFQjCNEdprge4FI0wjgvLta1GPeTRB3b_g&ust=1374681372784113

Fox Internet TrafficXO & AT&T – 24 hours



http://cacti.ihi.local/cacti/graph.php?action=view&local_graph_id=8782&rra_id=all



Fox Internet Traffic - Monthly




Port Errors



Network Dashboard



Thresholds and Alerting



TroubleshootingA few quick things to check if you have problems talking to devices:

– Double-check the community (case counts… it’s a password)

– Double-check the IP address (ping it)– Double-check that SNMP is Enabled on the device

(often it’s disabled by default)– Double-check that there isn’t any restrictions from

where the device accepts SNMP packets from– Double-check the SNMP version. Version 1 or 2c are

the most common and compatible. Version 3 uses advanced authentication and may be overkill so stick with 1 or 2c.




Free Tools (almost better than free beer!)Net-SNMP http://www.net-snmp.org/Cacti http://www.cacti.net/Manageengine MIBBrowser http://www.manageengine.com/Domino MIB (domino.mib) is located in Domino program directory

http://www.net-snmp.org/



http://www.cacti.net/

http://www.cacti.net/

http://www.manageengine.com/

http://www.manageengine.com/

Questions?


http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=dbuDPt0GqcODQM&tbnid=x_hDhtR2tRXkHM:&ved=0CAUQjRw&url=http://www.dilbert.com/strips/comic/2012-09-04/&ei=_CzwUaPFNMuZqQH0joCQBg&bvm=bv.49641647,d.aWM&psig=AFQjCNElAzQkUB7QSBGKQ4Zdvv6DCo66rA&ust=1374781019819476

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=PuXCmpjuy2uu-M&tbnid=Xdwsu2lXdjFvAM:&ved=0CAUQjRw&url=http://dougmcclure.net/blog/2007/05/dilbert-does-dashboards/&ei=bS3wUebwM4WfqQGFtYCIAQ&bvm=bv.49641647,d.aWM&psig=AFQjCNElAzQkUB7QSBGKQ4Zdvv6DCo66rA&ust=1374781019819476

Technology

Homebrew Your Own Metrics - An IBM Domino Administrator's Guide to SNMP (MWLUG 2014 Session SA103)