© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Keeping your HP ArcSight connectors healthy Tracy Barella Chief Services Strategist

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2

Agenda

HP ArcSight Connector Health Check

• What is a Health Check?

• Health Check steps by ArcSight component

• Connectors

• Connector Appliances

• Q & A

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Health Check overview

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4

What is a health check?

Purpose The purpose of performing a health check is to identify and remove performance bottlenecks to enable top performance of the HP ArcSight implementation. Minor issues can result in major performance degradations over time impacting system availability and user satisfaction. Performing regular health checks will identify issues allowing them to be remediated quickly and ensure continued top performance of the HP ArcSight implementation. In a nutshell… A Health Check consists of common administrative tasks to verify the ArcSight solution is configured and performing optimally.

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Health Check steps by ArcSight component Note: It’s impossible to cover every scenario in this presentation, so only the common checks will be discussed.

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6

Health check steps by ArcSight component

1 Loggers

CPU, Memory, and EPS In/Out Check

Search Performance Check

Custom Report Performance Check

Receivers and Forwarders Check

Storage Group Check

Index Configuration Check

Configured Alerts Check

Scheduled Task Check

Event Archive and Configuration Backup Check

Logger System Health and Audit Event Forwarding Check

Network Configuration Check

Online Event Storage Check (Only Software-based or SAN Logger)

ESM Database and storage

DBCheck and Oracle RDA

‘Database Performance Statistics’ Dashboard Check

Partition Check (Oracle)

Trend Jobs Check

Hardware and Operating System Check

CPU and Memory Utilization Check

Oracle version and patch level check

Oracle alert log check

Oracle memory parameters check

ESM Database Storage Check

ESM Manager

‘Event Throughput’ Dashboard Check

‘Current Event Sources’ Dashboard Check

Hardware and Operating System Check

CPU and Memory Utilization Check

ESM Manager JVM (memory) Utilization Check

Data Monitor Utilization Check

Active List/Session List Utilization Check

Rules Engine Check

Event Persistence (insertion) Performance Check

Error Check

Scheduled Task Check

server.properties Check

Agent and Console Threads Check

Connector appliances

Version Check

CPU and Memory Check

Network Settings Check

Configuration Backup Check

Connectors

Up/Down Check (Connector or Container)

Version Check

Connector Event Rate Check (by EPS)

Cache Check

Logs Check

Configuration Check

Connectors Tip: Check each ArcSight Component by the order of the Event Flow

It’s just simple plumbing!!!

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7

Connectors

Connector (or Container) Up/Down Check

Connector Version Check • Are there any Connectors running a version older than ~1 year? • A minimum version of 4.8.1 is required to leverage the ESM v5.2 schema.

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8

Connectors (cont.) Connector Cache Check • All Connectors should have 0 events in the cache

– If most Connectors are ‘continuously’ caching = Possible ESM level ‘Event Insertion’ problem

– If one or two Connectors are ‘continuously’ caching = Possible Connector level problem or network issue – If a Connector caches for a moment and then clears the cache (batched events) = This is normal

Connector Event Rate Check (by EPS) • Are there any Connectors receiving a high event rate?

See below for definition of high EPS on ‘common’ Connector types: – Syslog Connector or CheckPoint Connector : >= 1,500 EPS

– Windows Unified Connector: > 500 to 1,000 EPS – DB-based Connector or SourceFire eStreamer Connector: >=

200 EPS

• Is the high EPS Connector stable? If not, we should recommend another Connector to spread the load?

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9

Connectors (cont.)

Connector Logs Check • ../current/logs/agent.out.wrapper.log

– Java Heap Memory Utilization

• Memory utilization • Frequency of Full GCs • Memory in Red Zone alerts

– Unexpected Connector restarts – Connectivity errors

• End Devices

• ArcSight Destinations

• ../current/logs/agent.log – Parsing errors

– DOSProtector – Chronic WARN and ERROR messages

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10

Connectors

Connector Logs Check (cont.) • Use Connector LogFu to graph the event

flow and memory utilization – ../current/bin/arcsight agent logfu –a

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 11

Connectors (cont.)

Connector Configuration Check

• Destination Settings – Are there more than 2 Destinations on each

Connector? • Too many Destinations can negatively

impact performance of a Connector. – Common problems found:

Networks and CustomerURI are not applied on every Connector

Fields-based Aggregation is not properly applied (by Connector Type)

No tuning (Filter Out) applied on high EPS Connectors

Settings are not the same on every Destination (ESM, Logger, etc.)

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 12

Connectors (cont.)

Connector Configuration Check (cont.) • Only check the following on ‘problematic’ Connectors discovered in previous checks

– ../current/user/agent/agent.properties Optimal settings are different for each Connector type High EPS Connectors (>1200 EPS) such as Syslog, WUC, CheckPoint, and Blue Coat can be ‘tweaked’ quite a bit here

– ../current/user/agent/agent.wrapper.conf Only increase the Java Heap size if memory issues were found in agent.out.wrapper.log Default Java Heap is 256MB

Maximum configurable Java Heap is 1024MB (1 GB)

Reminder: If you have 50+ Connectors in your environment, try to stay focused on problematic Connectors!

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13

Health check steps by ArcSight component

1 Loggers

CPU, Memory, and EPS In/Out Check

Search Performance Check

Custom Report Performance Check

Receivers and Forwarders Check

Storage Group Check

Index Configuration Check

Configured Alerts Check

Scheduled Task Check

Event Archive and Configuration Backup Check

Logger System Health and Audit Event Forwarding Check

Network Configuration Check

Online Event Storage Check (Only Software-based or SAN Logger)

ESM Database and storage

DBCheck and Oracle RDA

‘Database Performance Statistics’ Dashboard Check

Partition Check (Oracle)

Trend Jobs Check

Hardware and Operating System Check

CPU and Memory Utilization Check

Oracle version and patch level check

Oracle alert log check

Oracle memory parameters check

ESM Database Storage Check

ESM Manager

‘Event Throughput’ Dashboard Check

‘Current Event Sources’ Dashboard Check

Hardware and Operating System Check

CPU and Memory Utilization Check

ESM Manager JVM (memory) Utilization Check

Data Monitor Utilization Check

Active List/Session List Utilization Check

Rules Engine Check

Event Persistence (insertion) Performance Check

Error Check

Scheduled Task Check

server.properties Check

Agent and Console Threads Check

Connector appliances

Version Check

CPU and Memory Check

Network Settings Check

Configuration Backup Check

Connectors

Up/Down Check (Connector or Container)

Version Check

Connector Event Rate Check (by EPS)

Cache Check

Logs Check

Configuration Check

Connector Appliances Tip: Check each ArcSight Component by the order of the Event Flow

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 14

Connector appliances

Connector appliance version check • Is the version outdated? • Are there any ‘known issues’ with the current version?

Connector appliance CPU and memory check

• Review the following for excessive utilization: – CPU utilization is continuously above 70-80% in Logger Dashboard – EPS In is continuously above 5,000 EPS (a single C5400 is designed

for 5,000 max EPS) – Check the Connector Appliance’s Monitor Dashboards for unusual

peaks or drops

– Check the System Process Status section of the Connector Appliance – If possible, SSH to the Connector Appliance and run commands such

as top, df, ifconfig, etc. to perform a deeper dive at the OS level

Connector appliance network settings check

• Common problems to check: – Incorrect duplex settings on the network interface – DNS or NTP not configured properly

Connector appliance configuration backup check

• The daily Configuration Backup job should be scheduled on all Connector Appliances.

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Additional resources

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16

My favorite resources for keeping ArcSight healthy!

1. Any HP Protect presentation on ArcSight best practices or troubleshooting: https://protect724.arcsight.com

2. KB Articles on the HP Support Site

3. Solutions listed in previous Support Tickets

4. HP ArcSight University

5. HP ArcSight product documentation

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 17

Please fill out a survey. Hand it to the door monitor on your way out.

Thank you for providing your feedback, which helps us enhance content for future events.

Session TB3257 Speaker Tracy Barella

Please give me your feedback

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Thank you

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.