Improving the Effectiveness of Log Analysis with HP …docs.media.bitpipe.com/io_12x/io_128669/item_1260842/SANS...SANS ANALYST PROGRAM 2 Improving the Effectiveness of Log Analysis

A SANS Product ReviewWritten by Dave Shackleford

April 2015

Sponsored by Hewlett Packard Enterprise

Improving the Effectiveness of Log Analysis with HP ArcSight Logger 6

©2015 SANS™ Institute

Most organizations today collect logs and actively use them for monitoring, forensics,

troubleshooting, and detecting and tracking suspicious behavior, according to the ninth

SANS Log Management Survey, in which 97 percent of organizations reported they are

currently collecting and leveraging logs for all of these reasons and more.1 How well they

use logs is another matter entirely.

In the same survey, 50 percent of respondents for whom detection and tracking of

suspicious behavior was a stated need confirmed that such detection and tracking is

moderately difficult to accomplish, with another 30 percent stating that log collection

and analysis is difficult for this purpose. Many organizations are also struggling with

large amounts of log data from a vast variety of distributed sources and are spending

significant amounts of time analyzing logs each week—22 percent of respondents

spend more than one full day per week analyzing logs.

It’s clear that log collection and analysis is a critical aspect for most IT security teams.

However, even with the advances in log management techniques seen in recent years,

many teams are still struggling to get control of their logs and properly manage them,

both effectively and efficiently.

We recently reviewed HP ArcSight Logger 6, which includes significant updates

over earlier releases. The new Logger’s standout features include improved incident

analysis and response flexibility, overhauled reporting and monitoring, and general

enhancements for ease of use.

Our evaluation focused on three areas that HP notably updated and enhanced in Logger 6:

• Flexibility, customization and ease of use

• Security monitoring, investigation and incident analysis

• Reporting

We can summarize our review process using this question: How can this tool help

security analysts and operations teams perform their jobs more effectively? Fortunately,

Logger 6 performed admirably for all the major use cases, and we found numerous

capabilities that would help many organizations improve the effectiveness of their log

management.

SANS ANALYST PROGRAMImproving the Effectiveness of Log Analysis with HP ArcSight Logger 61

Introduction

1 “Ninth Log Management Survey Report,” October 2014; www.sans.org/reading-room/whitepapers/analyst/ninth-log-management-survey-report-35497


Ease of Use

We reviewed ArcSight Logger 6 in a test environment that HP installed and configured,

simulating many events across 20 logging devices to represent a typical enterprise.

The first use case that we explored—flexibility, customization and ease of use—directly

relates to the user friendliness of the dashboards and interfaces available to analysts. Our

first stop was the main dashboard, shown in Figure 1.

Figure 1. ArcSight Logger 6 Main Dashboard

Although Logger 6 includes a number of “stock” dashboards (packaged for various roles

and job functions), we used a dashboard prebuilt by the ArcSight team to demonstrate

what current product users report to be the most popular graphs and charts. The Logger

dashboard shown in Figure 1 provides an at-a-glance view in four categories, listed from

upper left to lower right:

• All Failed Logins by User. Contains the aggregate number of failed login events

across all users and platforms.

• UNIX - All SSH Authentications by User. Displays administrative SSH sessions to

UNIX platforms; this information can assist in monitoring privileged activities.

• NetFlow Top Destination Ports. This panel shows patterns of network traffic

throughout the environment, emphasizing services in active use.

• Intrusion - Malicious Code. Presents all malware-related events and occurrences

within the environment.

Ease of Use (CONTINUED)


Beyond these examples, we noted the flexibility to quickly change between saved

dashboards in a variety of different categories. Custom dashboards are usually where

security analysts spend their time, looking at aggregate events and trends that allow for

easy access to more granular datasets. In the Dashboards menu area, a drop-down list is

available to rapidly switch between saved dashboard views, making it simpler than ever

to navigate to the desired dashboards.

We quickly switched from this original custom dashboard to another one, labeled

“Intrusion and Configuration Events,” that was configured for us. Much like the main

dashboard, the Intrusion and Configuration Events dashboard shows popular and useful

collections of security-related information such as “Top Malicious Code Activity” (upper

left pane), “Top Firewall Drops by Source” (upper right pane) and others, as shown in

Figure 2.

Figure 2. Intrusion and Configuration Events Dashboard

While reviewing the malicious code activity, we noticed a number of events labeled

“ICMP Packet Flood” (upper left pane of Figure 2); these events can signal a potential

denial of service (DoS) attack or hostile network discovery activity.



To get a sense for how simple it is to drill down on events, we simply clicked into the

graph area on the entry for ICMP Packet Flood. Doing so provided more granular results

and automatically redirected us to the Analyze category, as shown in Figure 3.

Figure 3. Drilldown Malicious Events for ICMP Packet Flood

The screen in Figure 3 provided a wealth

of data related to the captured events,

including the time of the events, what

devices observed the events and which

logging engine captured and recorded

the events for analysis. We could also

easily use this data to build a custom

dashboard on the fly, using the top

malicious IP addresses or another data

type from within the events. To create

quick dashboard charts and graphs, all

we had to do was click the “save” button

(in the toolbar on the query response

page) and choose to save to an existing

dashboard or create a new one, as shown

in Figure 4. Figure 4. Creating a Custom Dashboard on the Fly



The Logger interface also allowed us to easily view the overall status of the monitored

systems and events. By selecting the Summary menu item at the top of the dashboard

window, we were able to quickly review the number of different event types across

devices and endpoint agents that forward events to log collector servers in the test

network. Clicking any of the various categories yielded more data, and simple metrics

such as events per second (EPS) could be viewed in the upper-right corner of the main

view, shown in Figure 5.

Figure 5. Global Summary of Events

Having immediate access to a central view of event count, types, systems and logging

platforms (known as “receivers” in Logger jargon) is invaluable to security operations

teams that need to manage a large environment. From this view a security analyst can

immediately determine whether a particular system is seeing a higher count of events

than normal, which receivers are getting the most logs and events sent to them, and

what types and categories of events are being seen most frequently. This visibility allows

large, distributed teams to focus on particular types of events or one or more receivers

that are seeing higher event counts; teams can then scrutinize those platforms to see the

cause of the changes.



Figure 6 shows another view of the entire environment and its performance, with

emphasis and details on receivers, events, utilization and processing stats from the

ArcSight host and, finally, storage.

Figure 6. Logger Monitoring Summary of the Environment

This view presents a wide range of data, including CPU usage for the Logger platform

over specified time periods, total event flow, receiver status and a list of storage

repositories defined for use within the event management infrastructure. This data is

valuable for security professionals who need to keep up with changes in performance

and events over time, as well as operations teams that need to track how much space is

in use for event storage.

One of the best ease-of-use features we were able to test was also one of the simplest:

the direct navigation query field (shown at the top of the screen throughout the UI).

This intelligent search query box autopopulates suggestions based on keywords or

even just letter combinations and strings that a user

types, making it exceedingly simple to locate various

dashboard pages, analysis pages, specific data types

and other UI elements. Figure 7 shows a search that

starts with the term “Data” and the suggested search

options that Logger 6 automatically creates. Figure 7. Dynamic Search Query

Field



The Logger interface was incredibly simple to use. Within seconds, enormous amounts

of data were readily visible and available, and finding specific events, dashboards,

metrics and other important elements of the monitoring environment was easy.

More importantly, any salt-worthy analyst should be able to rapidly get up to speed

on how Logger works, where to find data of interest, and how to create and monitor

custom dashboards. This element is critically important for most enterprises that

are struggling with the increasing volume of log data in their environments. The

respondents to the latest SANS Log Management Survey were in many cases spending

hours—or even days—each and every week simply analyzing logs and trying to bring

log management under control. Security analysts will be as efficient and effective as

their log management products are easy to learn and use. Logger 6 should enable any

organization to cut the time needed to perform maintenance, keep the systems up and

running properly, and track events for security monitoring and response.

The second area we focused on in this review is the real-life applicability of the

product—its usability and effectiveness for security operations team members who

would need to:

• Monitor event data

• Quickly identify unusual behavior or events that warrant attention and

investigation

• Use the product as an aid to assessing incidents and identifying the root cause of

security issues

We began evaluating Logger’s capabilities by reviewing some of its monitoring

dashboards. The first dashboard we looked at was Login and Connection Activity, shown

in Figure 8.

Figure 8. Login and Connection Activity Dashboard

This dashboard displays the total failed logins, both by “product” (system type) and

user name. In our test network, the majority of failed logins occurred within the UNIX

environment, which would immediately cause an experienced analyst to wonder:

• What is happening in the UNIX environment—are we under attack?

• Are we seeing brute force authentication attacks?

• Is an application or service rejecting logins for some unknown reason?


Security Monitoring and Incident Response

Security Monitoring and Incident Response (CONTINUED)


We can also pinpoint the user accounts experiencing the most login failures and

determine whether these failures correlate with the failed logins for UNIX servers.

Clicking the graph labeled “Top Failed Logins by User” provided details of specific

account activity, as shown in Figure 9.

Figure 9. Account Login Failure Detail

We compared users and the failed logins to their accounts with ease. We then had

the option to click on individual users to get more detail on when and where each

failed login occurred, as well. Such details are useful for any security analyst who is

investigating a potential breach or suspected account compromise, because correlation

with specific times and dates of other activities will likely be useful.



One of the most practical and useful features that can aid in monitoring and investigation activities is the “free text search” function within the Analyze category. As we found, entering a keyword into the search field triggers Logger to provide options for filter and event selection, as well as a search history, examples and suggestions for additional search operators that fit with the entered keyword. An example of this feature, with a simple search for the term “netflow,” appears in Figure 10.

Figure 10. Free Text Search

A more advanced and specific query for “netflow” and top destination ports was simple to create using Logger’s flexible and reasonably intuitive syntax. (An analyst might use such a query when looking for network scanning in the environment or for actively seeking out top data flow destinations.) The syntax for this query was netflow | top dpt, and its search results came back in seconds, as shown in Figure 11.

Figure 11. A More Targeted Logger Query



Although this query is a simple example, Logger has an enormous number of syntax

options, so analysts will definitely need to take some time to get comfortable with many

of them.

The result of this query—as Figure 11 shows—was that “123” was the top netflow

destination port, which may indicate (in normal situations) traffic headed to the Network

Time Protocol (NTP) service or, alternatively, a new channel for malware distribution or

some other attack. (This column appears in light blue.)

We easily expanded the query to determine what the top source addresses (senders)

are for these data flows, using the syntax netflow | where dpt=123 | top

sourceAddress, as shown in the query interface in Figure 12.

Figure 12. Filtering Netflow Source Addresses to Port 123

(Note that Logger retrieved our search operator history, based on the string we entered.)

Another example we explored was searching for all information and events related to

our testbed’s IDSes. We performed a free-form query for “ids,” and within a few seconds

Logger returned a distillation of all events and IDS platforms producing log and alert

events in the environment, as shown in Figure 13.

Figure 13. Querying All IDS Events



The results showed us what the IDSes were reporting, which is usually a valuable start

to network intrusion analysis. We could query with ease across all such devices by

using advanced syntax (ids AND categoryDeviceGroup CONTAINS "IDS/

Network" | top categoryTechnique) to evaluate their responses against a list

of most-frequently-used attack types. This query, shown in Figure 14, ranks the attacks

detected by the test environment’s IDS platforms.

Figure 14. Searching for Top IDS Attack Categories

(This query took just over five seconds to process and report on more than 514,000

aggregate events, doing so in real time.)

We kept exploring our use case, entering even more detailed queries and examining

known exploits and vulnerabilities in the environment. In particular, we explored a

common scenario in enterprise security monitoring environments.

The premise in this case was based on a “new” attack profile—identified either by a

member of the IT operations or information security team or through a vendor-supplied

IDS update. After the IDS sensors were updated with the signatures for this attack, how

would an analyst go about seeing whether the signature tripped all the sensors in the

environment?



Enter ArcSight Logger. Because we were concerned with one event type only, we could

easily build on the last IDS sensor query we created to find out whether any of our IDS

sensors spotted this attack. Assuming the name of the IDS event was “HTTP IIS Root.exe

Execute Command,” we could add this event name to our existing query, to end up with

the following:

ids AND categoryDeviceGroup CONTAINS "IDS/Network" | where

categoryTechnique="/Exploit/Vulnerability" | where name="HTTP

IIS Root.exe Execute Command"

The query also appears in Figure 15, which shows the view after the dashboard

dynamically updated with the new query.

Figure 15. A Targeted Query for a Specific Exploit

The results provided us with useful tactical data on which to focus. We could see how

many events came in and when those events took place. We could also see which

sensors detected the events; such information can help analysts pinpoint what services

are targeted and where the attacks are happening.

We also noticed that Logger assists analysts in constructing queries by providing

unobtrusive suggestions, which appear in Figure 15 as “Examples” (below the search

operator history).



We finished this example by finding the top sources of this attack—which we could then

use for firewall rules, IP blacklists or other monitoring efforts—simply by adding the filter

top sourceAddress to the query, as shown in Figure 16.

Figure 16. Top Malicious Source Addresses

With a list in hand of IP addresses that were sending malicious exploits and attacks to

systems in our environment, we could add these addresses to firewall filtering and block

rules, watch lists for monitoring additional activity, or threat intelligence cases in case

they represent part of a larger attack campaign.



We explored several additional scenarios where information security and IT operations

teams may need to monitor user activity in the environment for troubleshooting

or other reasons. First, we looked for the top source IP addresses connecting to the

corporate VPN services to determine who was connecting and how often, using a simple

query of vpn | top sourceAddress, as shown in Figure 17.

Figure 17. Top VPN Access by Source IP Address

As before, we could drill down into any areas of the graph, providing further visibility

into who was connecting and from where. (Incidentally, this data could also help us in

areas such as license or network management.)

For example, we clicked the first result shown in Figure 17 (indicated by arrow)—the IP address 10.0.27.221—which took us to a detailed view of exactly when this address connected to the VPN. We also loaded the same query with a saved search that the ArcSight product team created for this review; this appears in Figure 18 in the row labeled “VPN Connections.”



Figure 18. A Saved Search for VPN Connections

For larger environments generating many events and numerous Logger receivers collecting and aggregating data, Logger allows the receiver platforms to be peered together, facilitating searches across them all. In addition, analysts can search against the local log repository they’re accessing or across them all very simply.

We spent by far the most time during our review exploring real-world use cases and features within the Dashboards and Analyze menus that analysts would find tactically useful in their jobs. To summarize our experiences:

• Logger provides a large number of out-of-the-box dashboards and analysis categories that can quickly get security teams up to speed, whether they are merely setting up monitoring or launching a time-critical investigation.

• The free-form search capabilities within Logger are spectacular and allowed for highly intuitive and rapid query creation that returned results in seconds.

• Logger users can easily create dashboards on the fly, as well as filters and queries within multiple areas of the product that users can save for later use. Logger also remembers the most recent history of queries and filters.

• Although the syntax for creating queries is not overly complex, an enormous number of options is available, which may take time for analysts to learn and understand fully. The suggestions provided in the Logger UI go a long way to mitigate this wide span of options.

In our final area of review, we looked at the newly enhanced reporting facility in Logger

6. (The previous versions of Logger’s reporting engine were highly capable but also

complex and potentially challenging to use, by HP’s own admission; the new version of

the product offers significantly streamlined reporting.) First, the reporting engine now

has a dashboard with the most-frequently-run report types, based on the queries and

graphs used; the reporting dashboard within our test environment included reports for

bandwidth usage by source IP address, top IDS alerts and several others the ArcSight

product team added to the testbed as examples. Our reporting dashboard appears in

Figure 19.

Figure 19. Reporting Dashboard


Reporting

Reporting (CONTINUED)


The Report Explorer (in the menu bar at left) is also easy to navigate and offers analysts

the ability to run on-demand, prebuilt reports for compliance and monitoring tasks.

Figure 20 shows a custom report we ran that searched the collected logs for database

errors and warnings.

Figure 20. Database Reports in Report Explorer



Customizing reports—in case analysts need to modify parameters such as the period

to examine, the device groups from which events should be selected or the storage

locations for events—is a simple task. In Figure 21, we changed the window for our

database report to 30 days’ worth of events.

Figure 21. Report Customization

The report ran in seconds, and we could easily export it to a PDF or email. Security or

operations teams could easily use such a report to discover database issues.



A more security-centric report that we explored was the “SANS Top 5 Log Reports”—a

canned report that HP includes with the product, based on a SANS reference document.2

We ran the first log report listed, which showed attempts to gain access to the environment

through existing accounts, with failed logins as the primary event type. Figure 22 shows

us selecting this report, which was easy to find within the Report Explorer menu.

Figure 22. Running the SANS Top 5 Log Reports

2 “Top 5 Essential Log Reports,” Version 1.0; www.sans.org/security-resources/top5-logreports.pdf



The report finished quickly, and the output is shown in Figure 23.

Figure 23. The Final SANS Top Failed Logins Report

Customizing any report was easy. Selecting the Customize Report link when running

a report enables analysts to add new graphs or data, include custom headers and

graphics, or add or remove detail to tailor the report for different audiences.

The reporting engine was so simple to use that we had a solid grasp on features and

navigation within a brief time. Security teams will appreciate how easy it is to create new

reports, customize existing reports, and schedule reports to run regularly and deliver

them via email to analysts or management for review. Reporting is a critical part of

security monitoring and event analysis, and the easier it is, the better.

Reporting is a critical

part of security

monitoring and event

analysis, and the

easier it is, the better.

Security analysts who need to collect and monitor logs look for certain key features in a

product:

• Scalability and performance. The ability to collect, analyze, and search across

logs quickly is paramount.

• Flexibility. Customization in queries and dashboards will be essential to handle

any number of unforeseen cases and scenarios that come up over time.

• Reporting. Any log management product should come with a variety of prebuilt

reports and offer analysts the ability to create new and customized reports easily.

• Powerful analysis tools. Security teams want the tools they use daily to have

features that enable powerful searches across logs and provide the ability to drill

down into data for granular viewing.

• Broad support for log and event data. A log management platform should be

able to consume many different log data types and formats.

HP ArcSight Logger 6 offers analysts all these capabilities and more. We found the

product to be intuitive and easy to use, with powerful features that can save analysts

time in analyzing and reporting on events within their environments.


Conclusion

Dave Shackleford is the founder and principal consultant with Voodoo Security, a SANS analyst,

instructor and course author, and a GIAC technical director. He has consulted with hundreds

of organizations in the areas of security, regulatory compliance, and network architecture and

engineering. He is a VMware vExpert and has extensive experience designing and configuring secure

virtualized infrastructures. He has previously worked as chief security officer for Configuresoft and

CTO for the Center for Internet Security. Dave is the author of the book Virtualization Security (Sybex).

Recently, Dave co-authored the first published course on virtualization security for the SANS Institute.

Dave currently serves on the board of directors at the SANS Technology Institute and helps lead the

Atlanta chapter of the Cloud Security Alliance.


About the Author

Sponsor

SANS would like to thank its sponsor:

Documents

Improving the Effectiveness of Log Analysis with HP …docs.media.bitpipe.com/io_12x/io_128669/item_1260842/SANS...SANS ANALYST PROGRAM 2 Improving the Effectiveness of Log Analysis