Symantec™ Integrated Cyber Defense Exchange SOC

Symantec™ Integrated Cyber Defense Exchange SOC Investigator App for the Elastic Stack Installation and Configuration Guide

September 2019

SOC Investigator App for the Elastic Stack Installation and Configuration Guide 2

Symantec™ Integrated Cyber Defense Exchange SOC Investigator App for the Elastic Stack Installation and Configuration Guide

Documentation version: 1.3

Legal Notice

Copyright © 2019 Symantec Corporation. All rights reserved.

Symantec, the Symantec Logo, and the Checkmark Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

The product described in this document is distributed under licenses restricting its use, copying, distribution, and decompilation/reverse engineering, and subject to an applicable end user license agreement with Symantec. No part of this document may be reproduced in any form by any means without prior written authorization of Symantec Corporation and its licensors, if any. If you have not obtained a license to use the product from Symantec under an applicable end user license agreement, you do not have authorization to use the product or any product documentation, including this document.

THE DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. SYMANTEC CORPORATION SHALL NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH THE FURNISHING, PERFORMANCE, OR USE OF THIS DOCUMENTATION. THE INFORMATION CONTAINED IN THIS DOCUMENTATION IS SUBJECT TO CHANGE WITHOUT NOTICE.

This Symantec product may contain third party software for which Symantec is required to provide attribution to the third party (“Third Party Programs”). Some of the Third Party Programs are available under open source or free software licenses. The License Agreement accompanying the Software does not alter any rights or obligations you may have under those open source or free software licenses.

Please see the Third Party Legal Notice Appendix to this Documentation or TPIP ReadMe File accompanying this Symantec product for more information on the Third Party Programs.

The Licensed Software and Documentation are deemed to be commercial computer software as defined in FAR 12.212 and subject to restricted rights as defined in FAR Section 52.227-19 "Commercial Computer Software - Restricted Rights" and DFARS 227.7202, et seq. "Commercial Computer Software and Commercial Computer Software Documentation," as applicable, and any successor regulations, whether delivered by Symantec as on premises or hosted services. Any use, modification, reproduction release, performance, display or disclosure of the Licensed Software and Documentation by the U.S. Government shall be solely in accordance with the terms of this Agreement.

Symantec Corporation

350 Ellis Street

Mountain View, CA 94043

https://www.symantec.com

https://www.symantec.com/

https://www.symantec.com/


Contents I. Introduction ........................................................................................................... 5

Business Requirement ................................................................................................... 5

What Is ELK? ................................................................................................................. 5

Advantages of Using Elasticsearch, Logstash, and Kibana ............................................ 5

II. About the SOC Investigator App for the Elastic Stack ...................................... 6

III. Overview of the SOC Investigator App for the Elastic Stack Architecture ...... 6

Details of Various Components ...................................................................................... 6

IV. Prerequisites and Hardware Recommendations ............................................... 7

Prerequisites .................................................................................................................. 7

Recommended Hardware Configuration ......................................................................... 7

V. Kibana Dashboards .............................................................................................. 8

Welcome Dashboard ...................................................................................................... 8

Overview Dashboard ...................................................................................................... 9

File Overview Dashboard ..............................................................................................10

URL Overview Dashboard .............................................................................................11

Threat Overview Dashboard ..........................................................................................12

URL Investigation Dashboard ........................................................................................13

File Investigation Dashboard .........................................................................................14

Endpoint Investigation Dashboard .................................................................................15

Threat Investigation Dashboard .....................................................................................16

VI. Installing Pre-built Panels .................................................................................. 17

VII. Installation and Configuration ........................................................................... 18

Installing and Configuring the SOC Investigator App .....................................................18

Importing Visualization Panels in Kibana .......................................................................23

Cloning Kibana Components .........................................................................................23

Upgrading the SOC Investigator App .............................................................................27

Upgrading the SOC Investigator App Installed on Elastic Stack 6.x to 7.x ........................................................................................................................27

Uninstalling the SOC Investigator App ...........................................................................28

VIII. Known Limitations and Troubleshooting ......................................................... 30

Limitations .....................................................................................................................30


Troubleshooting .............................................................................................................30 Data Is Not Displayed on a Dashboard Panel ......................................................................... 30

The setup.py Script Gets Stuck During Installation ................................................................. 30

Input Controls Show an Error Message .................................................................................. 31

Input Controls Show an Alert Icon and Message .................................................................... 31

Labels of Pie Chart Are Not Displayed in Full ......................................................................... 32

Slow Rendering Dashboards ................................................................................................... 32

Events Are Getting Rejected ................................................................................................... 32


Introduction This document provides overall specifications for the SOC Investigator App for the Elastic Stack, built for Symantec, Inc. It contains details of app specifications and workflows that are executed as part of this integration.

Business Requirement Symantec Integrated Cyber Defence Exchange (ICDx) provides a holistic view for the SOC Admin to inspect security issues and improve overall security posture of the organization. Integration with the Elastic Stack allows users to see the security incidents that ICDx creates.

What Is ELK? The Elastic Stack is sometimes called the “ELK Stack.” ELK is an acronym for three open source projects: Elasticsearch, Logstash, and Kibana, all developed, maintained, and supported by the company Elastic, and all parts of the Elastic Stack.

• Elasticsearch is a search and analytics engine. • Logstash is a server‑side data processing pipeline that ingests data from multiple sources

simultaneously, transforms it, and then sends it to a "stash" such as Elasticsearch. • Kibana lets you visualize data with charts and graphs in Elasticsearch.

Advantages of Using Elasticsearch, Logstash, and Kibana 1. The Elastic Stack features an elegant user interface that streamlines data analysis.

2. Kibana features out-of-the-box visualizations, including histograms, line graphs, pie charts, etc.

3. Logstash can be configured to accept data from a variety of sources.


About the SOC Investigator App for the Elastic Stack Symantec’s Security Operations Center (SOC) is a centralized unit that deals with security issues on an organizational and technical level.

The primary objective of SOC Investigator App for the Elastic Stack is to provide end users with a platform to highlight malicious files, malicious URLs, risky users, and risky endpoints in graphical representation.

Overview of the SOC Investigator App for the Elastic Stack Architecture

Details of Various Components • ICDx Forwarder – collects machine generated data from multiple Symantec products. The data or

events are then normalized and forwarded to Elasticsearch • Elasticsearch – is the search and analytics engine, the results of which can be seen via different

visualizations in Kibana. It includes a component called Pipeline, which is basically a set of processors which are executed sequentially to process the events

• Kibana - lets users visualize data with charts and graphs for the data in Elasticsearch


Prerequisites and Hardware Recommendations Prerequisites

• Python 2.7.x or later version on the node where Kibana is installed • “Request” module for Python • Elasticsearch – version 6.2.3 until 6.5.x, 7.3.x • Kibana – version 6.2.3 until 6.5.x, 7.3.x • ICDx 1.3.0 and ICDx 1.3.1 • symantec_soc_investigator_app-sxx-x.x.x.zip

NOTE: 1. Ensure that the versions of Kibana and Elasticsearch are the same. In addition, the SOC

Investigator App for the Elastic Stack does not work on any version earlier than 6.2.3. 2. The setup script used to install the SOC Investigator App relies heavily on using several

Kibana APIs, some of which may be in beta stage or are experimental in nature. Therefore, you should install the App only on the specified version of Kibana as mentioned in the pre-requisites.

Recommended Hardware Configuration

Elasticsearch Node - CPU – 4 cores - RAM – 8 GB - Java Heap Size – 4 GB

Kibana Node - CPU – 4 cores - RAM – 2 GB


Kibana Dashboards Welcome Dashboard This main dashboard shows all the installed dashboards. Navigate to any dashboard by clicking its title.


Overview Dashboard


File Overview Dashboard


URL Overview Dashboard


Threat Overview Dashboard This dashboard gives a bird’s eye view of the incoming threat names and their types.


URL Investigation Dashboard


File Investigation Dashboard


Endpoint Investigation Dashboard


Threat Investigation Dashboard


Installing Pre-built Panels The SOC Investigator App comes installed with several dashboards and panels as detailed in the last section. These panels in general display the summary of documents from more than one Symantec product forwarded from ICDx.

Pre-built panels are the panels that are created to display the summary of documents forwarded from a specific Symantec product. The current version of the App includes some pre-built panels created specifically for the Symantec Data Loss Prevention (DLP) product.

These pre-built panels do not come preinstalled but have to be added as per the requirement.

To install pre-built panels in an existing dashboard

1. Open any existing dashboard (for example, the Overview Dashboard), and then click Edit.

2. Click Add, and then in the Visualization search box, search for DLP.

3. Select any panel from the list display to add it to the existing dashboard.


Alternatively, you can add the pre-built panels in a completely new dashboard.

To install pre-built panels in a new dashboard

1. Navigate to the Dashboard menu, and then click Create new dashboard.

2. Click Add, and then in the Visualization search box, search for DLP. 3. Select any panel from the list display to add it to the new dashboard.

NOTE: You can also remove the pre-built panel from the dashboard. Click the gear icon in the upper-right corner of the panel, and then click Delete from Dashboard.

Installation and Configuration Installing and Configuring the SOC Investigator App The following procedure provides instructions for performing a clean installation of the SOC Investigator App. Ensure that you do not have a version of the SOC Investigator App already installed. If there is different version of the App installed, remove it first by following the procedure in Uninstalling the SOC Investigator App.

Downloading the App

SOC Investigator App is hosted on Symantec Connect webpage - https://www.symantec.com/connect/articles/soc-investigator-app-elastic

To install and configure the SOC Investigator App

The Elasticsearch can be deployed on a standalone machine or in a cluster. The Kibana service should be pointing to the Elasticsearch. SOC investigator app for the Elastic Stack will be installed on the node on which Kibana is installed.

The following steps provide step by step instructions to install SOC Investigator App for the Elastic Stack

https://www.symantec.com/connect/articles/soc-investigator-app-elastic


1. Before starting the App installation, ensure that Kibana and Elasticsearch services are running. You can see the active status of Kibana and Elasticsearch by executing systemctl status <service>

2. Before installing, ensure that you do the following:

• Stop the Elasticsearch forwarder, if it is already configured in ICDx. • Delete the index named icdx_events-*. To delete the ICDx index in Kibana, delete the

Index Pattern icdx_events-* using the DELETE <es-host:es-port>/icdx-events-* REST call.

3. Login to the machine where Kibana service is running. 4. Download and unzip the SOCInvestigator-x.x.x.zip file in the /home directory of the

machine. 5. Go to the directory /home/symantec_soc_investigator_app/kibana and run the python

script setup.py 6. Copy kibana/data/symantec_logo.png file in

/usr/share/kibana/optimize/bundles/ directory

7. Change owner and group of /usr/share/kibana/optimize/bundles/symantec_logo.png file to kibana:kibana using the command chown /usr/share/kibana/optimize/bundles/symantec_logo.png kibana:kibana

8. Change value of kibana.defaultAppId to "dashboard/elkicdx-dash-welcome" in /etc/kibana/kibana.yml file

9. Restart Kibana using systemctl restart kibana or service kibana restart 10. Repeat Steps 6 to 9 for each node of Kibana, if multiple Kibana instances are pointing to the

same Elasticsearch cluster and same .kibana index 11. Go to Management console and select icdx_events-* under Index Pattern. Click on the ★ to make

this the default index pattern


12. To configure the Elasticsearch forwarder in ICDx, do the following: • Log on to ICDx. • Click Configuration > Forwarders. • Add an Elasticsearch forwarder. • Fill the following fields

i. Name: any logical name of the Forwarder ii. Source: select all the archives from where events must be forwarded to Elastic iii. Hosts: it is to filled in this format <ip>:<port> where the IP and Port both belong

to the node where the Elastic service is hosted. (The default port for Elasticsearch is 9200)

iv. User Name and Password: of the Elastic service • Click Show Advanced and set the following parameters as indicated:

For Elastic Stack 6.x: • Index Name: icdx_events-* • Document Type: icdx_event • Pipeline ID: icdx-pipeline • Filter = This field should be set to filter the type of events that must be forwarded

from ICDx to Elasticsearch.


For Elastic Stack 7.x: • Index Name: icdx_events-* • Document Type: _doc • Pipeline ID: icdx-pipeline • Filter = This field should be set to filter the type of events that must be forwarded

from ICDx to Elasticsearch.


13. Start the Elasticsearch forwarder.


Configuring using SSL

• If your Elastic service is running over a secured connection you must enable SSL and SSL verification in the ICDx Elasticsearch forwarder settings to ingest events in Elasticsearch seamlessly

• Enable the SSL toggle • Add the complete path of the self-signed certificate file saved the on ICDx node • Enable SSL verification

Importing Visualization Panels in Kibana The Python script detailed in Installing and Configuring the SOC Investigator App creates all the required dashboards. However, if you accidentally delete a dashboard, you can restore it by running the Python script setup.py again. Note that running the script overrides the changes made to the app’s components like dashboards, panels, searches, etc. It is recommended that you clone dashboards/panels/searches first before upgrading to preserve the changes made in the app’s components.

Cloning Kibana Components All the components in Kibana are separated. In order to clone the entire dashboard and the panels within it, you must clone the dashboard and all panels and searches associated with that dashboard.

The following steps provide instructions to clone dashboards/panels/searches in Kibana to preserve the changes made in the app’s components.

To clone a Kibana visualization

1. Go to Kibana > Visualize, and then select the visualization that you want to clone.


2. Click Save.

3. Click Save as new a visualization, and then click Confirm Save.

To clone a Kibana dashboard

1. Go to Kibana > Dashboard, and then select the dashboard that you want to clone.


2. Click Clone.

3. Click Confirm Clone.

To clone Kibana saved searches

1. Go to Kibana > Discover, and then click Open to open existing saved searches.


2. Select any saved search that you want to clone.

3. In the opened search window, click Save.

4. Click Save as a new search, and then click Confirm Save.


Upgrading the SOC Investigator App To upgrade the app, run the setup.py script that is provided in the zip file of the build. The script can identify if an existing installation of the SOC Investigator App is present and performs the necessary upgrade and complies with the latest version automatically. If no existing build is present, the script installs the app. Note the following regarding the app upgrade:

• The upgrade only updates the components ( dashboards, panels, etc.) that were deployed during the initial installation of the app. The upgrade does not affect any custom components created by the user.

• Any changes made to the app’s components like dashboards, panels, searches, etc. are overridden during the upgrade process.

Upgrading the SOC Investigator App Installed on Elastic Stack 6.x to 7.x The following steps provide instructions to upgrade the SOC Investigator App installed if you are on an existing Elastic 6.x environment but want to move to Elastic 7.x without losing the data.

1. In ICDx, stop the Elasticsearch forwarder. 2. Upgrade Elasticsearch and Kibana from 6.x to 7.x. 3. Start Elasticsearch and Kibana after the successful upgrade. 4. Ensure that the Elasticsearch and Kibana services are running. You can see the active status of

Kibana and Elasticsearch by executing systemctl status <service>. 5. Execute the Python script setup.py, which installs the SOC Investigator App on Elastic Stack 7.x. 6. In ICDx, edit your configuration of the Elasticsearch forwarder as follows:

Old configuration:

• Index Name: icdx_events-* • Document Type: icdx_event • Pipeline ID: icdx-pipeline

New configuration:

• Index Name: icdx_events-* • Document Type: _doc • Pipeline ID: icdx-pipeline

7. In ICDx, restart the Elasticsearch forwarder.


Uninstalling the SOC Investigator App The following procedure shows you how to uninstall the SOC Investigator App from Kibana.

To uninstall the SOC Investigator App

1. Log on to ICDx. 2. Click Configuration > Forwarders, and then stop the Elasticsearch forwarder. 3. On Kibana, in the left navigation pane, click Dashboard. Select all the dashboards that you want

to delete, and then click Delete Selected to delete them permanently.

4. Click Management > Kibana > Index Patterns. Select the icdx_events-* index, and then click

the trashcan (delete) button to delete the index pattern.


5. Similar to Step 3, delete Searches and Visualizations starting with ICDx/DLP/WSS prefix from Management > Saved Objects

6. Remove pipeline and mapping by executing the following REST calls from Dev Tools menu: a. DELETE _ingest/pipeline/icdx-pipeline b. DELETE _template/icdx


Known Limitations and Troubleshooting Limitations

1. Elasticsearch is a distributed search and analytics engine. If you configure your Elasticsearch cluster with an insufficient number of Elasticsearch nodes, or with nodes having insufficient resources, then ingesting and querying excessively large number of events may slow down the rendering of Kibana dashboard visualizations.

2. Also, Elasticsearch aggregations are performed using the concept of buckets. A bucket is simply a collection of documents that meet certain criteria. In some cases, if the number of events returned in a single bucket in response to a query is larger than the maximum configured bucket size, displayed results may be incomplete.

Troubleshooting Data Is Not Displayed on a Dashboard Panel What to Check:

• Check the time range and set it to a suitable range. • Check if the ICDx Elasticsearch forwarder is up and running. Also, check and confirm the

configuration saved in the forwarder settings. • If only some panels are getting populated, check if raw events are being ingested in Elastic via

the Discover tab. • Check for any probable network issue between Elasticsearch node and ICDx.

How to Fix:

• Increase the time range. • Go to the Discover tab and check for data in the specified time range. • Confirm if the event is available on the Discover tab for the query applicable to the panel where

the data is not being displayed.

The setup.py Script Gets Stuck During Installation What to Check: The console on which the script is being run displays an appropriate error message. How to Fix: Rerun the setup.py script.


Input Controls Show an Error Message Issue: The input controls or custom filter of the app show an error message similar to the following:

What to Check: The panels below are be populated. How to Fix: This is the default functionality of Kibana and is not an issue. This is shown when there is no data for the given time range. Increase the time range to populate the filter and the panels below.

Input Controls Show an Alert Icon and Message Issue: In Elastic 7, you may see an alert icon and message stating: Terms list is incomplete. Adjust the autocomplete settings in kibana.yml for more results.

How to Fix: This is default behaviour of Kibana when it gets “terminated_early : true” in the response from Elasticsearch. To remove this alert message from showing up in Kibana, set kibana.autocompleteTimeout: 30000 and kibana.autocompleteTerminateAfter: 2000000000 settings in kibana.yml and restart Kibana. NOTE: This setting is only available in Kibana 7.3. For more information, see: https://www.elastic.co/guide/en/kibana/7.3/settings.html

https://www.elastic.co/guide/en/kibana/7.3/settings.html


Labels of Pie Chart Are Not Displayed in Full How to Fix: This is a known limitation of Kibana. A workaround is to see the panel in Full Screen by clicking the three dots provided in the upper-right corner of the panel. Alternatively, you can hover over the slices of the pie-chart to display the complete information.

Slow Rendering Dashboards How to Fix:

• Reduce the time range to narrow down the search, thereby aiding quick retrieval of results. • Ensure that the Java heap size is set to 4 GB or above. Heap size can be changed by editing the

Xmx and Xms parameters in /etc/elasticsearch/jvm.options. • Ensure that not more than 50% of your total RAM is set as the total heap size.

Events Are Getting Rejected What to Check:

• If the document count ingested in Elasticsearch is significantly less than the number of the events forwarded by ICDx, you should check the Elasticsearch logs for any possibility of events getting rejected.

• The following errors in Elasticsearch logs indicate that a document is rejected. Check Elasticsearch logs /var/log/elasticsearch/<cluster.name>.log.

[2019-02-08T10:51:57,463][DEBUG][o.e.a.b.TransportShardBulkAction] [node-2] [test][2] failed to execute bulk item (index) index {[test][doc][2], source[{ "time" : "2018-04-11T23:34:50.480Z" }]}org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [time] of type [long]at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:301) ~[elasticsearch-6.5.3.jar:6.5.3]


[2019-02-15T19:34:08,721][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [node-2] failed to put mappings on indices [[[test/svblgwG4Q-Wg7zKfWm2y2Q]]], type [doc] java.lang.IllegalArgumentException: mapper [name] of different type, current_type [long], merged_type [text] at org.elasticsearch.index.mapper.FieldMapper.doMerge(FieldMapper.java:352) ~[elasticsearch-6.5.3.jar:6.5.3]

This could be due to different data types for the same attribute in events forwarded from ICDx. For example, in the sample logs above, the time attribute is of long and text data types for two different events. How to Fix:

• This is a known limitation of ICDx. The ICDx team is working to resolve it. • As a workaround, add the time attribute to the Excluded Attributes list in the Advanced

Elasticsearch Forwarder settings in ICDx.

• Restart the Elasticsearch forwarder from the More menu option provided on the right side of the

selected forwarder.

Documents

Symantec™ Integrated Cyber Defense Exchange SOC