Click here to load reader

HPESecurity ArcSight Developer'sGuide October17,2017 LegalNotices Warranty TheonlywarrantiesforHewlettPackardEnterpriseproductsandservicesaresetforthintheexpresswarrantystatements

  • View

  • Download

Embed Size (px)

Text of HPESecurity ArcSight Developer'sGuide October17,2017 LegalNotices Warranty...

  • HPE SecurityArcSightConnectors


    Developer's Guide

    October 17, 2017

  • Legal Notices

    WarrantyThe only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statementsaccompanying such products and services. Nothing herein should be construed as constituting an additional warranty.Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

    The information contained herein is subject to change without notice.

    The network information used in the examples in this document (including IP addresses and hostnames) is for illustrationpurposes only.

    HPE Security ArcSight products are highly flexible and function as you configure them. The accessibility, integrity, andconfidentiality of your data is your responsibility. Implement a comprehensive security strategy and follow good securitypractices.

    This document is confidential.

    Restricted Rights LegendConfidential computer software. Valid license from Hewlett Packard Enterprise required for possession, use or copying.Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and TechnicalData for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license.

    Copyright Notice Copyright 2016, 2017 Hewlett Packard Enterprise Development, LP

    Follow this link to see a complete statement of copyrights and acknowledgements:


    Phone A list of phone numbers is available on the HPE Security ArcSight Technical SupportPage:

    Support Web Site

    Protect 724 Community

    Contact Information

    Developer's Guide

    HPE Connectors Page 2 of 241

  • Revision History

    Date Description

    10/17/2017 l Updated "Set Global Parameters" section to include encryption parameters.

    l Updated information for downloading SQL Server JDBC drivers.

    l In Appendix E: ArcSight Built-in Event Field Mappings, the ArcSight Mappingsfields have been changed to camel case.

    l Only non-blocking I/O is available for syslog connectors; therefore, thetcpmaxidletime, tcpsetsocketlinger, and tcppeeerclosedchecktimeout parametersare no longer relevant and have been removed from the Advanced Parametersappendix.

    l As flexString fields are for the use of customers, examples have been updated toshow deviceCustomString or deviceCustomNumber fields rather than flexStringfields.

    05/15/2017 l Added a notice about ODBC connections not being supported after release 7.2.1 tothe "ArcSight FlexConnector ID-Based Database", "ArcSight FlexConnector MultipleDatabase", and "ArcSight FlexConnector Scanner Database" sections.

    02/15/2017 l Added JSON to the list of available extra processors. See "Extra Processors".

    l Clarified the configuration file names and locations for vulnerabilities, open ports,and URIs for scanner FlexConnectors for normal text reports. See "GettingVulnerabilities for Scanned Hosts", "Getting Open Ports on Scanned Hosts", and"Getting OS and Applications (URIs) on Scanned Hosts".

    11/30/2016 l Updated installation procedure for setting preferred IP address mode. UpdatedFlexConnector information for IPv6-aware parsers.

    08/30/2016 l Reorganized and expanded content for increased usability.

    l Updated the "Configure the JDBC Driver and Windows Authentication section.

    l In "Advanced Parameters", updated information regarding preservestateparameters.

    06/30/2016 l Added parameters to Parameters Common to all SmartConnectors.

    Developer's Guide

    HPE Connectors Page 3 of 241

  • Date Description

    05/16/2016 l Updates and clarifications in the Log Rotation Types section. Added a section onthe unparsed events detection feature: Unparsed Events Detection.

    l In "Advanced Parameters", clarified the descriptions of several advancedparameters.

    03/31/2016 l Added advanced parameters to customize connector behavior as Chapter 5.

    l In "Advanced Parameters", noted that for Syslog connectors, thepersistenceinterval parameter must be a positive integer to enable persistence.

    l In "Advanced Parameters", noted that the rawlogfolder and usefilequeueparameters cannot be applied to Syslog Pipe/File Connector.

    l Added configuration properties for JSON Folder Follower FlexConnector.

    l Noted that only one question mark is supported for time-based databaseFlexConnector queries.

    l Removed agents[x].maxfilesize parameter.

    02/15/2016 l End of life for FlexConnector SNMP (install the SmartConnector for SNMP Unified).

    l Added the new feature to detect and log unparsed events.

    l Updated the time format for __parseMutableTimeStamp function.

    l Updated the wildcard parameter default value to use *.

    Developer's Guide

    HPE Connectors Page 4 of 241

  • Contents

    Chapter 1: Overview 13

    FlexConnector Development 13IPv6-Aware Parsers 14

    Event Fields 14Operations 14Developer Considerations 15

    Folder Structure 15Key Files 16

    FlexConnector Management 16ArcSight Connector Appliance 16ArcSight Management Center 17

    Chapter 2: Choose a FlexConnector Type 19

    FlexConnector Types 19

    Event Data Format Examples 21Log File FlexConnector 22ID-Based Database FlexConnector 22JSON Folder Follower FlexConnector 22Multiple Database FlexConnector 23Regex FlexConnectors (Variable-Format File FlexConnectors) 24Scanner FlexConnector 24SNMP FlexConnector 25Syslog FlexConnector 27Time-Based Database FlexConnector 27XML File FlexConnector 28

    Chapter 3: Install and Configure the FlexConnector 30

    FlexConnector Installation 30Install Core Software 30Set Global Parameters (Optional) 31Select Connector and Add Parameter Information 32

    ArcSight FlexConnector File 33ArcSight FlexConnector ID-Based Database 34ArcSight FlexConnector JSON Folder Follower 36

    Developer's Guide

    HPE Connectors Page 5 of 241

  • ArcSight FlexConnector Multiple Database 37ArcSight FlexConnector Multiple Folder File 40ArcSight FlexConnector Regex File 41ArcSight FlexConnector Regex Folder File 42ArcSight FlexConnector REST 43ArcSight FlexConnector Scanner Database 44ArcSight FlexConnector Scanner Text Reports 47ArcSight FlexConnector Scanner XML Reports 49ArcSight FlexConnector XML File 50ArcSight FlexConnector Simple Network Management Protocol (SNMP Unified) 51ArcSight FlexConnector Syslog 52

    Select a Destination 53Complete Installation and Configuration 53

    Additional Configuration for Database Connectors 54Install SQL Server JDBC Driver 54Install MySQL Driver 56Add a JDBC Driver to the Connector Appliance/ArcSight Management Center 57Configure the JDBC Driver and Windows Authentication 57Oracle 8i Support 58Troubleshooting Duplicate Events 59

    Example 1: ID-based Database Connectors Only 59Example 2: ID-based and Time-based Connectors 60Example 3: Complex Main Query with a Join 60

    Chapter 4: Create a Configuration File 62

    Parser File Locations and Names 62

    Example Parser File 63

    Parser File Structure 64Token Declarations 65Token Types 66Event Mapping 66RequestUrl Event Field 66Operations Table 67Severity Mapping 68

    Examples 68Extra Processors 69Key-Value Parsers 71

    FlexConnector Creation Wizard for Delimited Log Files 72

    Developer's Guide

    HPE Connectors Page 6 of 241

  • Regex Tool for Regex FlexConnectors 75

    Start the FlexConnector 78

    Chapter 5: Configuration File Examples 79

    Configuration Properties for a Log File FlexConnector 79

    Configuration Properties for all Regex FlexConnectors 80

    Configuration Properties for a Time-based Database FlexConnector 81Version 81Query 82Timestamp 83UniqueID 83

    Configuration Properties for an ID-based Database FlexConnector 83Version 84MaxID 84Query 84ID 85UniqueID 85Query Limit 85

    Configuration Properties for an SNMP Connector 85

    Configuration Properties for an XML FlexConnector 87Namespace 88Hop Nodes 88Trigger Nodes 88Token Mappings 89

    Examples of Token Mappings 89Extra Events 90

    Configuration Properties for a JSON Folder Follower FlexConnector 90Trigger Node 92Token Location and Mappings 92JSON Parsers for Complex Event Schemas 92

    Working with Hierarchical Schemas 92Representing a JSON Array with a Key Element 94Representing a Token Value in URI Format 95Sample JSON Array 96

    Configuration Properties for Scanner FlexConnectors 96Scanner FlexConnectors for Normal Text or XML Scan Reports 96How Scanner FlexConnectors Parse Scan Reports 97

    Developer's Guide

    HPE Connectors Page 7 of 241

  • Parser Files for Normal Text Reports 97Getting a List of Hosts 98

    Ignore or Include Line 99Regular Expression and Token Mappings 99Use IP 100Invalid Vulnerabilities 100Extra Events 100

    Getting Vulnerabilities for Scanned Hosts 101Token Mappings 102Event Mappings 102Severity Mappings 103Ignore or Include Line 103

    Getting Open Ports on Scanned Hosts 104Token Mappings 105Event Mappings 105Ignore or Include Line 105

    Getting OS and Applications (URIs) on Scanned Hosts 106Token Mappings 106Event Map