McAfee Enterprise Security Manager · Writing Custom Parsing Rules in McAfee ESM Page 3 of 20. 3 Creating a Custom ASP Rule . Follow the steps below to build a new SYSLOG data source

The information contained in this document is confidential and proprietary. Please do not

redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 1 of 20

McAfee Enterprise Security Manager

Writing Custom Parsing Rules in McAfee ESM

September 20, 2018




Table of Contents 1 Introduction...................................................................................................................................................................... 2

2 Prerequisites..................................................................................................................................................................... 2

3 Creating a Custom ASP Rule ............................................................................................................................................. 3

3.1 Create a New Data Source ......................................................................................................................................... 3

3.2 Create a Custom Parsing Rule for ASP ....................................................................................................................... 3

3.3 Enable the New ASP Rule in Policy .......................................................................................................................... 14

4 Creating a New Custom Type ......................................................................................................................................... 15

4.1 Create a Custom Type ............................................................................................................................................. 15

5 Testing the New ASP Rule .............................................................................................................................................. 17

5.1 Importing a Log Sample ........................................................................................................................................... 17

6 Best Practices ................................................................................................................................................................. 18

6.1 Know Your Data ....................................................................................................................................................... 19

6.2 Regular Expressions ................................................................................................................................................. 19

6.3 Content .................................................................................................................................................................... 19

1 Introduction This guide details how to create and deploy a custom advanced syslog parser (ASP) rule in the McAfee ESM.

A custom ASP rule could be necessary if there is an unknown/unparsed log in the McAfee ESM. This guide

also includes some “Best Practices” to follow when writing ASP rules.

2 Prerequisites The following will be required, or useful, in creating custom parsing rules:

• A sample of the log to parse.

• Account privileges to add data sources. In some cases, administrator privileges, such as the ability to

define custom types may also be needed.

• Vendor documentation can be helpful to determine the meaning of each field in the logs desired to

be parsed.

• A working knowledge of PCRE (Perl Compatible Regular Expressions).




3 Creating a Custom ASP Rule Follow the steps below to build a new SYSLOG data source and create an ASP parsing rule.

3.1 Create a New Data Source This step is optional if a data source already exists and is collecting the logs needing to be parsed.

1. Select the Receiver, and then click on the “+” icon to add a data source.

2. The Add Data Source window will open, set the Data Source Vendor as Generic and the Data Source

Model as Advanced Syslog Parser. Give the new data source a name, add the proper IP address,

and ensure that the Support Generic Syslogs field is left as Do nothing.

3. Select OK, and then apply the new data source settings by confirming in the next

window. This will create a new data source with an empty rule policy.

3.2 Create a Custom Parsing Rule for ASP

1. From the McAfee ESM console, select the data source and open the Policy Editor.




2. The Policy Editor window should appear. In the menu on the left, expand Receiver by selecting the

“+” symbol, and select Advanced Syslog Parser. Initially this list will be blank while looking at the

policy for the new data source.

3. Select New > Advanced Syslog Parser Rule.

4. The Advanced Syslog Parser Rule window will appear. Within the General tab, assign information

as displayed below:




Name – Assign a name, making it as descriptive and unique as possible. This is the event message

that will show in the ESM views when the rule matches a log, unless the message is mapped

directly from the log text within the rule.

Tags – Assign tags to the rule. It is best practice to assign one or more tags to which this rule

belongs. This helps in finding and grouping sets of rules created for a given device or application

within the policy editor. Any tags added to a rule, will cause ESM to automatically include the rule

in any policy that has enabled the given tagged rule set.

Default Normalized ID – The normalization or category is a key step as many views, correlation

rules, and reports use this field as a filter. It is very important to select the most relevant value in

the taxonomy for the parsing rule in order to get the greatest value from within the McAfee ESM.

Default Severity – If the log message does not contain a value for the severity, then the event will

be assigned the value that is set here (the default is 25, with a scale of 1-100, 1 being the lowest

and 100 being the highest).

Rule Assignment Type – Rules can be grouped together; this pull-down menu provides a list of

supported products to group the parsing rules by, separating the events from other data sources.

This allows the event to be reported for a specific product.

Description – Give the new rule a clear human-readable description. It should be a clear and

complete description of the rule with complementary information to what was given in the rule’s

name.

5. Click on the next tab, Parsing, assign information as displayed below:




Process Name – This is an optional filter that acts similar to the content string, but only applies to

the process name found within the SYSLOG header. Due to various syslog header formats used, and

no guarantee that a syslog header will be present in the log, it is recommended to use content

strings instead of the process name match.

Content String – If a fixed string is always going to be found in the log then add it as a content

string. The content string(s) of an ASP rule should uniquely identify each log if possible. Content

strings are very important as they will not only make matches more accurate, but will speed up the

execution of the rules in the running policy. It is highly recommended to have one or more content

strings per ASP rule. This is a pre-filter for optimization – only logs that match the given content

strings will be considered for matching and parsing by the regular expressions. The log must contain

all given content strings, otherwise the logs will not be parsed using the regular expressions.

Regular Expression – The first regular expression determines if the ASP rule will match the log. Any

additional expressions will be used to capture values from the log that may or may not be found in

all logs.

Named Captures – Named captures can be used in regular expressions to more easily identify

capture groups. The label used for the named capture can consist of alphanumeric and

underscore characters but cannot begin with a number or include a space. The regular expression

syntax for a named capture is: (?P<NAME>regular expression capture). Here is an

example of a named capture where hostname is the name assigned to the capture group:

Host\x3d(?P<hostname>\S+). When using named captures the policy editor will display the

capture name instead of the capture number, in the right-hand side of the Parsing tab as shown

below.




Targeted Regular Expressions – Targeted regular expressions allow for applying an additional

regular expression to the resulting value from another capture. For example, a regular expression

could be written to pull a user’s full name from a log. Referencing that capture as the target, two

targeted regular expressions could then be written to pull out the user’s first and last name. This

data can then be mapped to different database fields. Follow the steps below to create a

targeted regular expression:

1. On the Parsing tab, add a regular expression.

2. Add another regular expression and select the capture (name or number) from the Target

key drop-down list that references the first expression. The second regular expression

should contain an expression to capture additional information from the captured value of

the first regular expression.

In the image below, the named capture group full_name has a value of “Sam Johnson”. This

named capture is being used as the Target Key for the first_name and last_name targeted

regular expressions. The regular expression for first_name and last_name will be evaluated

against the value of the full_name named capture. The result shows the key of first_name:1 with

a value of “Sam” and last_name:1 with a value of “Johnson”. This specific example could be done

within a single regular expression, and is a simplified example to illustrate the purpose of the

Target Key field used with regular expressions. In general, it is recommended to run as few

regular expressions as possible, but that may vary with specific use cases.




Sample Log Data – In this field a sample log can be added and will be saved with the rule. The

parts of the log matching defined regular expressions will be highlighted blue.

Format – ASP can pre-process certain logging formats to simplify the mapping of data. The

following formats are available:

Generic – This is the default and should be used if the log does not match the other

available formats.

CEF (Common Event Format) – This eliminates the need to create a regular expression

for each capture, and will allow the data to be mapped using the CEF key names found

in the log.

JSON – Similar to CEF, this eliminates the need to create a regular expression for each

capture, and will allow data to be mapped using the JSON key names found in the log.

XML - Basic, Simple, or Positional – This will allow ASP to parse logs that are in XML

format and assign parsed data. The XML format choice will depend upon the type of XML

that is within the logs.

XML – Basic expects XML without any repeated elements.

XML – Simple expects XML with either a single node with attributes, or a single

set of non-repeated elements without nesting.

XML – Positional expects XML that can have multiple nodes with attributes and

multiple repeated elements with nesting.




Parsed Values – The Key/Value fields on the right displays what is being parsed from the log

sample(s) by the regular expression(s). The Key will display two numbers, divided by a colon. The

first number indicates the regular expression being used, and the second number indicates the

capture group used within that regular expression. If a captured value is the 4th capture in the 3rd

regular expression defined, the key would display “3:4”.

6. Add applicable Content Strings by selecting Edit. These are very important and can drastically

affect performance of the created rule. Any content strings added will need to be a string that is

guaranteed to be within the log, as these act as a type of “prefilter”. Multiple unique content

strings can be added to increase the speed and efficiency of the rule.

7. Select applicable regular expression options by adding checks within the boxes add one or more

Regular Expressions by selecting the “+” symbol.

Only use regular expressions for parsing purposes – Checking this box will remove the

requirement for the first regular expression to match the log. The parser will rely upon the

Content String for matching purposes, and will only use regular expressions to parse data when it

finds a matching pattern.

Case Insensitive – If the log may contain either upper or lower case letters in some fields, it may

be simpler to write the expression in the same case and then use this option. This enables the

case insensitivity option for all regular expressions defined in the parsing rule.

Trigger when data doesn’t match – This option will make the rule trigger when the regular

expression does not match the log.

a. The Edit Regular Expression window will appear. Add the expression and select OK.

8. With the expression added, the parsed values should be highlighted in blue, with the Group and

Value shown to the right. Verify that each value is parsed out correctly and that all of the log line

samples match. Make sure to know the meaning (from the vendor device documentation or own

knowledge) of all the fields within the logs and capture the values that are useful.

9. After all the regular expressions are written to extract out the needed values from the log, select

the Field Assignment tab at the top of the window.




10. Available fields will be displayed, along with the Group and Value section as shown in the

previous Parsing tab.

11. Drag and drop the Values from the right hand side to the Expression column next to the Field on

the left.

a. For more complicated log file formats, multiple regular expressions may be used to

capture the same data. In such cases, a field such as First Time might come from one of

several potential several expressions. McAfee ESM allows the ability to specify the

regular expression values in order of preference, so that if one does not match, it will

use the second, or third, or so on. An example of this is shown below:

12. If the field is not displayed that is needed, click “+” above the Sample Value column, to display all

custom type fields.

13. The Custom Types window will now be displayed. Select the desired field and then select OK.




14. If a custom type does not exist for the type of data being mapped, the McAfee ESM allows for

user-defined custom types to be added. Once a new custom type has been created, it will be

listed within the Custom Types window and be available for use to map the values to. Please see

Section 4 for details on creating a new custom type.

15. Click on the final tab, Mapping. This section will allow parsing the date format, mapping the

action, and mapping the severity found within the log message.




Time Format – The date/timestamp of a log message can be parsed using the variables defined

in these fields. Many standard date/timestamps are recognized automatically by the McAfee

ESM, however there may be a format that is either not recognized or that would be preferred to

display differently. This section will allow for formatting the time to show up in the proper

format when parsed.

Action Mapping – Use this option if there is an action found within the log to be mapped to an

available McAfee ESM action.

Severity Mapping – The severity mapping allows for a value within the log to be mapped to a

severity from 1-100. For example, a vendor might define their severity as either “Low”,

“Medium”, or “High” in their logs. With the Severity Map section the severity value can map

“Low” as 25, “Medium” as 50, and “High” as 75.

16. To add a Time Format, select the “+” symbol in the upper-right of the time format section.

a. The Add Time Format window will appear.

b. Enter a custom time format. As the values are entered, a small window will appear to

assist in creating the custom time. The values can either be typed or selected from the

given list.

c. Select the fields that this custom format will apply to. It is best practice to apply it to

both First Time and Last Time.

d. Select OK to save.

17. To add an Action Map, find an applicable Action Value by scrolling through the list and then click

inside the Action Key column. Enter the value that will be parsed from the device logs. If this

value is parsed and assigned to “Action”, then it will map to the assigned Action Value.




Below the Action Mapping table, the option will be given to use a default action when one is not

specified. This is generally a good option to set in order to insure that all logs receive an action

type. Depending on the device, the default action should be different.

To add the default action map, place a check next to Use the following action for the default if

one is not specified, and then select a default action from the drop down menu.

18. To add a Severity Mapping, select the “+” symbol in the upper-right of the Severity Mapping

section.

a. The Add Severity windows will appear.

b. Within the Severity Mapping field, enter the value that will be parsed from the device

logs. If this value is parsed and assigned to “Severity”, then it will map to the assigned

Severity Value.

c. Move the sliding scale to the desired Severity Value.

d. Select OK to save.

19. Below the Severity Mapping section, the option to add a default severity will be given. This is

generally a good option to set in order to insure that all logs receive a severity value. Depending

on the device, the default severity should be different. To add the default severity, place a check

next to Use the following severity for the default if one is not specified, and then enter a default

value.

20. With the previous steps complete, the new ASP rule should be finished. Select Finish, located in

the bottom-right hand corner to finish and save.




3.3 Enable the New ASP Rule in Policy Once the new rule has been created, it will need to be enabled and added to the policy in order for the

receiver to begin parsing the associated logs.

1. Open the Policy Editor for the data source that the new ASP rule was created for, and beneath

Receiver, select Advanced Syslog Parser. The newly created ASP rule(s) will be listed with the

Action column displaying “disabled”. Click on disabled and then change the value to enabled.

2. With the new rule set as enabled, click on the Rollout icon displayed in the upper-right hand

corner of the Policy Editor window.

3. If the rule didn’t already save previously, it will prompt to save. Select Yes to save.

4. Once saved, the Rollout window will appear and will display a list of devices to roll policy out to.

Ensure that the applicable data source is set to “Roll this policy out now.” and select OK to do

so.




4 Creating a New Custom Type

4.1 Create a Custom Type To create a custom type, login with a user that has system administrator privileges and then select the

System Properties icon.

1. Select the Custom Types option from the menu on the left.

2. Select Add from the options shown on the right.

Note: Select the Edit option to edit a user defined pre-existing custom type if needed.

3. The Add Custom Type window will appear. Setup the new custom type as desired and select OK.




Name – This will be the name for the new Custom Type that will be displayed within McAfee

ESM.

Data Type – This will be the type of data that the McAfee ESM will expect to be populated into

this field. Selection of the correct data type is important. Commonly used data types include:

Random String, IP Address, Decimal, Integer, and GUID.

Events Field – These fields are for Events. There are a specific number of fields that can be used

to have data parsed into. If two different custom types occupy the same field, only one can be

used. When creating custom types, make sure to place it within a field that is not already being

used by another important field that will be used.

Flows Field – These fields are for Flows. There are a specific number of fields that can be used to

have data parsed into. If two different custom types occupy the same field, only one can be used.

When creating custom types, make sure to place it within a field that is not already being used

by another important field that will be used.

Index Data – This option will determine on whether or not the data that is parsed is indexed

within the database. Indexed data is searchable within the McAfee ESM.

Description – This will be a simple description to explain what type of data the newly created

custom type contains.

4. With the desired custom type setting entered, select OK.




5 Testing the New ASP Rule

5.1 Importing a Log Sample Once the new ASP rule has been created and added to the current policy, it is a good idea to test it to

ensure that it is parsing as expected.

1. Obtain sample logs from the device that the new rule was created for and save these to a text

file.

2. Select the data source to be tested and click Properties.

3. The Data Source Properties window will appear, select the Upload button.

4. Navigate to the log sample file and select it to upload. Select the Upload button and then a

confirmation window will be displayed showing the file was successfully uploaded. Select Close

to close out the window.




5. To initiate the log and flow gathering process manually, click on the Get Events and Flows

button.

6. The Get Events and Flows window will appear. Select Events and then click the Start button to

collect the events.

7. Once the events are collected, find the events in the dashboard and verify the newly created ASP

rule is parsing as expected. If any adjustments need to be made, make the changes and then

send the sample logs through again to verify that the rule is parsing as expected.

6 Best Practices The following are given as best practices that can help with creating ASP rules that work well within the

McAfee ESM.

Following these recommendations will help ensure reporting is accurate, not overly complex, and will

help parsers and Receivers maintain intended performance and efficiency.




6.1 Know Your Data It is important to understand the logs that are being sent to the SIEM, and whether or not they have value

in your organizations reporting needs. It is recommended to only parse logs that meet your organization’s

reporting goals.

• Ensure that captured values mapped to ESM database fields align with the intended use of the

specific custom type fields.

• Avoid indexing fields that contain unique and random or high cardinality data such as URLs.

• Ensure rules mapping event messages directly from the log are not mapping unique and random

or high cardinality strings as messages. The ESM creates a data source rule for each unique event

message, and ESM performance can be negatively impacted if there are a high rate of unique

strings being mapped to message.

• Categorize events by adding a normalized category to the rule. Data source rules, generated by

parsing rules, inherit the normalization assigned to the main parsing rule. This means that if the

main parsing rule is left normalized to “Uncategorized”, then the parsed events will also be

normalized as “Uncategorized”, making a search for “Uncategorized” events to find unparsed

events in accurate.

6.2 Regular Expressions By default, the first regular expression determines if the ASP rule will match the log and will need to

match a pattern that is always present in the log. Additional regular expressions can be written to capture

values from the log and mapped to custom types within the McAfee ESM. Subsequent regular expressions

do not determine the rule match, and are used for parsing only.

While it is possible to test regular expression results on a few log lines within the McAfee ESM console

itself, we recommend using a graphical tool. There are many free web based tools that can be used in

addition to standalone installable tools. Optionally, another useful tool would be a text editor that

supports regular expression searches. Any tools used to test regular expressions need to support pcre

expressions.

Make sure that regular expressions are efficiently written because poorly written expressions can take

more system resources to run, and can affect the parsing performance of a specific data source or even

the entire Receiver.

6.3 Content Ensure there is at least one value in the content field section. Content strings should be at least 3

characters in length and should be as unique as possible for the specific event. It is advised to include

enough content matches to uniquely identify the log. Using one or more content fields within the ASP rule

will significantly speed up the matching and parsing process on the Receiver.

The table below shows an example of how to use the content field to uniquely identify a log sample.

Syslog Sample Content Fields




<180>Jan 1 00:00:00 testhost ftpd[4325]: FTP

LOGIN FROM test.org [192.168.1.1], anonymous

“ftpd”

“FTP LOGIN FROM”

<180>Jan 1 00:00:00 testhost ftpd[4325]: FTP

LOGIN REFUSED (ftp not in /etc/passwd) FROM

192.168.1.1 [192.168.1.1], anonymous

“ftpd”

“FTP LOGIN REFUSED”

Documents

McAfee Enterprise Security Manager · Writing Custom Parsing Rules in McAfee ESM Page 3 of 20. 3 Creating a Custom ASP Rule . Follow the steps below to build a new SYSLOG data source