Upload
gregmefford
View
1.460
Download
3
Embed Size (px)
Citation preview
Grokking GrokA Magic Show of
Regular Expressions
@ferggo (Twitter)GregMefford (GitHub)
Key Take-Away:
Check Out Logstash
Logstash Does Lots of ThingsOutputsInputs Filters
Codecs
Grok
Grok is Magic
Grok is Magic(thanks, @jordansissel )
Grok
https://flic.kr/p/8zAUi6
TransmutationTurning Lead into Gold
TransmutationSpinning Straw into Gold
https://flic.kr/p/j4Jg1u
In TheoryFirewalls are simple
https://xkcd.com/730/http://www.startrek.com/database_article/scott
there’s Variety (T_T)In The Enterprise™
“Syslog”
“Syslog”
“Syslog”
“Syslog”
https://www.etsy.com/listing/154952800/unicorn-poo-adjustable-ring-polymer-clay
SparklyUnicorn
https://www.etsy.com/listing/154952800/unicorn-poo-adjustable-ring-polymer-clay
SparklyUnicornPoo
https://www.etsy.com/listing/154952800/unicorn-poo-adjustable-ring-polymer-clay
SparklyUnicornPoo
sometimesadjustable?
Cisco ASA
<134>Sep 02 2014 11:50:10: %ASA-6-302013: Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80)
<134>Sep 02 2014 11:50:10: %ASA-6-302014: Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs
Cisco ASA
<134>Sep 02 2014 11:50:10: %ASA-6-302013: Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80)
<134>Sep 02 2014 11:50:10: %ASA-6-302014: Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs
grok { match => ["message", "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_msg}" ]}
<134>Sep 02 2014 11:50:10: %ASA-6-302013: […]{ "@timestamp" => "2014-09-02T15:50:10.000Z", "cisco_tag" => "ASA-6-302013", "cisco_msg" => "[…]"}
Cisco ASA
cisco_msg:
Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80)
Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs
Cisco ASA
cisco_msg:
Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80)
Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs
http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs.html
http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs.html
http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs.html
730Pages!?
http://geektyrant.com/news/2013/1/10/the-ultimate-movie-training-montage.html
grok {match => [ "cisco_msg", "%{CISCOFW106014}", "cisco_msg", "%{CISCOFW106015}", "cisco_msg", "%{CISCOFW106021}", "cisco_msg", "%{CISCOFW106023}", "cisco_msg", "%{CISCOFW110002}", # ... "cisco_msg", "%{CISCOFW302010}", "cisco_msg", "%{CISCOFW302013_302014_302015_302016}", "cisco_msg", "%{CISCOFW302020_302021}", "cisco_msg", "%{CISCOFW305011}", "cisco_msg", "%{CISCOFW313001_313004_313008}"]}http://www.gregmefford.com/blog/2014/09/24/analyzing-cisco-asa-firewall-logs-with-logstash/
302013: Built {in|out}bound TCP connection <ID> …
302014: Teardown TCP connection <ID> …
302015: Built {in|out}bound UDP connection <ID> …
302016: Teardown UDP connection <ID> …
CISCOFW302013_302014_302015_302016
Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80)
{ "action" => "Built", "direction" => "inbound", "protocol" => "TCP", "src_interface" => "inside", "src_ip" => "10.0.1.1", "src_port" => "1234", "dst_interface" => "outside", "dst_ip" => "10.0.2.2", "dst_port" => "80"}
Ta-da!
Bonus!
Transfooooorm!(ation)
http://www.deviantart.com/art/Hungry-Luma-210132138
Check Point FW-1
<166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;
Check Point FW-1
<166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;
Transfooooorm! (ation)grok {match => ["message", "^<%{POSINT:syslog_pri}>%{WORD}: + (?<cp_time>%{MONTHDAY}[a-zA-Z]{3}%{YEAR} %{TIME}) + %{WORD:action} +%{IP} +%{DATA:interface} + %{GREEDYDATA:cp_msg}" ]}
<166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 …
{ "@timestamp" => "2015-05-11T18:48:00.000Z", "action" => "drop", "interface" => ">bond1.5" "cp_msg" => "rule: 150; rule_uid: […]"}
Transfooooorm! (ation)rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;
kv { source => "cp_msg" value_split => ":" field_split => ";" trimkey => " " trim => " "}
Transfooooorm! (ation)rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;{ "rule": "150", "rule_uid": "{DEADBEEF-4444-5555-6666-DECAFBAD1234}", "rule_name": "CleanUp", "src": "5.6.7.8", "dst": "9.10.11.12", "proto": "udp", "product": "VPN-1&FireWall-1", "service": "domain-udp" "s_port": "67890", "product_family": "Network",}
Transfooooorm!(ation)
http://www.deviantart.com/art/Hungry-Luma-210132138
Transfooooorm! (ation)mutate { rename => [ "dst", "dst_ip", "src", "src_ip", "s_port", "src_port", "proto", "protocol", "service", "dst_port", "interface", "src_interface" ]}
Transfooooorm! (ation)rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;{ "rule": "150", "rule_uid": "{DEADBEEF-4444-5555-6666-DECAFBAD1234}", "rule_name": "CleanUp", "src_ip": "5.6.7.8", "dst_ip": "9.10.11.12", "protocol": "udp", "product": "VPN-1&FireWall-1", "dst_port": "domain-udp" "src_port": "67890", "product_family": "Network",}
Ta-da!
Thanks!@ferggo (Twitter)GregMefford (GitHub)
{ "hostname": "FileServer.example.com", "EventType": "AUDIT_SUCCESS", "Severity": "INFO", "EventID": 5145, "SourceName": "Microsoft-Windows-Security-Auditing", "Channel": "Security", "Category": "Detailed File Share", "SubjectUserName": "somebody", "SubjectDomainName": "DOMAIN1", "ObjectType": "File", "IpAddress": "67.89.12.34", "ShareName": "\\\\*\\MyFiles", "ShareLocalPath": "\\??\\E:\\MyFiles", "RelativeTargetName": "Documents\\Somebody", "AccessMask": "0x80", # …}
grok { match => [ "ShareName", "\\\\\*\\%{GREEDYDATA:Share}" ]}mutate { add_field => { "ShareFullPath" => "\\%{hostname}\%{Share}\%{RelativeTargetName}" }}
grok { match => [ "ShareLocalPath", "\\\?\?\\%{DATA:LocalPath}(\\)?$" ]}mutate { add_field => { "LocalFullPath" => "%{LocalPath}\%{RelativeTargetName}" }}
{ "hostname": "FileServer.example.com", "ShareName": "\\\\*\\MyFiles", "ShareLocalPath": "\\??\\E:\\MyFiles", "RelativeTargetName": "Documents\\Somebody", "ShareFullPath": "\\\\FileServer.example.com\\MyFiles\\Documents\\Somebody", "LocalFullPath": "E:\\MyFiles\\Documents\\Somebody", "AccessMask": "0x80", # …}
Ta-da!
ruby { code => "mask = event['AccessMask'].to_i(16)field_names = { 0 => 'READ_DATA_LIST_DIRECTORY', 1 => 'WRITE_DATA_ADD_FILE', 2 => 'APPEND_DATA_ADD_SUBDIRECTORY', 3 => 'READ_EA', 4 => 'WRITE_EA', 5 => 'EXECUTE_TRAVERSE', 6 => 'DELETE_CHILD', 7 => 'READ_ATTRIBUTES', 8 => 'WRITE_ATTRIBUTES', 16 => 'DELETE', 17 => 'READ_CONTROL', 18 => 'WRITE_DAC', 19 => 'WRITE_OWNER', 20 => 'SYNCHRONIZE'}event['AccessMaskFields'] = Hash.newfield_names.each do |index, name| event['AccessMaskFields'][name] = mask[index] unless mask[index].nil?end"}
Bonus!
{ "AccessMask": "0x80", "AccessMaskFields": { "READ_DATA_LIST_DIRECTORY": 0, "WRITE_DATA_ADD_FILE": 0, "APPEND_DATA_ADD_SUBDIRECTORY": 0, "READ_EA": 0, "WRITE_EA": 0, "EXECUTE_TRAVERSE": 0, "DELETE_CHILD": 0, "READ_ATTRIBUTES": 1, <==== "WRITE_ATTRIBUTES": 0, "DELETE": 0, "READ_CONTROL": 0, "WRITE_DAC": 0, "WRITE_OWNER": 0, "SYNCHRONIZE": 0 }}
Bonus!
CISCO_TAGGED_SYSLOG:^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}( %{SYSLOGHOST:sysloghost})?:%%{CISCOTAG:ciscotag}:
<134>Sep 02 2014 11:50:10: %ASA-6-302014: [...]
Behind the Scenes: Grok is Magic
syslog_pri timestamp ciscotag
CISCO_TAGGED_SYSLOG:^<(?<syslog_pri>\b(?:[1-9][0-9]*)\b)>(?<timestamp> %{MONTH} + %{MONTHDAY}(?: %{YEAR})? %{TIME})((?<sysloghost> (?:%{HOSTNAME}|%{IP})))?:%(?<ciscotag> [A-Z0-9]+-%{INT}-(?:[A-Z0-9_]+)):
Behind the Scenes: Grok is Magic
CISCO_TAGGED_SYSLOG:^<(?<syslog_pri>\b(?:[1-9][0-9]*)\b)>(?<timestamp>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b + (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?: (?>\d\d){1,2})? (?!<[0-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]) (?::(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))(?![0-9]))((?<sysloghost>(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)|(?:%{IPV6}|%{IPV4}))))?:%(?<ciscotag>[A-Z0-9]+-(?:[+-]?(?:[0-9]+))-(?:[A-Z0-9_]+)):
Behind the Scenes: Grok is Magic
IPV6:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
Behind the Scenes: Grok is Magic
CISCOFW302013_302014_302015_302016:
%{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for
%{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( \(%{IP:src_mapped_ip}/%{INT:src_mapped_port}\))?(\(%{DATA:src_fwuser}\))? to
%{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( \(%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}\))?(\(%{DATA:dst_fwuser}\))?
( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( \(%{DATA:user}\))?
Behind the Scenes: Grok is Magic
302013: Built {in|out}bound TCP connection <ID> …
302014: Teardown TCP connection <ID> …
302015: Built {in|out}bound UDP connection <ID> …
302016: Teardown UDP connection <ID> …
CISCOFW302013_302014_302015_302016
Behind the Scenes: Grok is Magic
Built inbound TCP connection 123456789 for
inside:10.0.1.1/1234 (10.0.1.1/1234)
to outside:10.0.2.2/80 (10.0.2.2/80)
(302013)
Behind the Scenes: Grok is Magic
action direction protocol
src_interface src_ip & src_port src_mapped_ip & _port
dst_interface dst_ip & _port dst_mapped_ip & _port
connection_id
Teardown TCP connection 123456789 for
inside:10.0.1.1/1234 to outside:10.0.2.2/80
duration 0:00:00 bytes 420 TCP FINs
(302014)
Behind the Scenes: Grok is Magic
action protocol
src_interface dst_interface dst_ip & _port
duration
src_ip & _port
bytes reason
connection_id