Understanding Encrypted Monitoring and Forensics · •Certificates, SNI, Ciphersuites, Extensions ... 520 2048 776 1024 4096 768 512 1016 3072 Key Size (bits) Public Key Lengths

Understanding Encrypted Traffic Using "Joy" for Monitoring and Forensics

Bill Hudson, [email protected] and Trust Organization

DEVNET-1218

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#DEVNET-1218

• Introduction

• Encrypted Traffic

• What Data is Available

• Enhanced Telemetry

• What is “Joy”

• Differentiating Traffic with TLS

• Using Machine Learning Classifiers

• Conclusion

Agenda


Network encryption is increasingly important

• Distributed security architectures

• Public Cloud

• Private Cloud

• Zero Trust

• Virtualization

• Data privacy and security

• Government Regulations

• Healthcare, Banking, etc.

• Sophisticated attackers

Gartner predicts that by 2019, 80% of all traffic on the network will be encrypted!

DEVNET-1218 5


Trusted Man-In-The-Middle Inspection

InternetPremises

MITM

Detects

Malicious

Behavior

DEVNET-1218 6


Trusted Man-In-The-Middle Inspection

InternetPremises

MITM

Detects

Malicious

Behavior

Certificates

Computational CostSecurity &

Privacy

DEVNET-1218 7


Where do we need to go?

• Know about crypto vulnerabilities, attacks, threats

• Know about malicious communication

• Minimal use of MITMs

DEVNET-1218 8


Where do we need to go?

• Know about crypto vulnerabilities, attacks, threats

• Know about malicious communication

• Minimal use of MITMs

DEVNET-1218 9

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco PublicDEVNET-1218 10


Server

Destination Address

Destination Port

Session

# Bytes

# Packets

Client

Source Address

Source PortTCP/IP

DEVNET-1218 11


Server

Destination Address

Destination Port

Session

# Bytes

# Packets

Packet Lengths

Packet Arrival Times

Client

Source Address

Source PortTCP/IP

Intraflow

DEVNET-1218 12


TLSv1.2 metadata

DEVNET-1218 13


Server

Destination Address

Destination Port

Certificate Chain

Selected Ciphersuite

Session

# Bytes

# Packets

Packet Lengths


Client

Source Address

Source Port

Ciphersuite Offer Vector

Extensions Offer

Supported Elliptic Curves

SNI

TCP/IP

Intraflow

TLS

Record Length

Record Times

Record Types

DEVNET-1218 14


Server

Destination Address

Destination Port

Certificate Chain

Selected Ciphersuite

Response Code

TTL

Headers

Session

# Bytes

# Packets

Packet Lengths


Client

Source Address

Source Port

Ciphersuite Offer Vector

Extensions Offer

Supported Elliptic Curves

SNI

Name

Headers

TCP/IP

Intraflow

TLS

DNS

HTTP

Co

nte

xtu

al

Flo

ws

Record Length

Record Times

Record Types

Headers

File Magic

DEVNET-1218 15

Enhanced Telemetry


src dst

Enhanced Telemetry Data Types

• SPLT – Sequence of Packet Lengths and Arrival Times

• Byte Distribution

• Byte Entropy

• TLS unencrypted header data

• Certificates, SNI, Ciphersuites, Extensions

• DNS linked flows

• HTTP linked flows

17DEVNET-1218

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18DEVNET-1218

Sequence of Packet Lengths and Times

src dst

Time

Clie

nt p

acke

tsS

erv

er

pa

cke

ts

"packets": [

{ "b": 22, "ipt": 33, "dir": ">" } ,

{ "b": 1432, "ipt": 4, "dir": "<" } ,

{ "b": 30, "ipt": 1, "dir": ">" } ,

{ "b“: 4, "ipt": 145, "dir": "<" },

...

]


Byte Distribution and entropy

“entropy": 7.165,

"bd": [

23, 7, 4, 8, 4, 12, 7, 4,

12, 5, 98, 6, 5, 101, 14, 8,

9, 9, 6, 8, 10, 6, 10, 6,

16, 8, 3, 16, 7, 7, 3, 11,

189, 6, 24, 9, 10, 10, 5, 7,

19, 8, 16, 8, 34, 79, 61, 90,

102, 91, 56, 47, 35, 47, 30, 25,

...

]

What is “Joy”?


”Joy”

joypcap

joy

Offline

Collector

exporter collector

joyq.py

sleuth

Model.py

json

json

joyAvailable at https://github.com/cisco/joy

joy jsonOnline


Download, Build, Run, Install

• Download• git clone https://github.com/cisco/joy

• Build * (for windows there is a Visual Studio project file)• cd joy

• ./config

• make

• Run• ./bin/joy [options]

• (Optional) Install• Make install

DEVNET-1218 22

https://github.com/davidmcgrew/joy


joy [OPTIONS] file1 [file2 ... ]

23DEVNET-1218

General options

-x F read configuration commands from file F

interface=I read packets live from interface I

promisc=1 put interface into promiscuous mode

output=F write output to file F (otherwise stdout is used)

logfile=F write secondary output to F (otherwise stderr used)

count=C rotate output files so each has about C records

upload=user@server:path upload to user@server:path with scp after rotation

keyfile=F use SSH identity (private key) in file F for upload

anon=F anonymize addrs matching the subnets listed in file F

retain=1 retain a local copy of file after upload

nfv9_port=N enable Netflow V9 capture on port N

verbosity=L verbosity level: 0=quiet, 1=pkt metadata, 2=payloads

https://github.com/cisco/joy/doc/using-joy-05.pdf


joy [OPTIONS] file1 [file2 ... ]

24DEVNET-1218

Data feature options

bpf="expression" only process packets matching BPF "expression”

zeros=1 include zero-length data (e.g. ACKs) in packet list

bidir=1 merge unidirectional flows into bidirectional ones

dist=1 include byte distribution array

entropy=1 include byte entropy

tls=1 include TLS data (ciphersuites, record lengths, ...)

exe=1 include information about host process assoc w/flow

classify=1 include results of post-collection classification

num_pkts=N report on at most N packets per flow (0 <= N < 200)

idp=N report N bytes of the init data packet of each flow

label=L:F add label L to addrs that match the subnets in file F

model=F1:F2 change classifier parameters, SPLT=F1, SPLT+BD=F2

URLmodel=URL specify URL to update classifier data

URLlabel=URL specify URL to update label data

dns=1 include dns names

hd=1 include header description

wht=1 include walsh-hadamard transform


Using joy to process PCAP files

25DEVNET-1218

sh$ bin/joy bidir=1 http=1 dns=1 tls=1 dist=1 output=test.gz test.pcap

sh$ ./sleuth --pretty test.gz | less


> bin/joy bidir=1 tls=1 dns=1 http=1 output=sample.gz tls12_handshake.pcap

> ./sleuth sample.gz

26DEVNET-1218

{"pr": 6, "tls": {"tls_crandom":

"0a9aab7b80199b7965aab25ee11ac9d9d562d541e2a7e0015010c0385d744ba2", "tls_ov": 5, "SNI":

["www.facebook.com"], "tls_ext": [{"data": "00130000107777772e66616365626f6f6b2e636f6d",

"length": 21, "type": "0000"}, {"data": "", "length": 0, "type": "0017"}, {"data": "00",

"length": 1, "type": "ff01"}, {"data": "0008001d001700180019", "length": 10, "type":

"000a"}, {"data": "0100", "length": 2, "type": "000b"}, {"data": "", "length": 0, "type":

"0023"}, {"data": "000c02683208687474702f312e31", "length": 14, "type": "0010"}, {"data":

"0100000000", "length": 5, "type": "0005"}, {"data": "", "length": 0, "type": "0012"},

{"data": "", "length": 0, "type": "ff03"}, {"data":

"001604030503060308040805080604010501060102030201", "length": 24, "type": "000d"}],

"srlt": [{"b": 196, "tp": "22:1", "ipt": 0, "dir": "<"}], "cs": ["c02b", "c02f", "cca9",

"cca8", "c02c", "c030", "c00a", "c009", "c013", "c014", "0033", "0039", "002f", "0035",

"000a"]}, "ts": 1491509125.654878, "sp": 38388, "packets": [{"b": 201, "ipt": 0, "dir":

"<"}], "ob": 201, "da": "31.13.69.228", "ottl": 64, "sa": "10.0.2.15", "te":

1491509125.654878, "dp": 443, "op": 1}

{"pr": 6, "tls": {"srlt": [{"b": 74, "tp": "22:2", "ipt": 0, "dir": "<"}], "tls_srandom":

"b1ae9dda9138839f8d338138727c931587b2248712bef8fbbca710b110b10245", "s_tls_ext": [{"data":

"", "length": 0, "type": "0000"}, {"data": "00", "length": 1, "type": "ff01"}, {"data":

"03000102", "length": 4, "type": "000b"}, {"data": "", "length": 0, "type": "0023"},

{"data": "0003026832", "length": 5, "type": "0010"}], "scs": "c02b", "tls_ov": 5}, "ts":

1491509125.663982, "sp": 443, "packets": [{"b": 2760, "ipt": 0, "dir": "<"}, {"b": 729,

"ipt": 0, "dir": "<"}], "ob": 3489, "da": "10.0.2.15", "ottl": 64, "sa": "31.13.69.228",

"te": 1491509125.664122, "dp": 38386, "op": 2}


> bin/joy bidir=1 tls=1 http=1 output=sample.gz rc4_sample.pcap

> ./sleuth --pretty --select sa,da --where “tls{scs}=0004” sample.gz

27DEVNET-1218

{

"sa": "192.168.56.117",

"da": "192.168.56.202”

}

https://github.com/cisco/joy/doc/using-joy-05.pdf

Differentiating Traffic with TLS


Passive Network Crypto Audit

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

520 2048 776 1024 4096 768 512 1016 3072

Key Size (bits)

Public Key Lengths

0

0.05

0.1

0.15

0.2

0.25

0.3

c02f c028 0035 c02b c014 c030 0004 c013 002f c027

hex code

Selected Ciphersuites

FIPS and PCI Compliance

RC4


SSL implementation detection

• What devices and applications use unpatched software?

0.9.8

1.0.0

1.0.1

1.0.2


Vulnerability Detection - Heartbleed

0.9.8

1.0.0

1.0.1

1.0.2

TLS pad extension to fix

TLS hang bug


Improved Threat Detection

• Independent source of weak convictions

• Reduces overall false positive rate

Using Machine Learning Classifiers


Flow classification

Classifier

Flow

Records

sh$ joy bidir=1 dist=1 classify=1 capture.pcap > capture.gz

sh$ joyq.py capture.gz --where "p_malware > 0.01"


Training architecture

Malware

Detonation

Training

Benign

Records

Malware

RecordsClassifier

analysis/model.py


Results

• L1-logistic regression

• SPLT + 7-tuple + BD

• 172.2 non-zero parameters

• 0.01 FDR: 0.1%

• Total Accuracy: 96.1%

• L1-logistic regression

• SPLT + 7-tuple + BD + TLS

• 137.2 non-zero parameters

• 0.01 FDR: 90.4%

• Total Accuracy: 99.6%


Combining views of data features reduces false positives

(v: 1.0.1r)

(v: 52.0)

+

+

+

+

Firefox C2

Bestafera


Conclusions

• Machine learning and rules applied to passively obtained network data features can

• Detect malware communication

• Detect misused or unpatched cryptography

• SPLT, Byte Distribution, and TLS header data are valuable

• Training classifiers is key!

• Better than MITM with respect to security, privacy and cost

• “Joy” open source package implements these features

• Support: best effort mail alias [email protected]

38DEVNET-1218

https://github.com/davidmcgrew/joy


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#DEVNET-1218


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/


Continue Your Education

• Demos in the World of Solutions!

• Enterprise Networking Area• Encrypted Traffic Analysis (ETA) with the new C9300 switch!

• Security Section• Encrypted Traffic Analysis (ETA) integrated in the Stealthwatch!

• Walk-in Self-Paced Labs

• Tech Circle

• Meet the Engineer 1:1 meetings

• Self Paced Workbook

• https://github.com/cisco/joy/doc/workbench.pdf

• ETA Overview Documentation

• http://cisco.com/go/eta

41DEVNET-1218

https://github.com/cisco/joy/doc/workbench.pdf

https://github.com/cisco/joy/doc/workbench.pdf

Thank you

Documents

Understanding Encrypted Monitoring and Forensics · •Certificates, SNI, Ciphersuites, Extensions ... 520 2048 776 1024 4096 768 512 1016 3072 Key Size (bits) Public Key Lengths