View
2
Download
0
Category
Preview:
Citation preview
1
Asi@Connect DCNDSBotnet Monitoring & Mitigation Workshop
20-21 November 2019
Network and Security MonitoringNurefsan Sertbas
2
Agenda
1. What is network monitoring?
– How to monitor?
– Monitoring network by NetFlow
2. Fault Detection
– Intrusion Detection Systems
• Understand your network by Zeek
– Honeypots
§ Summary
3
The big picture
Detection
ReactionRecovery
Prevention
Secure System design /Classic IT-Security Intrusion Detection
Systems (IDS)
Firewalls
Firewalls
Honeypots
PDRRProcess
4
Network Monitoring
§ What do we want to know? – who accessed – when, where and how – whose and which resource?– Performance, quality– which protocols and services?
§ Why do we want to know?1. Fault Management/Intrusion Detection2. Analysis
• System & Service monitoring for reachability, availability,..• Resource Management (e.g., capacity planning)• Performance Monitoring & Optimization (e.g., RTT, throughput,..)
3. Statistics for accounting & metering• Bill customers according to usage
This Photo by Unknown Author is licensed under CC BY-SA-NC
5
How to monitor ?
Configure device to generate flow accounting records
Export the flows from the device to a collector
Receive the flows, write them to disk
Analyze the flows
§ Goal:– Collection of useful
information from various parts of the network
– The network can be managed and controlled using the collected information
6
How to monitor ?
Passive Collectore.g., Unix host
Receives a copy of every packet Requires a mirror port Resource intensive
All packets
Flowrecords
7
How to monitor ?
On a network device e.g., router
Device support is neededSome impact on performance
Flowrecords
queries
A flow is defined as a set of packets having common properties such as common packet header fields.
8
Flow Exporting Protocols
§ NetFlow by CISCO
§ Jflow by Juniper
§ NetStream by Huawei
§ IPFIX by IETF
§ s-Flow by Netgear, Dell and HP
§ Rflow by Ericsson
9
NetFlow flow definition
Key Fields
SrcIP 10.,3.3.3
DstIP 10.2.7.2
srcPort 23
dstPort 220178
Layer 3 prot TCP-6
TOS Byte 0
Interface Ethernet
NetFlow enabled device
10
NetFlow
§ Can be used for several application:
– Answers questions regarding IP traffic: who, what, where, when, and how
• Which user / department has been uploading / downloading the most?
• Which are the most commonly-used protocols on my network?
– Identification of anomalies and attacks
– More fine-grained visualization
• Presentation of collected and processed data in a user-friendly format
§ What we cannot achieve by NetFlow monitoring?
11
NetFlow- DoS Attack Example - I
§ Potential DoS Attack on Router § Estimated: 660 pkt/s 0.2112 Mbps
Typical DoS Attacks Have the Same NetFlow Flow Entries(Same Input Interface, Destination IP, 1 Packet per flow and Bytes per packet)
12
NetFlow- DoS Attack Example - II
§ To show all flows to one destination leverage:• “ router#sh ip cache flow | include <destination> “
§ To look for known attack signatures – e.g. if we know of an attack using UDP port 666 (Hex 029A)
• router#show ip cache flow | inc 029A
13
Top NetFlow Analyzers & Collectors
§ Flow Collectors that – receive flow records from exporters – analyze these records to produce sensible information– Present results to user in a user friendly way
§ Some examples for flow collectors– Solarwinds NetFlow Traffic Analyzer– PRTG Network Monitor– Scrutinizer– ManageEngine NetFlow Analyzer– nProbe and ntopng
17
What comes next after Monitoring ?
Detection
ReactionRecovery
Prevention
Secure System design / classicIT-Security
Intrusion DetectionSystems (IDS)
Firewalls
Firewalls
Honeypots
PDRRProcess
Just monitoring does not make much sense
Intrusion PreventionSystems (IPS)
18
§ Overall goal:– Supervision of computer systems and communication infrastructures in order to detect
intrusions and misuse
§ Why detection of attackers? – Full protection is usually not possible!– Security measures too expensive or with too low flexibility, e.g., not possible to build every
functionality in ASICs– Unpatched systems for compliance reasons (medical systems etc.)– …
§ What can be attained with intrusion detection?– Detection of attacks and attackers– Detection of system misuse (includes misuse by legitimate users)
Goal of Intrusion Detection Systems - I
19
Goal of Intrusion Detection Systems - II
§ Using detection system only makes sense if there are consequences!§ Possible goals
– Limitation of damage if (automated) response mechanisms exist– Gain of experience to recover from attack, improve preventive measures– Deterrence of other potential attackers
(if and only if police is able to arrest them!)
Detection
ReactionRecovery
Prevention
PDRRProcess
IDS is a fraction of this step!
20
Operation of Intrusion Prevention/Detection Systems
Events Logging
Automaticreaction
Über-wachung
Terminal
Monitoring/Audit
Central IDS / SIEM
Detection Reaction?
Externalalert
Records all security relevant events of a supervised system
Automatic analysis of audit data
Reporting of detected attacks
Potentially also initiating countermeasures
21
Audit Data
§ Recording security relevant events
From a computer system:Opening of files
Execution of programs
Detected access violation
Failed password verification
From a network:Connection establishment and release
Packets transferred from / to specific systems/ ports
App specific events:Events are application specific and indicate security relevant activities
§ Input for the intrusion detection and response mechanisms• requires integrity protection !
22
Requirements of IDSs - I
§ High accuracy ( ↓ 𝐹𝑃 and ↓ 𝐹𝑁 )
TP FP
TNFNDet
ectio
n
Event Nature
AN
A N
False alarm:Corresponds to an anomalous event that is inoffensive
Harmless event that was successfully labeled as normal
Successful detection of attacks
Attacks that were notdetected
Confusion Matrix Source: [ET04]
23
Requirements of IDSs - II
§ Easy to integrate into a system / network
§ Easy to configure & maintain
§ Autonomous and fault tolerant operation
§ Low resource requirements
§ Self protection, so that an IDS itself can not easily be deactivated by a
deliberate attack (to conceal subsequent attacks)
24
Classification of IDS
Scope
Host based
NW based
Hybrid
Architecture
Centralized
Hierarchical
Distributed
Detection Mechanism
Signature based
Policy based
Anomaly based
25
Classification of IDS - I
Scope
Host based
NW based
Hybrid
analysis of system events
analysis of exchanged information (IP packets)
combined analysis of system events & NW traffic
26
Classification of IDS - Host IDS
§ Works on information available on a system,e.g., OS-Logs, application-logs, timestamps
§ Can easily detect – attacks by insiders,
• as modification of files, • illegal access to files, • installation of Trojans or rootkits
§ Problems: – has to be installed on every system– produces lots of information– often no real-time-analysis but predefined time intervals– hard to manage a large number of systems
27
Example of a Host-Monitor – Osquery (1)
§ Allows to use OS as high-performance relational database– SQL tables representing abstract concepts
§ Fast & tested§ Opensource
https://osquery.io
• running processes• logged in users• password changes• USB devices• firewall exceptions• listening ports• ….
28
Example of an Host-Sensor - Osquery (2)
§ High-performance and low-footprint distributed host monitoring– Query the system in an abstract way– It runs everywhere : Independent of OS, software or hardware configuration
§ Host monitoring daemon– allows to schedule queries to be executed across entire infrastructure– takes care of aggregating query results over time and generates logs which
indicate state changes in the infrastructure
§ Cross platform operating system instrumentation framework for
– intrusion detection,– infrastructure reliability – or compliance monitoring
§ Only monitoring, no intrusion detection capabilities on its own !https://osquery.io
29
§ Works on information provided by the network, mainly packets sniffed from NW layer
§ Existing systems use combination of – signature detection, – protocol decoding, – statistical anomaly analysis
§ Can detect – DoS with buffer overflow attacks, – invalid packets, – attacks on application layer, – DDoS, – spoofing attacks,– port scans
§ Often used on network hubs to monitor a segment of the network
Classification of IDS - Network IDS I
30
Classification of IDS - Network IDS II> Placement of NIDS
LAN
DMZ
InternetProbe monitors all incoming traffic• High load• High rate of false
alarms• Measurement of any
attack attempts
Probe monitors all traffic to and from systems in the DMZ• Reduced amount of data (less
unsuccessful attempts)• Can only detect attacks on these
devices, but potentially revealing compromised LAN devices
Probe monitors LAN traffic• Low load• Detection of inside
attacks (e.g., compromised devices) Switch forwarding all
data to a monitoring port
Central IDS/SIEM
MonitoringNetwork
SIEM = Security information and event management
31
Classification of IDS - II
Architecture
Centralized Hierarchical Distributed
Analysis Unit
Analysis Unit
IDS IDS
Analysis Unit
IDS
CentralAnalysis
Unit
IDS
IDS
IDS
IDS
IDS
IDS
IDS
IDS
IDS
IDS
IDS
IDS
Collaboration [VaKa+15] How to correlate alerts in collaborative scenario ?
33
Signature based Detection§ Basic idea:
– Some attack patterns can be described with sufficient detail ® “attack signatures”– The event audit analyzed if it contains known attack signatures
§ Identifying attack signatures:– Analyzing vulnerabilities– Analyzing past attacks that have been recorded in the audit
§ Specifying attack signatures:– Based on identified knowledge so-called rules describing attacks are specified – Most IDS offer specification techniques for describing rules
§ Drawbacks of signature-based detection:– Requires prior knowledge of potential attacks– Signature database requires continuous updating – High rate of false negatives if signature database is not up to date
34
Signature based Detection – Example: Snort
§ Network IDS and intrusion prevention system
§ Analysis of IP packets in real time
§ Mainly signature based, each intrusion needs a predefined rule
§ Preprocessing àDetection Engine à Action
35
Policy-based Detection
§ Basic Idea– Specify what is allowed in a network and/or what is forbidden– Violations create alerts– In that sense, similar to a Firewall
§ Drawbacks– You can only detect what you configured / what deviates from what you
have configured– Needs expert knowledge of the system to be protected
36
Policy-based Detection – Example: Zeek (aka Bro) (1)
§ Real-time network analysis framework– Primary a network monitoring tool– Can be used for pure traffic analysis– Powerful IDS
§ Focus on– Application-level semantic analysis– Policy-based detection in protocols– Tracking information over time
§ Intrusion prevention– Zeek can act as dynamic and intelligent firewall
[Pa99]
Network
libpcap
Event Engine
Policy Script Interpreter
Packet stream
Filtered packet stream
Event stream
Real-time notificationRecord to diskPolicy script
Event control
Tcpdump filter
37
Policy-based Detection – Example: Zeek (2)
Network
Programming Language
Packet Processing
Standard Library
Platform
Vulnerabilit.Mgmt
Intrusion Detection
File Analysis Compliance Monitoring
Traffic Measure-
ment
Traffic ControlA
pps
Tap
38
text/plain
application/octet-streamtext/html
application/xmlapplication/x-shockwave-flash
image/jpeg
image/pngimage/gifapplication/pdf
Zeek - Understand Your Network (1)
cat files.log | zeek-cut mime_type | sort | uniq -c | sort -rn
Top File Types
39
Chrome
Microsoft-CryptoAPI
Windows-Update-Agent
GoogleUpdate
Safari
FirefoxMSIE CaptiveNetworkSupport
DropboxDesktopClient
ocspd
Zeek - Understand Your Network (2)
cat software.log | zeek-cut host name | sort | uniq | awk -F '\t' '{print $2}' | sort | uniq -c | sort -rn
Top Software by Number of Hosts
40
More Zeek
§ Protocol analyzers– Zeek ships with analyzers for many different protocols
(FTP, HTTP, POP3, IRC, SSL, DNS, SSH, NTP, Portmapper, SMB,...)
§ Zeek comes with >10,000 lines of script code.– Prewritten functionality that’s just loaded.– Amendable to extensive customization and extension.– Growing community writing 3rd party scripts.
41
Anomaly based Detection
Basic idea – detect behavior that differs significantly from normal use:§ Users have certain habits in their system usage:
– Duration of usage, Login times, Amount of file system usage Executed programs, accessed files, ..
§ Assumption: “normal user behavior” can be described statistically– Requires a learning phase / specification of normal behavior– Most approaches require labeled data -> hard to obtain !!
§ Analysis: – compares recorded events with
reference profile of normal behavior § Advantage:
– An attack scenario needs not to be defined a priori– This approach can, in principle, detect unknown attacks
𝑁&
𝑁'
0)
𝑂&𝑂'
x
y
42
Automatic Anomaly Detection – System model
Generic anomaly detection system
[ET04]
Sensorsubsystem
Probe
Probe
Probe
Cen
tral
prep
roce
ssin
g
Modelingsubsystem
Modelderivation
Analysis subsystem
DetectionEvents
with attacks
Events(no attacks)
Model
Reaction?
Network
43
Anomaly based Detection – Challenges - I
§ Defining normal behavior– Including every possible normal behavior is difficult– Boundaries between normal and anomalous behavior is often not
precise– Anomalous observation that lies close to boundary can actually be
normal or vice versa (false positive and false negatives)
§ Adaption of the attacker– Dynamic behavior of anomalous behavior– Attackers often adapt themselves to make anomalous observation
appear normal (false negatives)– Renders task of defining normal behavior more difficult
§ Normal behavior is not static and evolves over time
[Chan09]
44
Anomaly based Detection – Challenges - II
§ Notion of an Anomaly– … different for different application domains– Applying a technique developed in one domain to another, is not
straightforward
§ Availability of labeled data– Obtaining accurate and representative labeled data is expensive
(labeling usually done via human experts)
§ Data Noise– Data often contains noise that tends to be similar to the actual
anomalies and hence is difficult to distinguish and remove
§ Privacy– Collecting user specific usage patterns– Work-related or personal habits
[Chan09]
45
Anomaly based Detection – Methods I
§ Nearest Neighbor-based Anomaly Detection– Assumption: Normal data instances occur in dense neighborhoods, whileanomalies occur far from their closest neighbors– Distance or similarity measure for data instances required– Ex: kNN
[Chan09]
+ Unsupervised+ Adapting to different data type only requires defining appropriate distance measure- Performance relies on distance measure- High FPR depending on nature of data
46
Anomaly based Detection – Methods II
§ Classification-based Anomaly Detection– Training + Testing phases– Ex: Neural Networks, Bayesian Networks, Support Vector Machines
§ More Methods ..
– Clustering-based Anomaly Detection
– Statistical Anomaly Detection
– Information Theoretic Anomaly Detection
– Spectral Anomaly Detection
[Chan09]
+ Fast testing phase due to precompiled model- Rely on availability of accurate labels for various normal classes,which is often not possible
47
Problems of IDS – Audit Data§ Amount of log data
– Significant storage capacities are required– Automated processing of audit data
§ Location of audit data storage– If stored on log server, data must be transferred to this server– If stored on system to be supervised, the log uses significant amounts of
resources of the system
§ Protection of audit data– If a system gets compromised, audit data stored on it might get
compromised either
§ Expressiveness of audit data– Which information is relevant?– Audits often contain a rather low percentage of useful information
48
Problems of IDS – Privacy (data protection)
§ User identifying data elements are logged, e.g.,– Directly identifying elements: user IDs– Indirectly / partly identifying elements: names of directories and
subdirectories (home directory), file names, program names– Minimally identifying elements: host type + time + action, access rights +
time + actionSo that IDS audits may violate the privacy of users
§ Collected information might be abused if not secured properly– Potential solution
• Pseudonymous audit: log activities with user pseudonyms and ensure, that they can only be mapped to user IDs upon incident detection
49
Problems of IDS - Analysis
§ Limited efficiency of analysis
– Most IDS follow a centralist approach for analysis:
“ agents collect audit data and one central evaluation unit analyzes this data”
Þ No (partial) evaluation in agents
Þ Performance bottleneck
– Insufficient efficiency, especially concerning attack variants and attacks
with parallel actions
50
Further Problems of IDS
§ Self-protection – Against insider attacks – Strategies to cope with high load
§ High number of false positives– In practice, many IDS report too many false alarms (some publications
report up to 10.000 per month)– Potential countermeasure: alert correlation (® hierarchical approach)
§ Cooperation between multiple IDS
§ Distributed attacks produces many alerts that actually belong to same attack
– Distributed port scan– Worm spreading– DDoS
IDS
IDS
IDS
IDS
IDSIDS
IDS IDS
51
Summary on IDS
§ Classification of IDS– Host sensors vs. network sensors– Signature-based IDS vs. Policy-based vs. Anomaly-based IDS– Passive vs. active (e.g., honeypots) monitoring
§ Collaborative IDS – Detection of large-scale and coordinated attacks– Centralized, hierarchical, and distributed CIDS– Alert correlation as major challenge
§ Evasion Attacks to bypass IDS– Signature Evasion– Anomaly Evasion, e.g., via covert channel attacks
52
Return to the big picture again…
Detection
ReactionRecovery
Prevention
Intrusion DetectionSystems (IDS)
Firewalls
Honeypots
PDRRProcess
§ “A security resource who's value lies in being probed, attacked or compromised”
§ We want to get compromised!
§ Not a standalone security mechanism
53
Honeypots§ Why?
– FUN!– No false-positives!– Research: Malware analysis/reverse engineering– Reducing available attack surface/early warning system
54
Honeypots – Advantages & Drawbacks
Limited field of viewAble to see/capture only what interacts with themNo capture between all other systems
Small data setsCollection of data only when someone or sth is interacting with them
Reduced FPsAny activity with honeypots is by definition unauthorized
Catching FNsIdentification of new attacks as any activity with the honeypot is an anomaly
Highly FlexibleHoneypots are extremely adaptable to a wide range of environments
Minimal ResourcesA single computer can monitor millions of IP addresses
Risk of being hackedAn attacker can use honeypot to attack other systems
55
Summary
§ IDS and Honeypots as reactive security tools to detect attacks
§ IDS– Signature-based vs. policy-based vs. anomaly-based IDS– In combination with Firewalls: IPS– Classification according to kind of sensors deployed, level of distribution
§ IDS problems– Huge amounts of data to process– Limited accuracy and large number of false positives– Privacy– IDS evasion techniques
§ Honeypots as purely passive monitoring component– Nearly no false positives!
57
Additional References[Chan09] Chandola, Varun, Arindam Banerjee, and Vipin Kumar. 2009. “Anomaly Detection: A Survey.”
ACM Computing Surveys 41 (3) (July 1): 1–58. doi:10.1145/1541880.1541882.
[Pa99] V. Paxson, “Bro: a system for detecting network intruders in real-time,” in USENIX Security Symposium, 1999, vol. 7, pp. 1–22.
[VaKa+15] E. Vasilomanolakis, S. Karuppayah, M. Muehlhaeuser, and M. Fischer, “Taxonomy and Survey of Collaborative Intrusion Detection,” ACM Computing Surveys, pp. 1–35, 2015.
[ET04] ????
Recommended