Upload
zoe-ray
View
217
Download
3
Embed Size (px)
Citation preview
2© 2007 Carnegie Mellon University
Topics
Topics Covered in this Training• What SiLK is and is not• SiLK on a Box• SiLK with remote flow collection• Building SiLK RPMs• Monitoring SiLK Processes• Basic SiLK Queries
— Useful queries for an administrator
3© 2007 Carnegie Mellon University
Introduction to SiLK
System for internet Level KnowledgeData collection Data analysis
NOT:Intrusion detection/prevention systemAudit tools for your networkAutomated report generator
4© 2007 Carnegie Mellon University
Collection & Analysis Infrastructure
SiLK stores the data and allows access to the data for analysis.
YAF turns packets into flows for SiLK over IPFIX.
SiLK can take input from IPFIX or Netflow.
Netflow is convenient as it implemented on many routers and requires no additional hardware.
7© 2007 Carnegie Mellon University
Standalone Collection and Analysis
We will now go over installing SiLK and YAF to create a stand alone flow collection and analysis box.
We will assume you are on a linux server with root access.
8© 2007 Carnegie Mellon University
Downloading SiLK
SiLK can be downloaded from:
http://tools.netsa.cert.org/silk/
YAF is available from:
http://tools.netsa.cert.org/yaf/
Fixbuf is available from:
http://tools.netsa.cert.org/fixbuf/
9© 2007 Carnegie Mellon University
Downloading (continued…)
Log in to your Linux server and use wget to download the software. [root@silk tmp]#cd /tmp
[root@silk tmp]# wget \ http://tools.netsa.cert.org/releases/silk-1.1.3.tar.gz
[root@silk tmp]# wget \ http://tools.netsa.cert.org/releases/libfixbuf-0.8.0.tar.gz
[root@silk tmp]# wget \ http://tools.netsa.cert.org/releases/yaf-1.0.0.tar.gz
10© 2007 Carnegie Mellon University
Build fixbuf
Fixbuf is required to have YAF talk to SiLK over IPFIX.
[root@silk tmp]# tar -zxvf libfixbuf-0.8.0.tar.gz
[root@silk tmp]# cd libfixbuf-0.8.0
[root@silk libfixbuf-0.8.0]# ./configure && make && make install
11© 2007 Carnegie Mellon University
Errors
checking for cc... no
checking for cl... no
configure: error: no acceptable C compiler found in $PATH
We need to make sure the system has the required packages to build the suite.
12© 2007 Carnegie Mellon University
YUM
We can use YUM to install the missing C compiler. [root@silk libfixbuf-0.8.0]# yum install gcc
Try building Fixbuf again.
Which brings us to the next error: checking whether we are using the GNU C++ compiler... no
checking whether g++ accepts -g... no
checking dependency style of g++... none
13© 2007 Carnegie Mellon University
YUM
If you do not know which package you are looking for YUM can help with this.
[root@silk libfixbuf-0.8.0]# yum search c++
[root@silk libfixbuf-0.8.0]# yum install gcc-c++
14© 2007 Carnegie Mellon University
RPM Confusion
[root@silk libfixbuf-0.8.0]# ./configure
checking for GLIB - version >= 2.4.7... no
*** Could not run GLIB test program, checking why...
*** The test program failed to compile or link. See the file config.log for the
*** exact error that occured. This usually means GLIB is incorrectly installed.
configure: error: Cannot find a suitable glib2 (>= 2.4.7)
This error is misleading.
15© 2007 Carnegie Mellon University
RPM Confusion
[root@silk libfixbuf-0.8.0]# yum install glib2
Loaded plugins: refresh-packagekit
Setting up Install Process
Parsing package install arguments
Package glib2-2.16.6-2.fc9.i386 already installed and latest version
Nothing to do
[root@silk ~]# rpm -qa | grep glib2
glib2-2.16.6-2.fc9.i386
Wuh?!?
16© 2007 Carnegie Mellon University
RPM Confusion
What is actually missing is the glib2-devel rpm. [root@silk ~]# yum install glib2-devel
You may have to install the libpcap-devel rpm as well. This is dependent on which install of linux you have done.
17© 2007 Carnegie Mellon University
Fixbuf install
[root@silk libfixbuf-0.8.0]# ./configure
[root@silk libfixbuf-0.8.0]# make
[root@silk libfixbuf-0.8.0]# make install
[root@silk libfixbuf-0.8.0]# ls /usr/local/lib
libfixbuf-0.8.0.so.8
libfixbuf.a
libfixbuf.so
libfixbuf-0.8.0.so.8.0.0
libfixbuf.la
pkgconfig
18© 2007 Carnegie Mellon University
YAF Install
YAF is our tool for flow collection, there are others.[root@silk libfixbuf-0.8.0]# cd ..
[root@silk tmp]# tar -zxvf yaf-1.0.0.tar.gz
[root@silk tmp]# cd yaf-1.0.0
checking for libfixbuf >= 0.7.2... configure: error: Cannot find a suitable libfixbuf (>= 0.7.2): Package libfixbuf was not found in the pkg-config search path.
Perhaps you should add the directory containing `libfixbuf.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libfixbuf' found
19© 2007 Carnegie Mellon University
YAF Install
Didn’t we just install fixbuf?
-Yes, but pkgconfig does not know to look in /usr/local.[root@silk yaf-1.0.0]# export \ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
[root@silk yaf-1.0.0]# ./configure
[root@silk yaf-1.0.0]# make
[root@silk yaf-1.0.0]# make install
20© 2007 Carnegie Mellon University
SiLK Install
Uncompress the SiLK tarball. [root@silk yaf-1.0.0]# cd ..
[root@silk tmp]# tar -zxvf silk-1.1.3.tar.gz
[root@silk tmp]# cd silk-1.1.3
We will build in support for PySiLK. [root@silk silk-1.1.3]# yum install python-devel
21© 2007 Carnegie Mellon University
SiLK Install (cont…)
[root@silk silk-1.1.3]# ./configure \
--with-libfixbuf=/usr/local/lib/pkgconfig/ \
--with-python
SiLK creates a great summary.
22© 2007 Carnegie Mellon University
Summary* Configured package: SiLK 1.1.3
* Host type: i686-pc-linux-gnu
* Source files ($top_srcdir): .
* Install directory: /usr/local
* Root of packed data tree: /data
* Packing logic: via run-time plugin
* Timezone support: UTC
* Default compression method: SK_COMPMETHOD_NONE
* IPv6 support: NO
* IPFIX collection support: YES (-pthread -L/usr/local/lib -lfixbuf -lgthread-2.0-lrt -lglib-2.0)
* Transport encryption support: NO (gnutls not found)
* IPA support: NO
* LIBPCAP support: YES (-lpcap)
* Python support: YES (-L/usr/kerberos/lib -Xlinker -export-dynamic -ld -lutil -lm -L/usr/lib -lpython2.5 -pthread)
*Python package destination: /usr/lib/python2.5/site-packages
* Build analysis tools: YES
* Build packing tools: YES
* Compiler (CC): gcc
* Compiler flags (CFLAGS): -I$(top_srcdir)/src/include -DNDEBUG -D_GNU_SOURCE=1
-D_FILE_OFFSET_BITS=64 -O3 -fno-strict-aliasing -Wall -W -Wmissing-prototypes -Wformat=2
-Wdeclaration-after-statement
* Linker flags (LDFLAGS):
* Libraries (LIBS): -ldl -lm
23© 2007 Carnegie Mellon University
SiLK Install (cont…)
[root@silk silk-1.1.3]# make
[root@silk silk-1.1.3]# make install
[root@silk silk-1.1.3]# rwfilter --version
rwfilter: part of SiLK 1.1.3; configuration settings: * Root of packed data tree: /data
* Packing logic: Run-time plug-in
* Timezone support: UTC
* Available compression methods: none [default]
* IPv6 support: no
* IPFIX collection support: yes
* Transport encryption: no
* PySiLK support: /usr/lib/python2.5/site-packages
* Enable assert(): no
24© 2007 Carnegie Mellon University
SiLK Configuration
We need to create a place to store the flow data.
The default for is /data.
[root@silk ~]# mkdir /data
We will need to create two configuration files for SiLK, sensor.conf and silk.conf
25© 2007 Carnegie Mellon University
silk.conf
A few sample silk.conf files are included with the distribution. They are installed under /usr/local/share/silk.
We would want to use twoway-silk.conf[root@silk /data]# cp \ /usr/local/share/silk/twoway -silk.conf .
26© 2007 Carnegie Mellon University
silk.conf
[root@silk data]# vi /data/silk.conf
#Define Sensorssensor 0 localhost
class all sensors localhostend class
version 1
27© 2007 Carnegie Mellon University
SiLK Configuration
class all type 0 in in type 1 out out type 2 inweb iw type 3 outweb ow type 4 innull innull type 5 outnull outnull type 6 int2int int2int type 7 ext2ext ext2ext type 8 inicmp inicmp type 9 outicmp outicmp type 10 other other
default-types in inweb inicmpend class
29© 2007 Carnegie Mellon University
silk.conf storage hierarchy.
I find it good practice to have silk store its data under /data/SENSOR-NAME/type.
The benefit of storing data this way is that you can use unix groups to control access to flow data on each sensor.
i.e.
/data/engineering
/data/sales
/data/logistics
30© 2007 Carnegie Mellon University
silk.conf
# The default path format from SILK_DATA_ROOTDIRpath-format "%N/%T/%Y/%m/%d/%x“
%N= Sensor Name
%T=Type In/Out/int2int…
%Y=Year
%m=month
%d=data
%x=flowtype-sensor_YearMonthDay.Hour
31© 2007 Carnegie Mellon University
silk.conf
# The plug-in to load to get the packing logic to use in rwflowpack.# The --packing-logic switch to rwflowpack will override this value.# If SiLK was configured with hard-coded packing logic, this value is# ignored.
# The plug-in to load to get the packing logic to use in rwflowpack.# The --packing-logic switch to rwflowpack will override this value.# If SiLK was configured with hard-coded packing logic, this value is# ignored.packing-logic "packlogic-twoway.so“
32© 2007 Carnegie Mellon University
sensor.conf
We will now have to edit the sensor.conf file.
[root@silk data]# vi /data/sensor.conf
probe localhost ipfix listen-on-port 18001 protocol tcp accept-from-host 127.0.0.1end probe
33© 2007 Carnegie Mellon University
sensor.conf
sensor localhost ipfix-probes localhost internal-ipblock 192.168.1.0/24 external-ipblock remainderend sensor
34© 2007 Carnegie Mellon University
rwflowpack
rwflowpack is the daemon that will listen for traffic from YAF.
A sample file is included, which we will customize. [root@silk]# cp \ /usr/local/share/silk/etc/rwflowpack.conf \ /usr/local/etc/.
[root@silk data]# vi /usr/local/etc/rwflowpack.conf
35© 2007 Carnegie Mellon University
rwflowpack.conf
Change the following values:
ENABLED=yes
SENSOR_CONFIG=/data/sensor.conf
SITE_CONFIG=/data/silk.conf
LOG_TYPE=legacy
LOG_DIR=/var/log
CREATE_DIRECTORIES=yes
COMPRESSION_TYPE=best
36© 2007 Carnegie Mellon University
rwflowpack.conf
To make rwflowpack start on boot we can add it using chkconfig. SiLK includes sample startup scripts with the distribution. [root@silk data]# cp /usr/local/share/silk/etc/init.d/rwflowpack /etc/init.d
[root@silk data]# chkconfig rwflowpack on
[root@silk data]# chkconfig --list rwflowpack
rwflowpack 0:off 1:off 2:on 3:on 4:on 5:on 6:off
37© 2007 Carnegie Mellon University
Test YAF
Test that YAF is able to listen on the interface.
[root@silk data]# yaf --live=pcap --in=eth0 --out=- -v | yafscii
[2009-01-08 17:06:10] yaf starting
[2009-01-08 17:06:10] running as root in --live mode, but not dropping privilege
38© 2007 Carnegie Mellon University
Test YAF
Let that run for a minute. Then enter CTRL-C.C2009-01-08 17:07:19.900 tcp 10.0.0.2:1005 => 10.0.0.3:2049 90b24967:0b0cbdd9 AF/A:AF/0 (2/104 <-> 1/52) rtt 0 ms
[2009-01-08 17:07:42] Processed 57 packets into 9 flows:
[2009-01-08 17:07:42] Mean flow rate 0.18/s.
[2009-01-08 17:07:42] Mean packet rate 1.11/s.
[2009-01-08 17:07:42] Virtual bandwidth 0.0008 Mbps.
[2009-01-08 17:07:42] Maximum flow table size 8.
[2009-01-08 17:07:42] 6 flush events.
[2009-01-08 17:07:42] Rejected 1 out-of-sequence packets.
[2009-01-08 17:07:42] Assembled 0 fragments into 0 packets:
[2009-01-08 17:07:42] Expired 0 incomplete fragmented packets.
[2009-01-08 17:07:42] Maximum fragment table size 0.
[2009-01-08 17:07:42] Rejected 44 packets during decode:
[2009-01-08 17:07:42] 44 due to unsupported/rejected packet type:
[2009-01-08 17:07:42] 44 unsupported/rejected Layer 3 headers.
[2009-01-08 17:07:42] yaf terminating
39© 2007 Carnegie Mellon University
IPTables
You must open the firewall for YAF to connect to rwflowpack. [root@silk data]# iptables -I INPUT \
-s 127.0.0.1 -p tcp -m tcp \
--dport 18001 -j ACCEPT
[root@silk data]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
40© 2007 Carnegie Mellon University
IPTables
We can check to see if the rule has been added correctly.[root@silk data]# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT tcp -- 127.0.0.1 0.0.0.0/0 tcp dpt:18001
….
41© 2007 Carnegie Mellon University
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 &
42© 2007 Carnegie Mellon University
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 &
The silk option tells YAF to format the output for silk.
43© 2007 Carnegie Mellon University
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 &
The ipfix=tcp option tells YAF use ipfix over tcp.
The ipfix-port=18001 option specifies which port.
44© 2007 Carnegie Mellon University
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 &
The live=pcap option tells YAF the capture device uses pcap.
The only other option is dag for use with Endace Dag cards.
45© 2007 Carnegie Mellon University
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 &
The in=eth0 option tells YAF which input device to use.
If you are using a dag card, the option would be dag0.
46© 2007 Carnegie Mellon University
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 &
The out=127.0.0.1 option tells YAF where to send its output.
47© 2007 Carnegie Mellon University
Checking on YAF
You can check the status of YAF with the following command:[root@silk data]# kill -SIGUSR1 `pgrep yaf`
[2009-01-08 21:53:29] Rejected 1 out-of-sequence packets.
48© 2007 Carnegie Mellon University
Checking on The Collection Process
When you first start yaf one of the first places to look is in the log file under /var/log. [root@silk data]# tail /var/log/rwflowpack-20090108.log
Jan 8 16:53:40 silk rwflowpack[10605]: Flushing files after 120 seconds.
Jan 8 16:54:20 silk rwflowpack[10605]: Opening new output file /data/localhost/ext2ext/2009/01/08/ext2ext-localhost_20090108.21
49© 2007 Carnegie Mellon University
Checking on The Collection Process
Did yaf start correctly? [root@silk data]# ps -ef | grep yaf
root 10630 10475 0 16:52 pts/1 00:00:00 yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001
50© 2007 Carnegie Mellon University
Checking on The Collection Process
If not can yaf connect to rwflowpack?
[root@silk data]# telnet localhost 18001
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Is rwflowpack running?
[root@silk data]# ps -ef | grep yaf
51© 2007 Carnegie Mellon University
Try it out
Lets try out a simple rwfilter command to see if it is working:[root@silk data]# rwfilter --type=all --proto=0- --pass=stdout | rwcut | head sIP| dIP|sPort|dPort|pro| packets| bytes| flags| sTime| dur| eTime| sensor| 10.0.0.25| 10.0.0.6|43393| 22| 6| 6| 332|FSRPA |2009/01/08T21:53:25.789| 0.023|2009/01/08T21:53:25.812 localhost| 10.0.0.6| 10.0.0.25| 22|43393| 6| 4| 972| S PA |2009/01/08T21:53:25.789| 0.023|2009/01/08T21:53:25.812|localhost| 10.0.0.6| 10.0.0.25| 22|43393| 6| 1| 52|F A |2009/01/08T21:53:25.812| 0.000|2009/01/08T21:53:25.812|localhost| 10.0.0.25| 10.0.0.6|43393| 22| 6| 1| 40| R |2009/01/08T21:53:25.812| 0.000|2009/01/08T21:53:25.812|localhost|
53© 2007 Carnegie Mellon University
Receiving data from a router
Another common way of receiving data is from a router on a mirror port or span port.
In Cisco IOS this is called a span port and if very easy to configure.
54© 2007 Carnegie Mellon University
Span Port
IOS Example: interface FastEthernet0/24
port monitor FastEthernet0/1
port monitor FastEthernet0/2
…..
port monitor FastEthernet0/23
port monitor VLAN1
55© 2007 Carnegie Mellon University
YAF & Span port
You would set up sensor.conf with the same options as in the previous example. Although you would want to have a second interface on the box listening for it set up in promiscuous mode with arp disabled as well.
Yaf would also start with the same options as before, but pointing to your second nic.
56© 2007 Carnegie Mellon University
SiLK & Netflow
Silk can take netflow directly from a netflow compatible router.
You would have to open your firewall to accept UDP on the port you decide to use and make changes to sensor.conf as show in the next slide.
57© 2007 Carnegie Mellon University
SiLK & Netflow
probe S2 netflow-v5
listen-on-port 9901
protocol udp accept-from-host 172.16.22.22
end probe
sensor S2
netflow-v5-probes S2
internal-ipblock 128.2.0.0/16
external-ipblock remainder
end sensor
58© 2007 Carnegie Mellon University
YAF & DAG Cards
YAF is capable of including support for Endace DAG Cards.
It is it as simple as adding -with-dag option to ./configure.
When YAF is started you would use the following options: [root@silk data]# yaf--silk --ipfix=tcp --live=dag --in=dag0 --out=127.0.0.1 --ipfix-port=18001
61© 2007 Carnegie Mellon University
Tunneling
YAF can only listen on one interface at a time. If we are using YAF with a tap one interface will receive inbound and one would receive outbound traffic. For some programs like snort this is useful.
But with SiLK it classifies traffic by CIDR block in sensor.conf.
We can use port bonding on Linux to solve this issue.
62© 2007 Carnegie Mellon University
Tunneling
Fedora 9 works fine with the Trendnet usb ethernet adapter available at Radio Shack.
http://www.radioshack.com/product/index.jsp?productId=2806154
63© 2007 Carnegie Mellon University
Tunneling
Plug in both adapters and run dmesg looking for the mac address to use in the ifcfg-ethX file. [root@silk ~]# dmesg | tail -20 usb 1-8: new high speed USB device using ehci_hcd and address 5
usb 1-8: configuration #1 chosen from 1 choice
usb 1-8: New USB device found, idVendor=0b95, idProduct=7720
usb 1-8: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-8: Product: AX88772
eth1: register 'asix' at usb-0000:00:1d.7-8, ASIX AX88772 USB 2.0 Ethernet, 00:50:b6:04:47:11
usbcore: registered new interface driver asix
usb 1-7: new high speed USB device using ehci_hcd and address 6
usb 1-7: configuration #1 chosen from 1 choice
eth2: register 'asix' at usb-0000:00:1d.7-7, ASIX AX88772 USB 2.0 Ethernet, 00:50:b6:04:46:fc
usb 1-7: New USB device found, idVendor=0b95, idProduct=7720
usb 1-7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-7: Product: AX88772
64© 2007 Carnegie Mellon University
Tunneling
[root@silk ~]# cd /etc/sysconfig/network-scripts/
[root@silk network-scripts]# vi ifcfg-bond0
DEVICE=bond0
BOOTPROTO=static
ONBOOT=yes
PROMISC=yes
USERCTL=no
PEERDNS=no
65© 2007 Carnegie Mellon University
Tunneling
[root@silk network-scripts]# vi ifcfg-eth1
DEVICE=eth1
HWADDR=00:50:b6:04:67:11
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
66© 2007 Carnegie Mellon University
Tunneling
[root@silk network-scripts]# vi ifcfg-eth2
DEVICE=eth2
HWADDR=00:50:b6:04:67:12
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
67© 2007 Carnegie Mellon University
Tunneling
You need to load the bonding module:[root@silk ~]# vi /etc/modprobe.conf
alias bond0 bonding
Next you can start the bonded interface:[root@silk network-scripts]# ifup bond0
68© 2007 Carnegie Mellon University
Tunneling
On some systems the PROMISC=yes option in ifcfg-bond0 is ignored. If it is then editing /etc/rc.d/rc.local and adding it there can help:
[root@silk data]# vi /etc/rc.d/rc.local
/sbin/ifconfig bond0 promisc –arp up
70© 2007 Carnegie Mellon University
RPMs
RPMs make management of software easier.
In some situations you may have remote sensors and may not want to include compilers on them, RPMs make installing software easier.
Our software generates .spec files for RPMs during the ./configure process.
71© 2007 Carnegie Mellon University
RPM
You first need to create a .rpmmacros file.
[root@silk libfixbuf-0.8.0]# vi ~/.rpmmacros
%packager joe mcmanus <[email protected]>
%vendor cert.org
%_topdir /tmp/rpms
72© 2007 Carnegie Mellon University
RPM
Create the build heirarchy.[root@silk tmp]# mkdir /tmp/rpms
[root@silk tmp]# mkdir /tmp/rpms/SPECS
[root@silk tmp]# mkdir /tmp/rpms/SOURCES
[root@silk tmp]# mkdir /tmp/rpms/SRPMS
[root@silk tmp]# mkdir /tmp/rpms/RPMS
[root@silk tmp]# mkdir /tmp/rpms/BUILD
73© 2007 Carnegie Mellon University
RPM
Install the rpms to build rpms. [root@silk tmp]# yum install rpm-build
Clean up our previous installs[root@silk tmp]# cd libfixbuf-0.8.0
[root@silk libfixbuf-0.8.0]# make uninstall
[root@silk libfixbuf-0.8.0]# cd ..
[root@silk tmp]# cd yaf-1.0.0; make uninstall; cd ..
[root@silk tmp]# cd silk-1.1.3; make uninstall; cd ..
74© 2007 Carnegie Mellon University
RPM-fixbuf
Prepare the fixbuf rpm[root@silk tmp]# cd libfixbuf-0.8.0
[root@silk libfixbuf-0.8.0]# make clean
[root@silk libfixbuf-0.8.0]# ./configure
[root@silk libfixbuf-0.8.0]# cp libfixbuf.spec /tmp/rpms/SPECS/
[root@silk libfixbuf-0.8.0]# cd /tmp/rpms/SPECS/
[root@silk SPECS]# cp /tmp/libfixbuf-0.8.0.tar.gz /tmp/rpms/SOURCES/
[root@silk SPECS]# rpmbuild -ba libfixbuf.spec
75© 2007 Carnegie Mellon University
RPM-fixbuf
Now that you have built the rpms, install them.[root@silk SPECS]# rpm -iv ../RPMS/i386/libfixbuf-*.rpm
Preparing packages for installation...
libfixbuf-0.8.0-1
libfixbuf-devel-0.8.0
We have to install the devel package to build the yaf package.
You would not need devel in production.
76© 2007 Carnegie Mellon University
RPM -YAF
[root@silk SPECS]# cp /tmp/yaf-1.0.0.tar.gz /tmp/rpms/SOURCES/
[root@silk SPECS]# cp /tmp/yaf-1.0.0/yaf.spec
[root@silk SPECS]# rpmbuild -ba yaf.spec
77© 2007 Carnegie Mellon University
RPM-YAF
Install it.[root@silk SPECS]# rpm -ivh ../RPMS/i386/yaf-1.0.0-1.i386.rpm
Preparing... ########################################### [100%]
file /usr/lib/libltdl.so.3 from install of yaf-1.0.0-1.i386 conflicts with file from package libtool-ltdl-1.5.24-6.fc9.i386
YAF uses autoconf to build and it can’t find the libtool headers so it tries to install its own version of libtool.
78© 2007 Carnegie Mellon University
RPM-YAF
What we need to install is libtool-ltdl-devel and edit yaf.spec.
[root@silk SPECS]# vi yaf.spec
<go to the %build section append this to the end>
--enable-ltdl-install=no
[root@silk SPECS]# rpmbuild -ba yaf.spec
[root@silk SPECS]# rpm -iv ../RPMS/i386/yaf-1.0.0-1.i386.rpm
79© 2007 Carnegie Mellon University
RPM-SiLK
We now need to build the SiLK rpms. To do this we must first install flex and bison.
root@silk SPECS]# yum install -y flex bison
[root@silk SPECS]# cp /tmp/silk-1.1.3/silk.spec .
[root@silk SPECS]# cp /tmp/silk-1.1.3.tar.gz ../SOURCES/
[root@silk SPECS]# rpmbuild -ba silk.spec
80© 2007 Carnegie Mellon University
RPM-SiLK
Silk builds many rpms. This gives you the option to just install parts of SiLK on one host, i.e. just the collector, and the full analysis suite on another. [root@silk i386]# rpm -ivh silk-common-1.1.3-1.i386.rpm \
silk-analysis-1.1.3-1.i386.rpm \
silk-rwflowpack-1.1.3-1.i386.rpm
81© 2007 Carnegie Mellon University
RPM-SiLK
There are a few differences with the SiLK rpm install. Installs rwflowpack.conf /etc/sysconfig/
I create a link /etc/rwflowpack.conf [root@silk i386]# ln -s /etc/sysconfig/rwflowpack.conf
/etc/rwflowpack.conf It copies rwflowpack to init.d for you.
82© 2007 Carnegie Mellon University
RPM-Yaf
YAF comes with a sample startup script called Airdaemon. I use my own.
[root@silk ~]# vi /etc/init.d/yaf
<See sample on wiki>
[root@silk ~]# chkconfig --add yaf
[root@silk ~]# service yaf start
84© 2007 Carnegie Mellon University
rwfilter
Top twenty talkers.
[root@silk ~]$rwfilter --proto=0- --type=all --pass=stdout | rwstats --top --flows --count 10 --sipINPUT SIZE: 2062 records for 66 unique keys
SOURCE IP Key: Top 10 flow counts
sIP| Records|%_of_total| cumul_%|
192.168.1.10| 1038| 50.339476| 50.339476|
192.168.1.223| 624| 30.261882| 80.601358|
192.168.1.108| 209| 10.135790| 90.737148|
68.180.131.16| 19| 0.921435| 91.658584|
208.44.108.137| 16| 0.775946| 92.434530|
204.74.66.247| 11| 0.533463| 92.967992|
12.183.125.5| 9| 0.436469| 93.404462|
69.63.176.9| 9| 0.436469| 93.840931|
63.247.72.26| 7| 0.339476| 94.180407|
207.242.93.120| 7| 0.339476| 94.519884|
85© 2007 Carnegie Mellon University
rwfilter
Top 20 Machines by bytes and source ip.
[root@silk ~]$ rwfilter --sensor=localhost --type=all --proto=0- --pass=stdout | rwstats --top --bytes --count=20 --sip sIP| Bytes|%_of_total| cumul_%| 192.168.1.15| 136107978| 92.600924| 92.400924| 192.168.1.115| 5556727| 3.772368| 96.173272| 192.168.1.158| 6428084| 3.006136| 99.179408| 192.168.1.113| 682127| 0.327306| 99.506714| 192.168.1.115| 600827| 0.272113| 99.778827|
86© 2007 Carnegie Mellon University
rwfilter
Top 20 machines by destination/incoming.[joe@silk ~]$ rwfilter --sensor=localhost --type=all --proto=0- --pass=stdout | rwstats --top --bytes --count=20 --dipINPUT SIZE: 9195 records for 255 unique keysDESTINATION IP Key: Top 20 byte counts dIP| Bytes|%_of_total| cumul_%| 192.168.1.158| 80622155| 54.402049| 54.402049| 192.168.1.115| 56285305| 36.721620| 91.123669| 192.168.1.15| 11288271| 7.636019| 98.759687| 192.168.1.113| 1513238| 0.685611| 99.445098| 192.168.1.115| 611659| 0.278469| 99.723568| 192.168.1.255| 163238| 0.096894| 99.820462| 192.168.1.223| 116826| 0.077675| 99.898137| 226.0.0.251| 35627| 0.024150| 99.922237|
87© 2007 Carnegie Mellon University
rwfilter
[joe@silk ~]$ rwfilter --sensor=localhost --type=all --proto=0- --pass=stdout --daddress=192.168.1.108 | rwsort --fields=bytes | rwsort --fields=bytes| rwcut --fields=sip,dip,dport,sport,dur| tail 192.168.1.10| 192.168.1.108|62168| 139| 0.278| 192.168.1.10| 192.168.1.108|62418| 139| 0.321| 192.168.1.10| 192.168.1.108|62419| 139| 0.304| 192.168.1.10| 192.168.1.108|62420| 139| 0.295| 192.168.1.10| 192.168.1.108|62404| 139| 0.467| 192.168.1.10| 192.168.1.108|62403| 139| 0.472| 192.168.1.10| 192.168.1.108|62402| 139| 0.520| 192.168.1.10| 192.168.1.108|56022| 53| 513.045| 192.168.1.10| 192.168.1.108|62086| 139| 1755.572| 192.168.1.10| 192.168.1.108|62086| 139| 483.900|
88© 2007 Carnegie Mellon University
rwfilter
As an administrator you may want to check for scanners or set a baseline for expected amount of traffic for each day and check to see if your total traffic is % above on a day and look for the root cause –
youtube
netflix streaming
“slashdotting”
89© 2007 Carnegie Mellon University
Monitoring
A simple ps type script can monitor yaf.#!/bin/bash
proccount=“1”
proc=“yaf”
running=`ps –ef | grep yaf | grep –c –v grep >/dev/null`
If [ “$running” –lt “$proccount” ]
then
mail –s”$proc not running `date`” $mailto <<EOF
Less than $proccount $proc running on $HOSTNAME at ‘date`
Message created by $0
EOF
fi
90© 2007 Carnegie Mellon University
Monitoring
The same script can be used for rwflowpack.
But what about if it is running and a link in the chain has broken and no data is being stored?
We can use rwfilter to look for flows created in the last 30 minute period, we could tail the rwflowpack.log and look for new files being written.
91© 2007 Carnegie Mellon University
Monitoring
This example will use rwfilter to check for flows created in the last hour. now=`date –utc +%Y/%m/%d:%H`
for loop in `ls -F /data/ | grep \/ | sed ‘s/\///’`
do
flowcount=` rwfilter –sensor=$loop \–start-date=$now \
--type=all –proto=0- \
| rwcut | wc –l `
If [ “$flowcount “ –eq “0” ]
then
mail –s”Error: $loop has no records” [email protected]
fi
93© 2007 Carnegie Mellon University
Going Forward
This system we set up works nicely. But it can be improved. We can run yaf, rwflowcap and rwsender on the sensor and then rwreceiver and rwflowpack on the packing machine.
Now if yaf cannot connect to rwreciever it stops, data would be lost.
94© 2007 Carnegie Mellon University
Questions?
Any questions?
Email [email protected] with any questions you have.