FreeBSD and DriversGili Yankovitch, Nyx Software Security Solutions

Key Points● What is FreeBSD?● FreeBSD Drivers: How to create, compile and run a driver.● Char devices● Network Hooking

○ L3○ L2

● Interaction with the network stack

What is FreeBSD?● “FreeBSD is a free UNIX-like operating system descended from Research

Unix via the Berkeley Software Distribution (BSD).” - Wikipedia● The BSD Project was founded in 1976 by Bill Joy.● Contained code written by AT&T (Who later sued people related to BSD)● In 1993 the first FreeBSD distribution was released.

○ Two years AFTER the Linux Kernel was founded.

What is FreeBSD?● https://www.freebsd.org/● Unlike Linux, it comes with a lot of user mode tools● Doesn’t come in many flavours (distributions)● Supported architectures: amd64, i386, ia64, powerpc, powerpc64, sparc64,

mips, armv6, aarch64.● Unfortunately, lacks a lot of features implemented in Linux.

○ Namespaces, Good L2 hooking (yay -_-) and more…

● It is very unfortunate but there is very little documentation onFreeBSD on the internet. :(

○ This means that if you are stuck, you need to deal with it on your own.■ True story.

How to get FreeBSD● Download

○ https://www.freebsd.org/where.html

● Git:○ https://github.com/freebsd

■ Yes they use GitHub.

● On a FreeBSD system, the sources are usually at:○ /usr/src/sys/

Folder Structure● In contrast to Linux, FreeBSD has a lot of folders in its root directory.

○ kern/ - Core kernel implementation.○ libkern/ - Core kernel libraries (printf, uprintf, strcpy etc…).○ fs/ - File systems implementation.○ net/ netinet/ - Core net and Inet implementation.○ sys/ - Include directory. Contains a lot of *.h files.○ amd64/ arm/ mips/ … - Architecture specific sources.○ modules/ dev/ - Drivers.○ ...

First And Foremost - Prints!● There are two types of prints from the kernel:

○ printf○ uprintf

● Both appear in dmesg● uprintf prints to your current console● printf prints to tty0

Char Devices

Compiling Our First Driverstatic int nethook_loader (struct module *m, int what, void *arg){ int err = 0; switch (what) { case MOD_LOAD: /* kldload */ uprintf ("Nethook KLD loaded. \n"); break; case MOD_UNLOAD : uprintf ("Nethook KLD unloaded. \n"); break; } return err;}static moduledata_t nethook_mod ={ "nethook", nethook_loader , NULL};DECLARE_MODULE (nethook, nethook_mod , SI_SUB_KLD , SI_ORDER_ANY );

nethook.c

Don’t return anything different than 0!Different value will prevent you from unloading the module!!

Compiling Our First Driver

● Yep. That simple.

SRCS=nethook.cKMOD=nethook

.include <bsd.kmod.mk>

Makefile

Running the Driver● Just like Linux, we need to inject it to the Kernel:

$ kldload ./nethook.ko

$ kldunload ./nethook.ko

$ kldstat

● Removing from Kernel:

● Modules list:

Creating A Char Device● This actually has a very good tutorial:

○ https://www.freebsd.org/doc/en/books/arch-handbook/driverbasics-char.html

● But here’s the snippets anyhow:○ Create a struct with function pointers to read, write, open, close.

■ ioctl seems to fail with this method...

/* Character device entry points */static struct cdevsw echo_cdevsw = {

.d_version = D_VERSION ,

.d_open = echo_open ,

.d_close = echo_close ,

.d_read = echo_read ,

.d_write = echo_write ,

.d_name = "echo",};

● Just like Linux...

Creating A Char Device● Then all you need to do is register:

static struct cdev *echo_dev;

static int nethook_loader (struct module *m, int what, void *arg){...error = make_dev_p (MAKEDEV_CHECKNAME | MAKEDEV_WAITOK ,

&echo_dev, &echo_cdevsw , 0, UID_ROOT , GID_WHEEL , 0600, "echo");

…}

Char Device Kernel ObjectPointer to File Operations

Owner UID of File System NodeOwner GID of File System NodeFile System Node PermissionsName of File System Node

Creating A Char Device● Read operation● Note the struct uio:

○ uio_resid - Space left in buffer sent from user mode (read length usually)○ uio_offset - Current write offset

● uio knows whether it’s a read or a write depending on current action.○ Here uiomove() writes from kernel buffer to user mode buffer.

static int echo_read (struct cdev *dev __unused , struct uio *uio, int ioflag __unused ){

size_t amt;int error;

amt = MIN(uio->uio_resid, uio->uio_offset >= echomsg->len + 1 ? 0 : echomsg ->len + 1 - uio->uio_offset );

if ((error = uiomove(echomsg->msg, amt, uio)) != 0)uprintf("uiomove failed! \n");

return (error);}

Creating A Char Device● Corresponding write().

static int echo_write (struct cdev *dev __unused , struct uio *uio, int ioflag __unused ){

size_t amt;int error;

if (uio->uio_offset != 0 && (uio->uio_offset != echomsg->len))return (EINVAL);

/* Copy the string in from user memory to kernel memory */amt = MIN(uio->uio_resid, (BUFFERSIZE - echomsg->len));

error = uiomove(echomsg->msg + uio->uio_offset , amt, uio);

if (error != 0)uprintf("Write failed: bad address! \n");

return (error);}

Networking● DISCLAIMER:

○ Before you begin to build your own network driver, be absolutely sure you understand the below.

● OK lets continue...

Network Stack

Networking● Just like Linux has its skb structure, FreeBSD has a basic buffer system● It’s called: mbuf● mbufs are buffer chains of size 256

○ Larger buffers are possible in an mbuf cluster but unfortunately usually it’s not the case.

● When you get a packet larger than 256 bytes, you get an mbuf chain● Mellanox created a module called OFED to help port drivers from Linux to

Freebsd.○ It’s a great place to start learning about networking in FreeBSD.○ Unfortunately it lacks a HELL LOT of functionality sometimes needed.

Meet struct mbuf

● Yeah I know it’s weird and complicated.○ Our interest is in m_hdr and in m_dat.M_databuf (Which means a normal packet)

struct mbuf { struct m_hdr m_hdr ; union { struct { struct pkthdr MH_pkthdr ; /* M_PKTHDR set */ union { struct m_ext MH_ext ; /* M_EXT set */ char MH_databuf [MHLEN]; } MH_dat; } MH; char M_databuf [MLEN]; /* !M_PKTHDR, !M_EXT */ } M_dat;};

/sys/mbuf.h

Meet struct m_hdr

● mh_next - Already mentioned this is an mbuf chain● mh_nextpkt - mbufs provide us with a linked-list of packets storage place.● mh_data - Pointer to beginning of data within the data buffer● mh_len - Length of data in this mbuf

struct m_hdr { struct mbuf *mh_next; /* next buffer in chain */ struct mbuf *mh_nextpkt ; /* next chain in queue/record */ caddr_t mh_data ; /* location of data */ int32_t mh_len ; /* amount of data in this mbuf */ uint32_t mh_type :8, /* type of data in this mbuf */ mh_flags:24; /* flags; see below */#if !defined(__LP64__) uint32_t mh_pad ; /* pad for 64bit alignment */#endif};

/sys/mbuf.h

nbuf Structure

mbuf

mh_data

mbuf aaaaa mh_next

mh_len

mbuf aaaaa

mbuf aaaaa

mh_

next

pkt

Whatever you do, do NOT access these directly!● Seriously. For everything you need there’s a function.● When in doubt, see man mbuf (9).● mbuf function names are non indicative, so I’ll explain a few here:

Allocating and freeing buffers● m_get(int how, int type) - Allocates a new mbuf and sets its type.● m_free(struct mbuf *m) - Frees a single mbuf.● m_freem(struct mbuf *m) - Frees an entire mbuf chain.● m_dup(struct mbuf *m, int how) - Duplicates an entire mbuf.● m_copym(struct mbuf *mbuf, int offset, int len, int how) - Copy only a portion

of the mbuf to a new mbuf chain.● m_copydata(const struct mbuf *mbuf, int offset, int len, caddr_t buf) - Copy

the mbuf data to a different buffer.● m_length(struct mbuf *m, struct mbuf ** last) - Returns the entire mbuf chain

length (in bytes).

Shorten or Lengthen the Buffer● m_adj(struct mbuf *m, int len) - Shorten the buffer from the beginning.

mbuf aaaaa mbuf

mh_data

mh_nextmbuf

mh_len mh_lenmh_len

mbuf

mh_data

void shorten_my_mbuf (struct mbuf *m){ m_adj(m);}

m is still pointing to the first mbuf!!

Shorten or Lengthen the Buffer● m_prepend(struct mbuf *m, int len, int how) - Prepend len bytes in te

beginning.

mbuf aaaaa mbuf

mh_data

mh_next

mh_len mh_len

mbuf aaaaa mh_next

mh_data mh_data

mh_len mh_len

mbuf

Accessing data● Because mbufs are divided to 256-bytes parts, header might fall between two

mbufs.● Accessing the header linearly might cause an unexpected behaviour.

mbuf aaaaa mbufhea derWrite.. OVERFLOW...

● NEVER access directly, or before using this:● m_pulldown(struct mbuf *mbuf, int offset, int len, int *offsetp)

mbuf aaaaa mbufhea dermbuf a mbufheader

Might allocate a new mbuf

Interfaces● Interfaces in FreeBSD are represented by struct ifnet

struct ifnet { struct vnet *if_vnet; /* pointer to network stack instance */ TAILQ_ENTRY (ifnet) if_link; /* all struct ifnets are chained */... char if_xname [IFNAMSIZ]; /* external name (name + unit) */... struct ifaddrhead if_addrhead ; /* linked list of addresses per if */... u_short if_index ; /* numeric abbreviation for this if */

int (*if_output) /* output routine (enqueue) */ (struct ifnet *, struct mbuf *, const struct sockaddr *, struct route *); void (*if_input) /* input routine (from h/w driver) */ (struct ifnet *, struct mbuf *);… void (*if_transmit ) /* initiate output routine */ (struct ifnet *, struct mbuf *); u_int if_fib ; /* interface FIB */...};

/net/if_var.h

L3 Hooking● Just like Linux has netfilter, FreeBSD has a framework called pfil● It enables to create a list of filters for both IN and OUT packets.● Unlike Linux, pfil allows hooking in only one place for incoming and outgoing

packets.

L3 Hooking● Hooking is easy. Use:

struct pfil_head *pfh_inet;

/* Initializing L3 Hooking */ if (!(pfh_inet = pfil_head_get (PFIL_TYPE_AF , AF_INET))) { uprintf ("Failed getting packet filter head \n");

return ESRCH; }

pfil_add_hook(in_filter, NULL, PFIL_IN | PFIL_WAITOK, pfh_inet);

static int in_filter(void *arg, struct mbuf **m, struct ifnet *ifp, int dir, struct inpcb *inp)

● Then, register the callback:

● Hook signature:

The L2-L3 Input StackDriver

ifp->if_input()

ether_input_internal

BPF

LAGG

ng_ether

BridgingHooks

ether_demuxVLAN Handlingvlan_input_p()

IP: ip_input() IPv6 ARP: arpintr() ATALK AARP

pfil_run_hooks()

PFil

The L2-L3 Input Stack

Driver

ip_output

pfil_run_hooks()

if_output()

ether_output

Bridge

ng_ether

PFilHooks

ifp->if_transmit()

L2 Hooking● Apparently, it’s not as trivial hooking to the network stack in L2● For example, in order to make Libpcap work, NIC drivers need to explicitly call

Libpcap kernel hooks to redirect L2 flow to it.● Suggested implementation in user mode:

○ BPF - Explained in previous lectures○ Libpcap - Explained above

○ Nethook - Memory-mapping based network handling. Exists in both Linux, Windows and FreeBSD.

● Despite what is said above, you can use netgraph to attach to ng_ether.○ There is a way to use it more easily. Source code will be uploaded later.

● DDB is the static kernel debugger. You can read about it here:○ https://www.freebsd.org/cgi/man.cgi?ddb(4)

● Compile kernel with:○ Options DDB

● Compiling the kernel:○ Configs are in:

■ amd64/conf/GENERIC■ Always copy GENERIC to a new file and edit it.

Other useful tips

$ cd /usr/src/$ make buildkernel KERNCONF=GENERIC.MYCONF && make installkernel KERNCONF=GENERIC.MYCONF && shutdown -r now

● If kernel hangs, useful VirtualBox command (Opens DDB):$ VBoxManage debugvm <VM Name> injectnmi

Questions? :)