Packet Mangling for Fun & Profit

A Brief Intro to Netfilter, User Mode Linux, and the Linux

TCP/IP Stack

Topics SplitTCP Summary (Context) Netfilter Introduction One buffer to rule them all:

sk_buffs User Mode Linux Current Status

SplitTCP In a Nutshell Work started by Michalis, Srikanth, and

Swastik. The idea is to add transport layer proxies to

long (in terms of hop count) TCP connections.

If a link fails (due to mobility, etc.) the packet can be re-transmitted by the proxy closest to the failure.

A “demo-quality” implementation was done which made many simplifying assumptions.

So What? My current task is to take that

demo implementation and turn it into a general implementation.

This presentation discusses the lessons learned up to this point in that process.

Design Goals Any and all modifications made to

the behavior of TCP should be backwards compatible. I.e. nodes should interoperate

regardless of weather or not they’re “split tcp enabled”.)

If possible, no changes should be made to the kernel proper. It’s just better for everyone that way…

Enter Netfilter Fortunately, Linux has a plugin API,

called Netfilter, which allows kernel modules to hook into strategic spots in the networking stack.

Unfortunately, in the grand tradition of open source, documentation takes a back seat to implementation – so there’s a non-trivial learning curve.

Sideline: Plugin APIs An example of the Delegation

design pattern (GoF) – a work unit is passed through a series of cooperating steps to produce the final result.

A well-designed plugin architecture can facilitate otherwise impossible tasks and simply possible ones.

Sideline: Kernel Modules Most of the linux kernel can be

built as a module – a bit of code that’s loaded into (and unloaded from) the kernel dynamically.

Device drivers, file systems, even networking protocols (IPX, Appletalk) are candidates for modularization.

Writing a Module Just define a few preprocessor

macros (__KERNEL__, MODULE) Include a few header files

(linux/module.h, linux/version.h, linux/config.h)

And honor a small interface (module_init(), module_exit())

You’ll have a no-op module.

Netfilter Overview Initial design by Paul “Rusty” Russell Netfilter is a series of callback

functions within the network stack. The API is non-portable and appeared in linux 2.3.x

Each protocol has it’s own set of callback points. We care about IPv4.

Netfilter Concepts A module expresses interest in being

invoked at an arbitrary subset of the available callback points – specifying the function and the (global) priority in which it should be called.

That function is passed (among other things) a pointer to a pointer to a packet buffer ( sk_buff ** ).

Return Values The netfilter function has five

possible return values: NF_ACCEPT: continue callback chain NF_DROP: drop the packet and stop the

chain NF_STOLEN: stop the chain NF_QUEUE: send the packet to

userspace NF_REPEAT: call the hook again

Netfilter Hooks in IPv

Routing Engine

Local Sockets

1In Out

2

34

5

Say that Again? 1: NF_IP_PREROUTING

any received packet which checksums OK. 2: NF_IP_LOCAL_IN

packets destined for local sockets 3: NF_IP_FORWARD

foreign packets being forwarded 4: NF_IP_POST_ROUTING

any outbound packet 5: NF_IP_LOCAL_OUT

packets originating from local sockets

Routing Engine

Local Sockets

1In Out

2

4

5

An sk_what?? Linux uses a structure called an

sk_buff to store packet data internally.

It contains a handful of pointers to other structures as well as a packet data region.

Data

Poin

ters

Head

room

Tailr

oom

Sk_buff’s and you The data area is like a stack, only

you can insert at the head and the tail (deque?). The kernel provides a handful of

helper functions to manage sk_buff’s and their data areas.

The various header pointers point into the data area – which can be thought of as a serialized packet.

Why do We Care? An sk_buff is built as a packet travels

down the stack – each layer (TCP, IP, Ethernet) adds their own special sauce.

This means that each header is “squashed” in against the next – so while modifying existing data is relatively easy, adding new header data is a bit trickier.

Don’t leave me in suspense… Basically, you make a copy of the

sk_buff, and ask it to “grow” a bit during the copy.

Once you have the copy – you “slide” the IP and TCP headers backwards a bit, insert the new option bytes, and re-checksum the packet.

A tale of “n” checksum’s Sounds easy, right? Remember

that these sk_buffs are built one layer at a time?

There is no nice friendly function which will take a TCP sk_buff and compute all the needed checksums.

Funny thing about checksums – almost isn’t good enough.

Ok, insmod and <BOOM> Remember developing on a system

without memory protection and having to reboot ?

Kernel modules execute in kernel space – so no one’s watching your back. If you goof, it’s time to reboot.

User Mode Linux to the rescue

User Mode Linux (UML) A kernel patch that allows running

the linux kernel as a user-mode process on a linux machine.

If you crash the user-mode kernel, you just restart the process, no reboot required.

Where do I sign up? Setup is (in principal) fairly easy – only it

turns out that the standard distribution doesn’t have netfilter enabled.

So I re-built with the appropriate options and placed the binaries in ~swift/user-mode-linux/bin

There’s a README there – some support executables must be installed as root on your workstation.

That’s It? Not quite – you also need a file system

to boot this kernel off. Good News: you can download a file

system image – you don’t have to make one. Bad News: it’s really big. (up to 700MB)

Good News: you can share one among many people Bad News: you have to read the HOWTO.

Building Modules Under UML Building a kernel module is the

same under UML as under “KML” – it is important that you build it against the source used to build the target kernel.

For “our” UML build – that source is in ~swift/user-mode-linux/src

Current Status A skeleton of a splittcp module

(tcpproxy.c) exists. It can inspect locally generated packets

and add our newly defined PROXY option, rechecksum the packet, and send it on it’s way.

It can inspect arriving packets, check for the option, and decide if that packet should be proxied.

So What’s Left? It doesn’t (yet) actually proxy the

packet. Nor does it send an

acknowledgement of receipt to the upstream proxy.

There are also issues around ICMP error messages, and what should be done about them.

Conclusion Once the code is “alpha” quality,

I’ll commit it to the swift CVS repository for your collective viewing pleasure.

Until then, if you have questions or suggestions, see me.