IPsec Used as a Protocol for a VPN

Michael Evans

December 16, 2013

CNT 4104 Software Project in Computer Networks

Instructor: Dr. Janusz Zalewski

Computer Science & Software Engineering Programs

Florida Gulf Coast University

Ft. Myers, FL 33965

© Copyright 2013 by Florida Gulf Coast University

Evans

2

1. Introduction

IPSec is a protocol suite used in Virtual Private Networks (VPN). IPSec can encrypt

traffic between two nodes using a Transport Mode. This encrypts the upper levels of the

datagram. The entire datagram can also be encrypted using Tunnel Mode, in the form of a VPN.

Its subprotocols consist of Authentication Headers (AH) and Encapsulating Security Payloads

(ESP).

ESP provides packet authentication, data integrity, and anti-replay features, plus

confidentiality. AH provides the above, except for confidentiality. Any fields in an AH header are

unencrypted, authenticating the outer header of an IPSec packet. ESP headers are also

unencrypted, but it also has a partially encrypted trailer, thus encapsulating the data. AH is used

with ESP because it provides an assurance of authenticity and integrity. Also, ESP must support

DES encryption and some policies have approved only AH for that purpose [1].

Figure 1.1 – A sample of a construction and interpretation of an AH packet

Figure 1.1 shows the (Security Parameters Index (SPI) and the Sequence Number.

The SPI is used by the receiver to determine which Security Association (SA) authenticated the

packet. The Sequence Number is what provides the aforementioned anti-replay protection. This

is to prevent malicious hackers from injecting their own packets int to the datastream. If the

recipient receives a second packet with the same Sequence Number, it is rejected. An ESP

packet, as shown in Figure 1.2, has the above features, as well as a padding section, used to tell

the receiver how much padding has been added to the packet.

Evans

3

Figure 1.2 – A sample of construction and interpretation of an ESP packet

Internet Key Exchange (IKE) is a third subprotocol in the IPSec suite. IKE builds upon

the Internet Security Association and Key Management Protocol ISAKMP) and Oakley protocol.

Cryptographic keys are traded over IKE using Diffie-Hellman key exchange. Diffie-Hellman

exchange operates using a public key system with numbers from a pre-agreed upon group of

prime numbers p, generated by a pre-agreed upon generator g. The products of gx mod p and gy

mod p are exchanged as public values and then generate the shared private key value of gxy mod

p, thus making it possible to secure an insecure channel. The drawback to this approach is

vulnerability to a man-in-the-middle attack [1].

IPSec implementation can be done within an end user’s computer or in the routers

between end points of the network connection. Native implementation, for which the logic is

roughly sketched in Figure 1.3, can be integrated into the operating system. A replacement stack

can be written by a third party to be used in place of the operating system vendor’s native

implementation. A third possibility is that it can be configured as a compatibility layer between

the network and the link layer of the protocol stack, also known as the “Bump in the Stack”

(BITS). Both host implementation and BITS have their advantages and disadvantages [2].

Evans

4

Figure 1.3 - An example hardware layout of host implemented IPSec

Router implementation, for which the logic is roughly sketched in Figure 1.4, has similar

options. The first is a native implementation that is integrated into the router’s embedded

operating system. The second is a “Bump in the Wire” approach, similar to BITS. It is less

frequently used as it requires dedicated hardware to be added to each router in the network.

Figure 1.4 - An example hardware layout of router implementation

Evans

5

As IPSec is a widely used method of sending encrypted communications over the

network layer, and it is of great importance for IT professionals everywhere. Few entities these

days rely entirely on in-house networks. Usually their networks are connected through an

internetwork operated by a third party. Therefore, many VPNs use IPSec for private and secure

corporate and personal communications.

The existence of VPN technology, which uses protocols like IPSec, provides great utility

to even those working outside the world of information technology. Encryption is a guarantor of

privacy to society, something desirable in itself. In a time in which nation-state governments

routinely monitor vast amounts of Internet traffic, IPSec becomes a valuable tool in maintaining

personal privacy.

Evans

6

2. Definition of the Problem

Florida Gulf Coast University has Wi-Fi hotspots located throughout the campus. The

problem that arises from this is that personal messages can be snooped on or stolen. The

objective of this project is set up a custom VPN server, which uses the IPSec protocol, that can

be accessed over the school Wi-Fi.

The original plan was to use one of FGCU’s existing Linux Fedora-based servers and

configure a virtual private network which uses IPSec. This was to be followed up by testing the

security of the VPN. When a reasonable degree of security is reached, the server can be beta

tested by setting up user accounts and interfaces. Ideally, the accounts can be based on students’

Eagle mail accounts and passwords, though time constraints may not allow that phase to be

reached.

Evans

7

3. Design Solutions

The design solution involves configuring two machines to run IPSec and being able to

exchange, at minimum, IPSec encrypted packets. One is a server running Linux Fedora 9 at

sanibel.cs.fgcu.edu. The other is a laptop running Ubuntu 12.04. Both will have Openswan, an

open source IPSec implementation [3].

Once the needed IPSec tools are installed and configured on both systems, theywill

exchange their public keys. Both will be connected to the Internet and a virtual private network

connection will be established. They can then exchange packets, encrypted according to the

flowchart in Figure 3.1.

Figure 3.1 - IPSec encrypting packets. © 1999, 2011 Oracle

Evans

8

4. Implementation of the Server

In theory, all that is needed to get IPSec running is to install a few packages and change a

few settings. This is more complicated in practice.

The initial intention was to deploy implementation on a virtual machine running the

server edition of Ubuntu 12.04. After first attempting to use a Fedora based server, work returned

to the desktop edition of Ubuntu 12.04. While this sounds simple enough, doing so from scratch,

without a step-by-step guide, took many hours of research and debugging. However, the script in

Appendix A gives an administrator with root privileges a good start installing and configuring

IPSec. It creates files throughout /proc/sys/net/ipv4/conf/ directories needed to pass

ipsec verify’s checks, then sets IPSec to run on start up.

The administrator will need to manually turn on IP forwarding by changing

"net.ipv4.ip_forward = 0" to "net.ipv4.ip_forward = 1" in /etc/sysctl.conf,

as shown in Figure 4.1. The sysctl must then be reloaded with the command sysctl -p

sysctl.conf. Finally, ipsec pluto starts the pluto daemon.

Figure 4.1 - Enabling IP forwarding

Evans

9

Key to encryption is a pre-shared key. ipsec showhostkey --left revealed that

ipsec.secrets was empty, as expected on a machine not previously configured. ipsec

newhostkey --output /etc/ipsec.secrets created an RSA key to be used for IPSec

communication. The /etc/ipsec.secrets file was also updated to include a pre-shared key

to be used between client and server. Figure 4.2 shows the RSA key, the left and right addresses

the PSK is to be shared between, and a commented out PEM (Privacy-enhanced Electronic Mail)

certificate, created for, but not used in, this project.

Figure 4.2 - Contents of /etc/ipsec.secrets

To authenticate users, a list of users and passwords were added to /etc/ppp/chap-

secrets. The format, shown in Figure 4.3, is username, method of connecting, password, and

IP addresses that a given user may connect from. For example, zalewski can connect using

L2TP from any IP address while mevans can connect using any method from any IP address.

Evans

10

Figure 4.3 - Sample formatting for /etc/ppp/chap-secret’s user list

Appendix B has the contents of the server’s /etc/ipsec.conf file, which can easily be

adapted for any IPSec server using Transport Mode instead of Tunnel Mode and pre-shared keys

instead of certificates.

Over the course of troubleshooting, openswan-2.6.39 was compiled from source and

installed to try to minimize bugs in the program. This did not prove successful. Openswan was

removed and replaced with strongSwan using apt. However, strongSwan also did not resolve the

issues and appeared to introduce new issues. Openswan was reinstalled using apt, which created

the problem of having to manually edit ipsec verify located in

/usr/local/libexec/ipsec/verify. The methods udp500check() and

udp4500check() were fixed to look for ss in /bin/ instead of /usr/sbin/. This is not

necessary for configuration, but is mentioned in the event that using a later version of Openswan

for ipsec verify becomes relevant to future troubleshooting.

Evans

11

5. Experiments with the Client

Initial tests of the client used a computer running Mac OS X, version 10.8.5.

Unfortunately, it repeatedly failed to connect to the server. Researching the problem suggested

that difficulties in connecting Mac OS X to a VPN over IPSec is often due to Apple’s

implementation of IPSec. Testing was then switched to computers running Windows 7 and

Windows 8. A new connection was configured from the Network and Sharing Center, shown in

Figure 5.1.

Figure 5.1 – Windows Network and Sharing Center

VPN configuration is listed under “Connect to a workplace”, as shown in Figure 5.2.

Connections are made over the local Internet connection instead of directly dialing the server,

shown in Figure 5.3. Then the setup wizard requires the IP address, in this case, 10.100.40.173,

and a name for the connection, shown as IPSecZal in Figure 5.4. Finally, a username and

password that match the one from /etc/ppp/chap-secrets is entered, shown in Figure 5.5.

Evans

12

Figure 5.2 – Connecting to a workplace (VPN)

Figure 5.3 – Choosing a means of connecting

Evans

13

Figure 5.4 – IPSecZal

Figure 5.5 – Credentials window

Evans

14

To advance further, in the network connection window, “Connect IPSecZal”, click

properties, as shown in Figure 5.6. In the security tab, change type of VPN to Layer 2 Tunneling

Protocol with IPSec (L2TP/IPSec), shown in Figure 5.7. Then, click Advanced Properties to

ender the PSK as shown in Figure 5.8.

Figure 5.6 – IPSecZal login window

Evans

15

Figure 5.7 – IPSecZal Properties

Figure 5.8 – IPSecZal Advanced Properties

Evans

16

Under normal circumstances, the client would connect to the server. However, the

network being tested on is behind a NAT (Network Address Translation), giving errors like the

one shown in Figure 5.9. Network Address Translation allows multiple computers to use a single

IP address. This is completely opposed to IPSec’s packet header authentication, and will often

break it. Further testing showed that it was not possible to telnet into the server on ports 500 and

4500 while the firewall was disabled, confirming the presence of a NAT.

Figure 5.9 – Error Connecting to IPSecZal

Evans

17

6. Conclusion The scope of this project proved that a longer period than one semester could be

involved. While configuring a computer to run IPSec was possible within the timeframe, a fully

encrypted virtual network was not. The bash script in Appendix A will configure IPSec on the

Sanibel server. For ease of use, it can also be found in /home/mevans/cnt/ at

sanibel.cs.fgcu.edu.

Several attempts were made to establish a working IPSec connection. The first attempt to

run the network virtually as a proof of concept failed. The size of the virtual machines, as well as

VirtualBox requiring an Ethernet connection not present on the developer's laptop, made this

impossible. Additionally, any virtual network would eventually have to be implemented on

physical machines.

An all-Linux host-to-host connection proved unfeasible as the Ubuntu laptop did not have

a static connection. A Linux/Windows mixed host-to-host connection using the server at

69.88.163.11 (Sanibel) also did not pan out. The network-to-network connection could not be

established due to the lack of access to the Sanibel server and the aforementioned incompatibility

between VirtualBox and bridged adapter mode.

Limited success was achieved with the server running the desktop edition of Ubuntu

12.04. A working IPSec server was configured and the logs from that server confirm the

beginnings of an IPSec connection. Unfortunately, the NAT prevents any real connection from

being established at this time.

Finally, an alternative for using Openswan may be a Freeswan [4], but this option was not

tired in this project.

Evans

18

7. References

[1] Kaufman, E. and A. Newman, Implementing IPsec: Making Security Work on VPNs,

Intranets, and Extranets. John Wiley and Sons. 1999.

[2] Doraswamy, N. and D. Harkins, IPSec: The New Security Standard for the Internet, Intranets,

and Virtual Private Networks. Prentice Hall PTR. 2003

[3] Openswan: An IPsec implementation for Linux. URL: https://www.openswan.org/

[4] Freeswan: An implementation of IPSEC & IKE for Linux. URL:

http://www.freeswan.org/intro.html

Evans

19

Appendix A. Script for Installing and Configuring IPSec

#!/bin/bash

apt-get install xl2tpd openswan ipsec-tools;

# fix redirect errors

for f in /proc/sys/net/ipv4/conf/*/accept_redirects;

do echo 0 > $f;

done;

for f in /proc/sys/net/ipv4/conf/*/send_redirects;

do echo 0 > $f;

done;

chkconfig ipsec on; # make sure ipsec is running on start up

Evans

20

Appendix B. Content of Server’s ipsec.conf file

version 2.0

config setup

nat_traversal=yes

virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12

oe=off

protostack=netkey

conn L2TP-PSK-NAT

rightsubnet=vhost:%priv

also=L2TP-PSK-noNAT

conn L2TP-PSK-noNAT

authby=secret

pfs=no

auto=add

keyingtries=3

rekey=no

ikelifetime=8h

keylife=1h

type=transport

left=10.100.40.173 # Your server's IP address here

leftnexthop=%defaultroute

leftprotoport=17/1701

right=%any

rightprotoport=17/%any