2009 CPUG CON EUROPE Kono Yasushi IPSec VPN How Does It Really Work

IPSec VPN:IPSec VPN:

How does it really work?How does it really work?

Yasushi Kono Yasushi Kono

(ComputerLinks Frankfurt)(ComputerLinks Frankfurt)

Before the AgendaBefore the Agenda

My intention of this presentation:My intention of this presentation:

I know that many people do not have any clue I know that many people do not have any clue what‘ s happening while establishing Security what‘ s happening while establishing Security Associations prior to create VPN tunnels.Associations prior to create VPN tunnels.

This topic is quite complex. I want you to get This topic is quite complex. I want you to get started into this topic taking away a bit of its started into this topic taking away a bit of its complexity.complexity.

AgendaAgenda

Introductory Information on IPSec VPNIntroductory Information on IPSec VPNWhy Diffie-Hellman Algorithm?Why Diffie-Hellman Algorithm?IKE SA in Main Mode IKE SA in Main Mode IPSec SA in Quick ModeIPSec SA in Quick ModeSome Troubleshooting TipsSome Troubleshooting Tips

AgendaAgenda


Introductory Information on IPSec VPNIntroductory Information on IPSec VPN

Before establishing a Site-to-Site VPN Before establishing a Site-to-Site VPN connection, both gateways must agree upon connection, both gateways must agree upon parameters for encrypting communication. parameters for encrypting communication.

This negotiation process is divided into two phases:

Phase 1 Phase 1 and Phase 2Phase 2.

In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore:





1.) Encryption Algorithm (3DES, AES-128, AES-256,…) to ensure 1.) Encryption Algorithm (3DES, AES-128, AES-256,…) to ensure privacyprivacy





1.) Encryption Algorithm (3DES, AES-128, AES-256,…) to ensure privacy

2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity






2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity

3.) Diffie-Hellman Group (1, 2, 5, or 14)3.) Diffie-Hellman Group (1, 2, 5, or 14)

Before establishing a Site-to-Site VPN connection, Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for both gateways must agree upon parameters for encrypting communication. encrypting communication.





2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity

3.) Diffie-Hellman Group (1, 2, 5, or 14)

4.) Method of mutual authentication (Preshared Key or Certificate)4.) Method of mutual authentication (Preshared Key or Certificate)

You have the choice between two different You have the choice between two different modes in Phase 1:modes in Phase 1:

• Main ModeMain Mode• Aggressive ModeAggressive Mode

• Main Mode consists of a 6 packet negotiationMain Mode consists of a 6 packet negotiation• In Aggressive Mode only three packets are In Aggressive Mode only three packets are

exchanged between both VPN gateways.exchanged between both VPN gateways.

Before beginning to analyze the Before beginning to analyze the information exchanged by the information exchanged by the gateways, let‘s have a look at the Diffie-gateways, let‘s have a look at the Diffie-Hellman Algorithm:Hellman Algorithm:

Why is Diffie-Hellman necessary?Why is Diffie-Hellman necessary?

AgendaAgenda


Usually, payloads are encrypted Usually, payloads are encrypted symmetrically by means of symmetric symmetrically by means of symmetric encryption algorithms, like 3DES or AES. encryption algorithms, like 3DES or AES.

The problem with symmetric encryption is The problem with symmetric encryption is exchanging the keys over the Internet exchanging the keys over the Internet while preventing them from falling into while preventing them from falling into the wrong hands.the wrong hands.

One answer is asymmetric encryption. One answer is asymmetric encryption.

So, when asymmetric encryption is So, when asymmetric encryption is addressing problems with symmetric addressing problems with symmetric encryption, why is latter technology still encryption, why is latter technology still necessary?necessary?

The answer is:The answer is:

Performance!Performance!

The Diffie-Hellman Algorithm is not an The Diffie-Hellman Algorithm is not an Encryption Algorithm rather than a Key Encryption Algorithm rather than a Key Agreement Protocol.Agreement Protocol.

To understand the Diffie-Hellman Algorithm, To understand the Diffie-Hellman Algorithm, you have to have basic knowledge on the you have to have basic knowledge on the Modulo operation.Modulo operation.

The Modulo operation finds the remainder The Modulo operation finds the remainder of division of one number by another.of division of one number by another.

Given two numbers, a and b,Given two numbers, a and b,n = a mod bn = a mod b

is the remainder on division of a by b.is the remainder on division of a by b.

Examples:Examples:

30 mod 16 = 1430 mod 16 = 14114 mod 100 = 14114 mod 100 = 148 mod 3 = 28 mod 3 = 2

Isn‘t that simple?Isn‘t that simple?

Diffie-Hellman:Diffie-Hellman:Initiator takes a prime number p and an Initiator takes a prime number p and an integer a with 1 < a < p and a secret integer a with 1 < a < p and a secret integer x with:integer x with:

X = ax mod p

X, a, p: public parameters,X, a, p: public parameters,x: secret parameter.x: secret parameter.

Initiator sends X, a, and p to the Responder.Initiator sends X, a, and p to the Responder.The Responder takes a secret integer y and

computes:

Y = ay mod p

The number Y, which is public will be sent The number Y, which is public will be sent back to the Initiator.back to the Initiator.

Public: X, Y, a, pa= 2p= 5

Private: x 5a^x= 32X=a^x mod p 2

Private: y 6a^y= 64Y=a^y mod p 4Y^x= 1024,00X^y= 64,00Kx=Y^x mod p 4Ky=X^y mod p 4

Initiator and Responder are computing their Initiator and Responder are computing their common encryption key:common encryption key:

• Kx = Yx mod p

• = (ay mod p)x mod p• = (ax mod p)y mod p

• = Xy mod p = Ky

• Hence Kx = Ky

Now, that we have the basic Now, that we have the basic understanding of the Diffie-understanding of the Diffie-Hellman Algorithm, we can now Hellman Algorithm, we can now fully understand IKE SA in Main fully understand IKE SA in Main Mode.Mode.

AgendaAgenda


As all of you might already know, As all of you might already know, IKE SA in Main Mode consists of 6 packetsIKE SA in Main Mode consists of 6 packets

First of all, the Initiator sends the IKE SA First of all, the Initiator sends the IKE SA Parameters to be negotiated upon to the Parameters to be negotiated upon to the Responder: Responder:

IKE SA: IKE SA: 3DES or AES-128?3DES or AES-128?SHA-1 or MD5?SHA-1 or MD5?DH-Group 5 or 14?DH-Group 5 or 14?Certificate or Preshared Key?Certificate or Preshared Key?

Initiator Initiator ResponderResponder

Packet Number 1:Packet Number 1:

The Responder sends back the parameters The Responder sends back the parameters to be used in common to the Initiator:to be used in common to the Initiator:

IKE SA: IKE SA: AES-128!AES-128!SHA-1! SHA-1! DH-Group 14!DH-Group 14!Preshared Key!Preshared Key!




Then, the Initiator sends the public Diffie-Then, the Initiator sends the public Diffie-Hellman Parameters and a random Hellman Parameters and a random number, which is called „Nonce“:number, which is called „Nonce“:



Then, the Responder sends back its own Then, the Responder sends back its own public Diffie-Hellman Parameters and its public Diffie-Hellman Parameters and its own random number („Nonce“):own random number („Nonce“):




Both parties know from each other, which Both parties know from each other, which parameters and encryption key to use in parameters and encryption key to use in common. common.

The Initiator then The Initiator then • builds the hash of the Preshared Keybuilds the hash of the Preshared Key• encrypts the Nonce of the Responder and encrypts the Nonce of the Responder and

the Hash of the Preshared Keythe Hash of the Preshared Key




What happens next?What happens next?

Each of the Gateways are receiving their Each of the Gateways are receiving their own Nonces encrypted by other parties.own Nonces encrypted by other parties.

The next step is to decrypt the encrypted The next step is to decrypt the encrypted Nonces to verify the identity of the Nonces to verify the identity of the communicating gateways.communicating gateways.

After receiving the sixth After receiving the sixth packet, you will get the packet, you will get the following message in following message in SmartView Tracker:SmartView Tracker:

IKE SA: Main Mode completionIKE SA: Main Mode completion

AgendaAgenda


Why is Quick Mode Why is Quick Mode necessary?necessary?

To establish the IPSec SA!To establish the IPSec SA!

Quick Mode to establish an Quick Mode to establish an IPSec SA consists of 3 IPSec SA consists of 3 packets.packets.

You have to negotiate upon:You have to negotiate upon:• The Encryption algorithmThe Encryption algorithm• The Hash AlgorithmThe Hash Algorithm• The IPSec Protocol (ESP, AH)The IPSec Protocol (ESP, AH)• If PFS is to be used or notIf PFS is to be used or not

(if yes, the DH-Group is to be (if yes, the DH-Group is to be determined again)determined again)

One word regarding IPSec One word regarding IPSec Protocols:Protocols:Authentication Header (AH)Authentication Header (AH)

provides: provides: • Data Integrity CheckingData Integrity Checking• Replay ProtectionReplay Protection

Encapsulating Security Payload Encapsulating Security Payload (ESP) provides:(ESP) provides:

• Payload EncryptionPayload Encryption• Data Integrity CheckingData Integrity Checking• Replay ProtectionReplay Protection

Why is it necessary to Why is it necessary to agree upon theagree upon the

• Encryption AlgorithmEncryption Algorithm• Hash AlgorithmHash Algorithm• Eventually DH-GroupEventually DH-Group again?again?

Because the parameters Because the parameters negotiated there is for negotiated there is for applying to the payload of applying to the payload of the packets and not to the the packets and not to the identity of the other gateway!identity of the other gateway!

The 3 Packets in Quick ModeThe 3 Packets in Quick Mode

The Message you will get after succesful The Message you will get after succesful IPSec SA Negotiation is:IPSec SA Negotiation is:

IKE SA: Quick Mode completionIKE SA: Quick Mode completion

AgendaAgenda


Some Troubleshooting Methods:Some Troubleshooting Methods:

Mostly, you will get the right information with Mostly, you will get the right information with SmartView TrackerSmartView Tracker

There, the most frequent error messages are:There, the most frequent error messages are:• IKE SA: No proposal chosenIKE SA: No proposal chosen• Encryption failure: No valid SAEncryption failure: No valid SA• INVALID_ID_INFORMATIONINVALID_ID_INFORMATION

Some Troubleshooting Methods:Some Troubleshooting Methods:

Received Notification from Peer: Malformed PayloadReceived Notification from Peer: Malformed Payload

Any solution?Any solution?

One standard method for Debugging One standard method for Debugging IKE/IPSec is IKE/IPSec is

vpn debug ikeonvpn debug ikeon

to generate the output file to generate the output file ike.elgike.elg. .

But, without any knowledge of the theory But, without any knowledge of the theory of IPSec, is it useful to analyze the of IPSec, is it useful to analyze the ike.elgike.elg file?file?

For more troubleshooting IKE/IPSec in a Check For more troubleshooting IKE/IPSec in a Check Point Environment, attend the next Point Environment, attend the next presentation:presentation:

305: Troubleshooting in the 305: Troubleshooting in the Check Point Environment – Check Point Environment –

Part IIPart II

By Tobias LachmannBy Tobias Lachmann

Any Questions?Any Questions?

Thanks a lot for Thanks a lot for your attention!your attention!

Should you have questions:Should you have questions:

[email protected]@computerlinks.de

Documents

2009 CPUG CON EUROPE Kono Yasushi IPSec VPN How Does It Really Work