VNS3 IPsec Troubleshooting Documentation

!!!

VNS3 IPsec Troubleshooting Guide

!Contents

VNS3 Virtual Network Controller

Overview 2

Diagram of Typical Deployment 2

Configuration and Launch Troubleshooting

Launching Controller Instance 3

Generating Clientpacks 3

Peering Controllers 3

Connecting Clients to the Overlay Network 3

IPsec Troubleshooting

Compatibility 4

Troubleshooting 4

CONNECTED STATE EXAMPLE - IPsec Tunnel is connected 5

IPsec Tunnel is connected without NAT-Traversal encapsulation 6

PAYLOAD_MALFORMED - Pre-Shared Key Mismatch 7

NO_PROPOSAL_CHOSEN - Algo/Hashing/DH/PFS Mismatch 8

INVALID_ID_INFORMATION - Remote Subnet Mismatch 9

Unstable Tunnel Troubleshooting 10

© Cohesive Networks 2015Page �� of ��1 10VNS3_3.x_IPsec_Troubleshooting

20 North Wacker Drive • Suite 1200 • Chicago, IL 60606

!VNS3 Vir tual Network Controller Overview VNS3 provides an overlay network which is a computer network built on top of another network. Nodes in the overlay network can be thought of as being connected by virtual or logical links, each of which corresponds to a path, perhaps through many physical links, in the underlying network. !Cloud-based overlay networks can be used to maintain control across multiple locations, allowing the customer to control their own network topology, the network addressing, encrypted communication, and desired network protocols, all in a scalable, highly redundant form. !Overlay networks are based on redundant, encrypted, point-to-point connections from cloud-based servers to VNS3 virtual appliances running in the cloud. These VNS3 hybrid devices act as virtual routers, virtual switches, SSL VPN concentrators, IPSec VPN concentrators, firewalls, and protocol re-distributors of this virtual network which essentially sits above your physical network. A virtual network gives a customer a common LAN-like network where the servers can be located in the physical data center (virtualized or not), private cloud, and public cloud. !The advantages of virtual networks in the cloud are:

• They allow user control over: - Addressing (custom private addressing for cloud-based servers). - Topology (virtual network comprised of virtual switches, virtual bridges, and virtual routers). - Protocols (protocols like UDP Multicast are available for service election/discovery).

• Encrypted communication between cloud-deployed servers. • Secure connection to existing data-center-based extranet solutions via an IPsec tunnel. • Abstraction of location from network identity which allows for quick and easy disaster recovery. • Provides a user-controlled layer of security for compliance reporting and auditing. • Allows existing NOC to monitor and manage the cloud-based servers.˙ !

Diagram of Typical Deployment

The Cloud Instances in the cloud-based Overlay Network are referred to as Connected Clients.

!!!!!!


L A N

IPsec Device Router (optional)

Your Data Center

Overlay Clients

Overlay Network

Public Cloud

VNS3

IPsec

If present enter Public IP as Endpoint

Use Local IP as NAT IP w/router Use Public IP as Endpoint IP w/o router


http://www.cohesive.net/products/vns3

!Configuration and Launch Troubleshooting Please send any support inbounds questions or requests that require escalation to [email protected]. !Launching Controller Instance

VNS3 Images are available in the public catalogs of many major cloud providers. Users simply launch an instance of the public images using either the hardcoded Free Edition or contacting Cohesive to purchase a license file for one of the premium Editions. !Generating Clientpacks Issues with generating Clientpacks are extremely rare and often relate to underlying cloud hardware errors when launching instances or the use of invalid licenses. If there is a problem generating clientpacks either launch a new instance from scratch or forward the support request to CohesiveFT. !Peering Controllers Problems associated with Controller Peering are either related to Security Group/Network Policy rules or the use of non-matching VNS3 Licenses, Security Tokens, or Overlay Network Settings between the Controllers to be peered. If there is a problem peering Controllers check that UPD ports 1195-1197 are accessible between Controllers and the topology and keyset checksums listed on the Runtime Status page match between the two Controllers. !Connecting Clients to the Overlay Network Clients connect to the Overlay Network using the OpenVPN client. Problems with connecting to the Overlay Network can be classified into three categories:

1. Network Access - Client Servers need network access to the Controller(s) on UDP port 1194. Access can be blocked by either the Client Servers’ local Firewall or the Inbound Hypervisor Firewall Rules on the Public Cloud. How a users opens up this network access depends on the cloud deployment target and is covered in the configuration document. Check that the necessary access for the OpenVPN connection has been enabled.

2. OpenVPN Version - Cohesive recommends OpenVPN 2.2+. 3. Duplicate Clientpacks on Multiple Clients - Client appears to be “hopping” on and off the network. This is

usually the result of the same client keys being installed on two client machines in the network. Only one client machine can use a set of credentials at a given time.

!



mailto:[email protected]?subject=

IPsec Troubleshooting Negotiating IPsec connections is not included in standard product support (see Cohesive Support Terms). This is due to the fact that the customer’s network architecture plays a significant role in the viability of such connections and access to the remote IPsec devices is usually prohibited. Cohesive offers IPsec implementation professional services (delivered remotely) at US$150 per hour or on a contract basis as our Enhanced Support. !Compatibility VNS3 supports a most IPsec datacenter extranet solutions but ultimately the configuration of the remote IPsec devices are the responsibility of the customer. !Preferred: Most models from Cisco Systems*, Juniper, Watchguard, Dell SONICWALL, Netgear, Fortinet, Barracuda Networks, Check Point*, Zyxel USA, McAfee Retail, Citrix Systems, Hewlett Packard, D-Link, WatchGuard, Palo Alto Networks, OpenSwan, pfSense, and Vyatta. Best Effort: Any IPsec device that supports: IKE1 or IKE2, AES256 or AES128 or 3DES, SHA1 or MD5, and most importantly NAT-Traversal standards. *Known Exclusions: Checkpoint R65+ requires native IPSec connections as Checkpoint does not conform to NAT-Traversal Standards and Cisco ASA 8.4(2)-8.4(4) bugs prevent a stable connection from being maintained.!Troubleshooting The following pages cover the most common problems encountered when negotiating IPsec tunnels between Customer IPsec devices and VNS3 Controllers. It is recommended that prior to tunnel negotiation both parties agree on the tunnel parameters and subnets to be advertised. Because the the complexity of customer network architectures, request for available private subnets to use for the Overlay Network Subnet may take time to process. !CONNECTED STATE EXAMPLES: IPSEC TUNNEL CONNECTED BUT NO TRAFFIC IS BEING PASSED PRE-SHARED KEY MISMATCH - PAYLOAD_MALFORMED ERROR ALGO/HASHING/DH/PFS MISMATCH - NO_PROPOSAL_CHOSEN ERROR REMOTE SUBNET MISMATCH - INVALID_ID_INFORMATION ERROR !



http://www.cohesive.net/support/support-plans/

http://www.cohesive.net/support/support-plans/

CONNECTED STATE EXAMPLE - IPsec Tunnel is connected IPsec tunnels show connected in three different places on VNS3:

1. Runtime Status Page

2. Remote Subnet Page

3. IPsec Log (either from the Remote Subnet page or https://<managerIP>:8000/logs/ipsec.log)

Note: The first message is Phase 1 (ISAKMP SA) and the second message refers to Phase 2 (IPsec SA). If tunnel traffic is not being passed but the tunnel is listed as connected as shown above, the customer is blocking communication via an intermediary Firewall/IPsec device access lists, the VNS3 deployment is running in an environment where NAT-Traversal encapsulation is required but not used (see IPsec Tunnel is connected without NAT-Traversal encapsulation), or the appropriate routes need to be pushed to the


ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=aes_256 prf=oakley_sha group=modp1536}

IPsec SA established tunnel mode {ESP/NAT=>0x3804dea4 <0x4c307861 xfrm=AES_256-HMAC_SHA1 NATOA=none NATD=54.202.22.40:4500 DPD=none}


IPsec Tunnel is connected without NAT-Traversal encapsulation NAT-Traversal Encapsulation is required when making connections to some cloud provider environments (i.e. AWS generic EC2 requires NAT-T encapsulation but AWS VPC doesn’t require NAT-T). VNS3 can make connections using either NAT-T or native IPsec, through a device wide setting on the IPsec page. The default VNS3 setting is to use NAT-Traversal. !When connecting to a customer IPsec device that is not using NAT-Traversal but it is enabled on VNS3 and/or required in the target overlay network cloud environment, the tunnel will show connected but no traffic will be passed. NAT-Traversal information is shown in two places on VNS3:

1. Remote Status Page

2. IPsec Log (IPsec Established line with NATD=none)

The solution to this issue is dependent on the deployment environment and whether or not NAT-Traversal encapsulation is required. If NAT-T is required, the customer will need to enable NAT-Traversal in order for the tunnel to pass traffic. If NAT-T is not required, toggle the VNS3 device wide setting to turn off NAT-T to get both sides of the connection using the same configuration for NAT-Traversal. Follow these steps on VNS3 to toggle NAT-T from the IPsec page:

• Delete all Remote Subnets • Delete all IPsec Endpoints • Turn off NAT-Traversal • Restart the VNS3 Controller via the VNS3 UI • Re-enter all IPsec Endpoints • Re-enter all Remote Subnets !


IPsec SA established tunnel mode {ESP/NAT=>0x3804dea4 <0x4c307861 xfrm=AES_256-HMAC_SHA1 NATOA=none NATD=none DPD=none}


PAYLOAD_MALFORMED - Pre-Shared Key Mismatch Pre-Shared Keys (PSK) are used to authenticate IPsec tunnel negotiations and are entered on both sides of the tunnel. If there is a mismatch due to incorrect entry, copy/paste extra characters, or omission the tunnel will show as not connected and the log will display the PAYLOAD_MALFORMED message.

Double check the PSK is entered correctly. It is recommended something short and easily remembered like test is used during initial tunnel configuration. !


sending notification PAYLOAD_MALFORMED to 50.77.151.179:4500


NO_PROPOSAL_CHOSEN - Algo/Hashing/DH/PFS Mismatch Both sides of the tunnel must use the same parameters for both Phase 1 and 2 of the negotiation. This includes the Algorithm, Hashing, Diffie-Hellman Group, and whether or not Perfect Forward Secrecy (PFS) is enabled. The customer usually decides what parameters to use based on their network policies/requirements. VNS3 will use an auto-discovery feature that attempts to match the settings of a customer’s remote device. It is recommended that the tunnel configuration is as specific as possible due to some IPsec makes/models’ incompatibility with this feature. The appropriate Extra Configuration Parameters syntax is included in the VNS3 Configuration Documentation. If there is a Algo/Hashing/DH/PFS mismatch the tunnel will show as not connected and the log will display the NO_PROPOSAL_CHOSEN message.

Double check what parameters are being used to negotiate the tunnel against what is entered on the VNS3 side in the Extra Parameters box (last text field on the IPsec Endpoint page). !!


ignoring informational payload, type NO_PROPOSAL_CHOSEN msgid=00000000


INVALID_ID_INFORMATION - Remote Subnet Mismatch Both sides of the tunnel must have the correct endpoint IP and remote subnet information entered as advertised by the other end of the tunnel. If there is a mismatch between what one side is advertising as the accessible remote subnet behind itself and what is configured on the other end of the tunnel the tunnel will show as not connected and the log will display the INVALID_ID_INFORMATION message.

Double check that both sides are using the appropriate Remote Subnets. Note all the information on the VNS3 side of the tunnel is displayed on the IPsec page.


ignoring informational payload, type INVALID_ID_INFORMATION msgid=00000000


Unstable Tunnel Troubleshooting IPsec tunnels to public cloud infrastructure connect via public Internet. As a result these connections can experience problems due to problems with intervening vendors, the public cloud’s edge, user’s edge and/or user’s internal routing. There are a number of techniques to maximize tunnel uptime. !If you have tunnels “flapping” (unexpected up/down events) there are two major reasons: !

1. Significant packet loss on the intervening Internet route between the IPSec peers. This is very rare, and to our experience has been limited to connections with remote peers in parts of APAC, Africa, and the Middle East. !

2. You have successfully connected but you do not have IDENTICAL configurations. IPSec interoperability is tricky even within vendors such that some Cisco products are very bad at talking to other Cisco products. With vendor interoperability you need to be even more precise. !

In the case of reason 2 above, there are 4 common culprits: !• IKE lifetime or SA/IPsec lifetime are not set to the same values on each end of the tunnel respectively.

These should be set to the same values for both ends to ensure coordinated re-key events.!• The IKE and SA lifetimes used have identical values for IKE and SA lifetime. For example both IKE

lifetime and SA lifetime are set to 28800 seconds. This can create a situation where IKE and SA re-key events happen simultaneously resulting in a hung tunnel where each side is using a different key set. They need to be different values. !

• There is a slight mismatch of the cryptographic algorithm family. While the initial negotiation of a tunnel using values like aes128 on one side and aes192 on the other appears successful, the resulting tunnel is unstable. !

• The remote IPsec device is End-of-Lifed and does not follow the most recent IPsec standard. If the IPsec device is 5 years old or NEWER - there should be no issue. If the IPsec device has been End-of-Lifed by its vendor or is on a very old software version a completely stable connection might not be possible. This is not a VNS3 issue, a stable connection would likely not be possible to any vendor. !

!



Technology

VNS3 IPsec Troubleshooting Documentation