The following topics will be covered by this overview:

NSX DISTRIBUTED FIREWALL OVERVIEW:.............................................................................................3

HOW NSX FIREWALL WORK..................................................................................................................4

NSX DFW ARCHITECTURE:....................................................................................................................7NSX Manager......................................................................................................................................7vCenter................................................................................................................................................7ESXi host..............................................................................................................................................7vShiled-Statefull-Firewal......................................................................................................................7Message Bus Client.............................................................................................................................8RabbitMQ............................................................................................................................................8vSIP......................................................................................................................................................8VPXA....................................................................................................................................................8IOChains..............................................................................................................................................8NSX DFW and VMtools:.....................................................................................................................11Spoofguard........................................................................................................................................13

NSX FIREWALL AND VMOTION:..........................................................................................................14vMotion.............................................................................................................................................14DFW table:........................................................................................................................................14DFW Connection table:......................................................................................................................14

NSX FIREWALL APPLIED TO:................................................................................................................16Cross cluster vMotion with “Apply To”..............................................................................................20

NSX L2 TO L4 FIREWALL:.....................................................................................................................21Default Policy....................................................................................................................................21

DFW EXCLUSION FUNCTIONALITY:.....................................................................................................22Resilience:..........................................................................................................................................23

NSX DFW LOGS:..................................................................................................................................30NSX Manager System Events.............................................................................................................30NSX Manager Audit Events:...............................................................................................................32ESXi DFW host Rules Messages:........................................................................................................32Live Flows:.........................................................................................................................................34

SERVICE COMPOSER:..........................................................................................................................36SECURITY GROUP........................................................................................................................................36DYNAMIC INCLUSION:.................................................................................................................................37STATIC INCLUSION:.....................................................................................................................................37STATIC EXCLUSION:.....................................................................................................................................39SECURITY POLICY USING SECURITY GROUPS:.....................................................................................................40

SECURITY TAG:...........................................................................................................................................43SECURITY POLICY:.......................................................................................................................................47

3RD PARTY VENDOR SERVICE INTEGRATION.......................................................................................54USING “PARTNER SECURITY SERVICE”:...........................................................................................................54USING SECURITY POLICY:..............................................................................................................................55THE FLEXIBILITY TO DEFINE:..........................................................................................................................55

BACKUP AND RESTORE:......................................................................................................................56

NSX DFW PRE-REQUIREMENTS:..........................................................................................................59

INSTALL NSX DFW:.............................................................................................................................60

TROUBLESHOOING.............................................................................................................................62DFW INSTALLATION...................................................................................................................................62

DNS Issues:........................................................................................................................................62TCP/80 from ESXi to vCenter is blocked:............................................................................................63Existing VIB’s Version........................................................................................................................64ESXi bootbank space issue:................................................................................................................64EAM TCP/80:.....................................................................................................................................64Download VIBs link:...........................................................................................................................64Reverting installation:.......................................................................................................................65DFW (UWA) agent issues:.................................................................................................................66DFW Kernel Space:............................................................................................................................68

PACKET CAPTURE COMMAND:...........................................................................................................71

NSX Distributed Firewall Overview:NSX DFW is an distributed firewall spread over ESXi host and enforced as close to source of the VM’s traffic (shown in each VM). The DFW runs as a kernel service inside the ESXi host.

With the NSX DFW we can enforce a stateful firewall service for VMs and the enforcement point will be at the VM virtual NIC – vNIC. Every packet that leaves the VM (before VTEP encapsulation) or enters the VM (After VTEP de-encapsulation) can be inspected with a firewall policy.

The DFW runs inside the ESXi host as a kernel space module, resulting in an impressive throughput.

What makes the DFW an amazing feature is that as we add more ESXi host to vSphere cluster we increase the DFW throughput capacity.

The DFW rules can be based on Layer 2 up to Layer 4 and with 3-Party vendor integration the NSX can implement security features up and including L7.

L2 rules are based on MAC address L2 protocols like ARP, RARP and LLDP etc. L3 rules are based on IP source destination and L4 uses a TCP or UDP service port.

The policy is created in centralized point at the vSphere vCenter server using vCenter web client. The objects used are being used from the vCenter inventory.

How NSX Firewall workThis section take from NSX Design guide:

The DFW instance on an ESXi host (1 instance per VM vNIC) contains 2 separate tables:

Rule table: used to store all policy rules.

Connection tracker table: cache flow entries for rules with permit action.

Note: a specific flow is identified by the 5-tuple information Source IP address/Destination IP address/protocols/L4 source port/L4 destination port. Notice that by default, DFW does not perform a lookup on L4 source port, but it can be configured to do so by defining a specific policy rule.

Before exploring the use case for these 2 tables, let’s first understand how DFW rules are enforced:

DFW rules are enforced in top-to-bottom ordering. Each packet is checked against the top rule in the rule table before moving down the subsequent rules in the table. The first rule in the table that matches the traffic parameters is enforced Because of this behavior, when writing DFW rules, it is always recommended to put the most granular policies at the top of the rule table. This is the best way to ensure they will be enforced before any other rule.

DFW default policy rule (the one at the bottom of the rule table) is a “catch-all” rule: packet not matching any rule above the default rule will be enforced by the default rule. After the host preparation operation, the DFW default rule is set to ‘allow’ action. The main reason is because VMware does not want to break any VM to VM communication during staging or migration phases. However, it is a best practice to change the default rule to ‘block’ action and enforce access controls through a positive control model (only traffic defined in the firewall policy is allowed onto the network).

Let’s now have a look at policy rule lookup and packet flow:

An IP packet (first packet - pkt1) that matches Rule number 2 is sent by the VM. The order of operation is the following: Lookup is performed in the connection tracker table to check if an entry for the flow already exists.

As Flow 3 is not present in the connection tracker table (i.e miss result), a lookup is performed in the rule table to identify which rule is applicable to Flow 3. The first rule that match the flow will be enforced.

Rule 2 matches for Flow 3. Action is set to ‘Allow’.

Because action is set to ‘Allow’ for Flow 3, a new entry will be created inside the connection tracker table. The packet is then transmitted properly out of DFW.

DFW policy rule lookup and packet – subsequent packets.

Subsequent packets are processed in this order:

Lookup is performed in the connection tracker table to check if an entry for the flow already exists.

An entry for Flow 3 exists in the connection tracker table => Packet is transmitted properly out of DFW

One important aspect to emphasize is that DFW fully supports vMotion (automatic vMotion with DRS or manual vMotion). The rule table and the connection tracker table always follow the VM during vMotion operation. The positive result is there is no traffic disruption during workload moves and connections initiated before vMotion remain intact after the vMotion is completed. DFW brings VM movement freedom while ensuring continuous network traffic protection.

Note: this functionality is not dependent of Controllers or NSX Manager being up and available.

NSX DFW brings a paradigm shift that was not possible before: security services are no longer dependent on the network topology. With DFW, security is completely decoupled from logical network topology.

In legacy environments, to provide security services to a server or set of servers, traffic from/to these servers must be redirected to a firewall using VLAN stitching method or L3 routing operations: traffic must go through this dedicated firewall in order to protect network traffic.

With NSX DFW, this is no longer needed as the firewall function is brought directly to the VM. Any traffic sent or received by this VM is systematically processed by the DFW. As a result, traffic protection between VMs (workload to workload) can be enforced if VMs are located on same Logical Switch (or VDS VLAN-backed port-group) or on different Logical switches.

NSX DFW architecture:The vCenter, NSX Manager and ESXi host are functioning as the 3 main components in this architecture.

NSX Manager: The NSX manager provides the single point of configuration and the REST API entry-points in a vSphere environment for NSX. The consumption of NSX can be driven directly via the NSX manager UI. In a vSphere environment this is available via the vSphere Web UI itself. Typically end-users tie in the network virtualization to their cloud management platform for deploying applications.

vCenter: VMware vCenter Server provides a centralized platform for managing your VMware vSphere environments so you can automate and deliver a virtual infrastructure with confidence.

ESXi host: VMware ESXi is the hypervisor running the virtual machines guest OS. DFW related modules:

1. vShiled-Statefull-Firewal service daemon run in the user space2. vSIP run in the kernel space.

vShiled-Statefull-Firewal: Service demon Runs constantly on the ESXi host and performs multiple tasks:

1. Interact with NSX Manager to retrieve DFW policy rules.2. Gather DFW statistics information and send them to the NSX Manager.3. Send audit logs information to the NSX Manager.4. Receive configuration from NSX manager to create (or delete) DLR Control VM,

create (or delete) ESG.5. Part of the host preparation process SSL related tasks from NSX manager

Message Bus Client: The NSX Manager communicates with the ESXi host using a secure protocol called AMQP.

“Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security”

Source: http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol.

RabbitMQ is the NSX AMQP implementation.

The vShiled-Statefull-Firewal is acting as a RabbitMQ Client in the ESXi. The vShiled-Statefull-Firewal is a user space service daemon and uses a TCP/5671 connection to the RabbitMQ server in the NSX manager. The message bus is used by the NSX Manager to send various information’s to the ESXi hosts:

Policy rules for the DFW module, controller nodes IP addresses, private key and host certificate to authenticate the communication between host and controller and requests to create/delete DLR instances.

vSIP: VMware Internetworking Service Insertion Platform. This is the distributed firewall kernel space module core component. The vSIP receives firewall rules from NSX manager (through vShiled-Statefull-Firewal) and downloads them down to each VM VMware-sfw.Note: VMware Internetworking Service-Insertion Platform is also a framework that provides the ability to dynamically introduce 3rd party and VMware’s own virtual as well as physical security and networking services into VMware virtual network.

VPXA: A vCenter agent, installed on the ESXi host when the vCenter communicates with the ESXi host for first time. With the VPXA the vCenter manage the ESXi host for vSphere related tasks. Although it is not a direct part of the DFW architecture the VPXA is being used to report the VM IP address with VMtools.

IOChains: VMware have a reserved IOchains handle packet process at the Kernel level.

Slot 0: DVFilter (Distributed Virtual Filter):Distributed Virtual Filter DVFilteris is the VMkernel between the protected vNIC at SLOT 0 associated Distributed Virtual Switch (DVS) port, and is instantiated when a virtual machine with a protected virtual NIC gets created. It monitors the incoming and outgoing traffic on the protected virtual NIC and performs stateless filtering.

Slot 1: sw-sec (Switch Security): sw-sec module learns VM’s IP and MAC address. sw-sec is critical component capture DHCP Ack and ARP broadcast message and forward this info as unicast to NSX Controller to perform the ARP suppression feature. sw-sec is the layer where NSX IP spoofgurd is implemented,

Slot-2: VMware-sfw: This is the place where DFW firewall rules are stored and enforced, VMware-sfw contains rules table and connections table.

NSX policy push process:

We create a security policy with vCenter GUI, this configuration then stored inside NSX manager. When we push the policy NSX Manager Will sends firewall rules in protobuf format to all vSphere clusters.

“Protocol Buffers are a method of serializing structured data. As such, they are useful in developing programs to communicate with each other over a wire or for storing data. The method involves an interface description language that describes the structure of some data and a program that generates from that description source code in various programming languages for generating or parsing a stream of bytes that represents the structured data”.

Source: HTTP://EN.WIKIPEDIA.ORG/WIKI/PROTOCOL_BUFFERS

All ESXi host part from vCenter cluster will receive this policy with vShiled-Statefull-Firewal daemon over the message bus.At this point vShiled-Statefull-Firewal need to parse this rules and convert them from protobuf messages into the vmkernel vSIPIOCTL format, then vSIP will apply the rules on every VM’s SLOT 2 VMware-sfw.

Note: This process flow describe “Applied To” filed in security policy is “Distributed Firewall” = rule applied everywhere.

A security administrator can create firewall rules built from vCenter objects like:

Cluster, DC, VDS port-group, Logical Switch, IPSets, Resource Pool, vAPP, VM, vNIC and Security Groups. The NSX firewall enforce point at the VMware-sfw can only understand IP address or MAC address.

In figure shows below we create a firewall rule to allow ICMP Source “Compute Cluster A” to Destination “Compute Cluster B”:

The NSX Manager will need to figure out what are the object IDs represented by “Compute Cluster A” and “Compute Cluster B” and then reveal what the IP address correspond to those VMs.

The NSX firewall rules inside ESXi host are created as VSIPIOCTL format and then applied on the VMware-sfw.

The NSX manager relies on the vCenter internal database to get object-ID/IP address mapping, we can view the data with the vCenter MOB (Managed Object Browser) Using this url: https://vCenter_IP/mob/

The NSX manager keeps this info inside his internal database.

The vCenter server represents any object with a unique id. For example “Compute Cluster A” in fact equals to domain-c25 and “Compute Cluster B” equal to domain-c26. Here is screenshots from vCenter MOB:

NSX DFW and VMtools:

The NSX (version 6.1.2) relies on the VMtools to learn the IP address of the Guest OS. The IP address learned from the VMtools is stored in the vCenter database and reported to NSX manager. We can tell if VMtools reports the IP address in VM Summery screen:

To view the vm-id we can be retrieve using vCenter at the following path: https://<vCenter server>/mob and select content -> rootFolder -> childEntity -> vmFolder.

To view the VM’s IP address go to click on the vm-id from the list above, for example vm-36 (web-sv-01a).

GuestInfo -> Net

In vCenter MOB “web-sv-01a” has Object ID: vm-36 with IP address “172.16.10.11”.

If the VMTools was stopped or removed the vCenter removes the IP address entry immediately. An update notification will send to NSX manager cause to firewall module send a list updates to all the vShiled-Statefull-Firewal processes using protobuf format. If we configure firewall rules using vCenter objects (not IP address) as show in screenshot below, there will be a match on the last firewall rule (most of the time called catch-all rule).

If this rule configure to block (as in this example) then this VM will be blocked from the network, but if this rules send the permit then the VM gets a full network access, which allows the user to bypass security policy.

Spoofguard: NSX feature that we can use to eliminate the need of VMtools to learn the VM IP address but this will be address in a different blog post.

NSX Firewall and vMotion:vMotion: enables to move a VM from one ESXi host to another while this VM is powered on and connected to network in vSphere environment.

This feature can be managed automatic by the vSphere DRS mechanism or manually by the vSphere administrator.

As we saw in the NSX DFW architecture for each VM we have two separate tables:

DFW table: Contains the firewall rules.

DFW Connection table: Contains the live active (approved) connection passing through this VM. When VM1 vMotion process starts these two tables will follow the VM1 movement over the vMotion link.

When the vMotion process completes the VM1 will land at esxcomp-01b and have same firewall rules and same connection table and as a result there’s no traffic disruption for VM1.

Note: The NSX Manager is not involved in this vMotion since we don’t use the “Applied To” feature (Explained later).

NSX Firewall Applied To:By default when we’re creating a firewall rule in NSX, the “Applied to” field is set to “Distributed Firewall”. The firewall rule will be stored in NSX manager’s database and will be applied to all VM’s vNICs, regardless of the VMs location. It’s important to mention that even when dFW rule is applied to all VMs, we still need a match on source/destination to take action on that rules.

The “Applied To” field is determined by vSphere objects: Cluster, Datacenter, vDS Distributed PortGroup, Logical Switch, Edge, Host system, Security Group, VM, or even vNIC!

When we start using the “Applied To” field, NSX Manager will map the “Applied To” object to the corresponding vSphere cluster. Only ESXi hosts in the cluster will receive this rule.

Each ESXi host that receives this rule will use vShiled-Statefull-Firewal demon to parse it and figure out which VMs need to apply it. When using the “Applied To” field, the perimeter scope limit is the vSphere Cluster.

The shows below “Rule ID” 1002 in which we’ve configured “Applied To” Distributed Firewall. (Default behavior)

When we will push this firewall rule, NSX manager will send this rule to all vSphere clusters.

As a result: all VMs will get rule id 1002 at their vNic level.

Continue our example we add another rules ID 1005 in which we use the “Applied To” on web-sv-01a and web-sv-01b. “Rule ID” 1002 stay the same with “Applied To” Distributed Firewall.

Assuming we have the following machines: web-sv-01a, web-sv-02a, app-sv-01A and sec-mgr-01a. And we’ve configured the rules above.

If web-sv-01a and app-sv-01A is part of “Computer Cluster A”, web-sv-02a is part of “Computer Cluster B” and sec-mgr-01a run in “Management Edge Cluster”.

Now when we push this policy the NSX Manager need will figure rules boundaries for each cluster.

Base this rules calculation NSX manager will push the firewall update to only to “Compute Cluster A” and “Compute Cluster B” , “Management Edge Cluster” will not receive any update because is not any vSphere object part of the “Applied To” filed. When “Compute Cluster A” received this rule firewall update vSIP kernel module will need to pars which VM will need to apply rule 1005. only web-sv-01a will

get update new rule id 1005 In addition to old rule id 1002. VM name app-sv-01a will not get any firewall rule update. When “Compute Cluster B” received the rule update all ESXi host will get the firewall update inside the cluster, vSIP demon will parse it but only the ESXi host run VM name we-sv-01b will applied rule id 1005.

“Applied To” benefits:

Reduce the amount of rules per VMware-sfw, this improves efficiency because the DFW will have less rules to evaluate for every new session.

In case of an overlap IP address within multi-tenancy environment we must use “Applied To” to distinguish between one tenant and others.

Cross cluster vMotion with “Apply To”:

When we do use the “Applied To” feature and the VM traffic perform a vMotion across clusters then the NSX Manger will be involved in the process to update destination VM cluster with the relevant firewall rules. The NSX manager must be up to complete this operation.

For example VM name web-sv-01a need to vMotion from “Compute cluster A” to “Management Edge Cluster”, vCenter will send vMotion notification to NSX Manager, as a results NSX Manager will trigger policy push to all ESXi host in “Management Edge Cluster”. web-sv-01a will get same rule before vMotion occur with just change in the domain object from “domain-c25” to “domain-c7”

If the NSX Manager is down, No update rule will be push! When VM land on destination cluster no VM specific rule apply for that VM. It’s important to note that when the NSX manager is down all existing VM’s forwarding plane with DFW rules continue to work, only “New” VMs cannot have firewall rules until NSX Manager come back.

The NSX DFW keeps the rule table as a “.dat” file at the ESXi host at the following path:

/etc/vmware/vShiled-Statefull-Firewal/vsipfw_ruleset.dat .