2. Why we come to here?Open source community growth is important
-Because we want to be free from specific vendor
products.Contribute technology for cloud scalability to accelerate
migration from customer onpremise to be on the cloud -By suggest
one of possible solution to have massive scalability.2 3. Cloudn
CloudStack based public cloud services(Compute) Currently available
in Japan and the US 2 interface for customer, Customer portal GUI
and APIs (APIs over 150 including AWS compatible) VPC type Coming
Soon in Japan33 4. Problem: VLAN ID limitation Advanced Zone oMore
functionality oNAT, FW, LB, VPN VPCIsolation required For each
guest network For each VPC tierVirtual Router Isolation Method:
VLAN oVirtual Router IsolatedVLAN IDs are limited oAdvanced
ZonePublic NetworkOnly 4096 Should be identical within a zoneGuest
NetworkVPC TierVPC Tier# of Domains is limited by VLAN Each domain
requires at least one VLAN IDVMVMVMVMVMVMVPC 4 5. VXLAN Overview
VXLAN [Virtual eXtensible Local Area Network] ObjectiveOvercome
VLAN scalability limitationNW TypeOverlay networkEnvelope typeUDP
packet (L4 packet)Standardization StatusUnder IETF standardization
processImplementationSoftware-based : Cisco Nexus Series Switches,
VMware vSphere Distributed Switch, Open vSwitch, and Linux kernel
hardware based : Arista 7150, Brocade ADX seriesCharacteristics 16M
(2^24) isolated networks On top of UDP packet Can utilize L4 port
based ECMP load balancing solutions Src UDP port is a hash of
payload MAC addr Ethernet broadcast is mapped to IP multicast L2:
IGMP (or MLD) snooping, otherwise it floods a little L3: If you
want to communicate across L3 subnet Dynamic tunnel endpoint
learning
http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-06 5
6. How traffic flows with VXLAN? Underlay Network for
VXLANethXethXvxlanMvxlanMvxlanMbrethY-MbrethY-MbrethY-MvnetvnetvnetVM1VM2VM3Host1Host2ethXHost31.
If Unicast and KVM host (Src) learned mapping between VM and KVM
host (Dst) VXLAN uses Unicast 2. If broadcast or Unicast but KVM
host (Src) doesnt know mapping VXLAN uses Multicast 6 7. Host2 VTEP
IP address resolution tableex. pingVNICapsule Dst IP addrNUnderlay
Network for VXLANPayload Dst MAC addr VM3 MAC addrHost3 IP
addrHost3 VTEP IP address resolution table VNIPayload Dst MAC
addrCapsule Dst IP addrNVM2 MAC addrHost2 IP addr VTEP : VXLAN
Tunnel End
PointethXethXvxlanNvxlanNvxlanNbridgebridgebridgevnetvnetvnetVM1VM2VM3Host1Host3Host2
Payload#ethXVXLAN headerSrc MAC addrDst MAC addrVNISrc IP addrDst
IP addrSrc UDP port number1ARP requestVM2 MAC addrBroadcastNHost2
IP addrMulticast IP addrHash(VM2 MAC addr)2ARP replyVM3 MAC addrVM2
MAC addrNHost3 IP addrHost2 IP addrHash(VM3 MAC addr)3ICMP Echo
requestVM2 MAC addrVM3 MAC addrNHost2 IP addrHost3 IP addrHash(VM2
MAC addr)4ICMP Echo replyVM3 MAC addrVM2 MAC addrNHost3 IP
addrHost2 IP addrHash(VM3 MAC addr)7 8. How VXLAN shrink broadcast
domain size? Underlay Network for VXLANUnderlay network segment 1
Underlay network segment 2 Underlay network segment 3ethY
vxlanNvxlanMbridgebridgebridgevnet(no VM associated with VNI :
N)vxlanNvnetvnetVM2VM3VM1 Host1Host2Host3Host41. Host1 contains no
VM belong to VXLAN segment N, Host 1 doesnt join the multicast
group N. 2. Since VM1 & VM2 belong to VXLAN segment N, Host 2
& 3 join same multicast group N. 3. Since Host 4 contains no VM
belong to VXLAN segment N, the path to Host4 is excluded from
multicast domain if the switch supports IGMP snooping8 9.
CloudStack Integration9 10. CloudStack KVM VLAN bridging Overview
Internet Public
NetworkethXKVMethXKVMcloudbrLcloudbrJvnetGvnetEVRvnetFbrethY-MbrethY-M
Underlay NetworkvnetGvnetHVRvnetGVMVMethY Encapsulate /
DecapsulateethY Encapsulate / DecapsulateGuest Network
vnetAVRvnetBbrethY-N vnetCVMethY.NethY.NVLAN Tunnel (VLAN ID :
N)brethY-N vnetDVM10 11. CloudStack KVM VXLAN bridging Overview
Internet Public
NetworkethXKVMethXKVMcloudbrLcloudbrJvnetGvnetEVRvnetFbrethY-MbrethY-M
Underlay NetworkvnetGvnetHVRvnetGVMVMethY Encapsulate /
DecapsulateethY Encapsulate / DecapsulateGuest Network
vnetAVRvnetBbrvxN vnetCVMvxlan-Nvxlan-N VXLAN Tunnel (VNI : N)brvxN
vnetDVM11 12. Demo video12 13. 13 14. Functional Tests14 15.
Functional test result overview We tested the basic functions
directly affected by VXLAN support. (ex. VM start/stop, Internet
connectivity, Inter-tier connectivity and live migration in
Isolated Network and VPC tier ) Case # VR/VM location 1 VR&VM
exist in a same hypervisorNetwork type isolatedTest target function
connectivity to VR connectivity to the internet VR restart
connectivity to VR after VR restart connectivity to the internet
after VR restart VM restart2 VR&VM exist in different
hypervisorsisolatedconnectivity to VR after VM restart connectivity
to the internet after VM restart connectivity to VR connectivity to
the internet VR restart connectivity to VR after VR restart
connectivity to the internet after VR restart VM restart
connectivity to VR after VM restart connectivity to the internet
after VM restart VM migration connectivity to VR after VM restart
connectivity to the internet after VM restart3 VM1&VM2 exist in
different isolated networkisolatedinter isolated network isolation4
VR&VM exist in different hypervisorsVPCconnectivity to VR
connectivity to the internet VR restart connectivity to VR after VR
restart connectivity to the internet after VR restart VM restart
connectivity to VR after VM restart connectivity to the internet
after VM restart VM migration connectivity to VR after VM restart
connectivity to the internet after VM restart5 VM1&VM2 exist in
different tier, and routing between two tier is allowed 6
VM1&VM2 exist in different tier, and routing between two tier
is deniedStep #ProcedureExpected resultResult1 ping to VR 2 ping to
the host in the internet (ex. 8.8.8.8) 3 stop VR 4 start VR 5 ping
to VR 6 ping to the host in the internet (ex. 8.8.8.8) 7 stop VM 8
start VM 9 ping to VR 10 ping to the host in the internet (ex.
8.8.8.8) 1 ping to VR 2 ping to the host in the internet (ex.
8.8.8.8) 3 stop VR 4 start VR 5 ping to VR 6 ping to the host in
the internet (ex. 8.8.8.8) 7 stop VM 8 start VM 9 ping to VR 10
ping to the host in the internet (ex. 8.8.8.8) 11 migrate VM to
another hypervisor 12 ping to VR 13 ping to the host in the
internet (ex. 8.8.8.8)ping reaches to the destination ping reaches
to the destination job finishes successfully job finishes
successfully ping reaches to the destination ping reaches to the
destination job finishes successfully job finishes successfully
ping reaches to the destination ping reaches to the destination
ping reaches to the destination ping reaches to the destination job
finishes successfully job finishes successfully ping reaches to the
destination ping reaches to the destination job finishes
successfully job finishes successfully ping reaches to the
destination ping reaches to the destination job finishes
successfully ping reaches to the destination ping reaches to the
destinationPass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass
Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass1 ping
from VM1 in one tier to the internet(ex. 8.8.8.8) 2 tcpdump from
VM2 in another tier 1 ping to VR 2 ping to the host in the internet
(ex. 8.8.8.8) 3 stop VR 4 start VR 5 ping to VR 6 ping to the host
in the internet (ex. 8.8.8.8) 7 stop VM 8 start VM 9 ping to VR 10
ping to the host in the internet (ex. 8.8.8.8) 11 migrate VM to
another hypervisor 12 ping to VR 13 ping to the host in the
internet (ex. 8.8.8.8)ping reaches to the destination ping packet
from VM1 cannot captured ping reaches to the destination ping
reaches to the destination job finishes successfully job finishes
successfully ping reaches to the destination ping reaches to the
destination job finishes successfully job finishes successfully
ping reaches to the destination ping reaches to the destination job
finishes successfully ping reaches to the destination ping reaches
to the destinationPass Pass Pass Pass Pass Pass Pass Pass Pass Pass
Pass Pass Pass Pass PassVPCinter-tier connectivity1 ping from VM1
in one tier to VM2 in another tierping reaches to the
destinationPassVPCinter-tier isolation1 ping from VM1 in one tier
to the internet(ex. 8.8.8.8) 2 tcpdump from VM2 in another tierping
reaches to the destination ping packet from VM1 cannot capturedPass
Pass15 16. VXLAN plugin restriction VXLAN is not available for
Public Network, Storage Network, and Management Network These
networks do not consume many VLAN IDs. KVM is the only supported
hypervisor Maybe we can add LXC support Mapping between VNI and
multicast address is hardcoded. multicastAddress= "239.$(( ($vni
>> 16) % 256 )).$(( ($vni>> 8) % 256 )).$(( $vni % 256
))"16 17. Resources CloudStack Plugin guide for VXLAN
http://jenkins.buildacloud.org/job/build-docs-vxlanmaster/lastSuccessfulBuild/artifact/Apache_CloudStack
-4.3.0-CloudStack_VXLAN_Guide-en-US.pdf Design Doc
https://cwiki.apache.org/confluence/display/CLOUDSTA
CK/Linux+native+VXLAN+support+on+KVM+hypervisor JIRA ticket
https://issues.apache.org/jira/browse/CLOUDSTACK2328Bug report,
suggestions, any feedbacks are welcome! 17 18. Wrap up VXLAN
integration for CloudStack we contributed is merged to CloudStack
4.3 branch. We confirmed basic functions work in Isolated Network
and VPC Tier. Please evaluate VXLAN integration, any bug report,
suggestions, feedbacks are welcome!Special Thanks: Toshiaki
HatanoNTT Communications Corp.Junji ArakawaNTT Communications
Corp.Chris CameronVerio Inc. 18 19. Appendix19 20. NVGRE Overview
NVGRE [Network Virtualization using Generic Routing Encapsulation]
ObjectiveOvercome VLAN scalability limitationNW TypeOverlay
networkEnvelop typeExtended GRE packet (L3 packet)Standardization
StatusUnder IETF standardization processImplementationMicrosoft
Hyper-V 2012 R2, Intel Ethernet Switch FM6000 SeriesCharacteristics
16M (2^24) isolated networks Extended GRE packet Utilize GRE
packets key option field as VSID and flow-ID. ECMP load balancing
solutions must be aware of NVGRE flow-IDSpec leaves Ethernet
broadcast undefined. Mapping to IP multicast is suggested.
Multicast network operation is required.
http://tools.ietf.org/html/draft-sridharan-virtualization-nvgre-03
20 21. STT Overview STT [Stateless Transport Tunnel]
ObjectiveOvercome VLAN scalability limitationNW TypeOverlay
networkEnvelop typeTCP like original L3 packet (protocol type is
same as TCP. Pretends TCP packet.)Standardization StatusUnder IETF
standardization processImplementationVMware NSX (formerly Nicira
NVP)Characteristics 2^64 isolated networks TCP-like header + STT
header Can utilize NICs TSO feature FW/router may drop STT packets
by statefull inspection.Spec leaves Ethernet broadcast undefined.
Mapping to IP multicast is suggested. Multicast network operation
is required. http://tools.ietf.org/html/draft-davie-stt-04 21 22.
Solutions comparison VXLANNVGRESTTOverhead Header Size (50 bytes)
(42 bytes)(76 bytes)NIC Offloading (Special NIC is required)
(Special NIC is required) (able to utilize normal TSO)Existing
Assets Fitness (MTU may need to be adjusted) (MTU may need to be
adjusted) (FW/router may drop STT packets)Interoperability (Spec
left only minor undefined points)Tunnel endpoint address resolution
is undefined.Tunnel endpoint address resolution is
undefined.)Ethernet Broadcast (Mapping to IP Multicast) (Mapping to
IP Multicast (suggestion)) (Mapping to IP Multicast
(suggestion))ECMP (Able to utilize L2 fabric's L4 port base
balancing) (L2 fabric must aware NVGRE flow-ID to balance) (Able to
utilize L2 fabrics L4 port base balancing)Multicast
OperationRequiredRequired (depends on implementation)Required
(depends on implementation)Supporting VendorsVMware/Citrix/Red Hat/
Cisco/Intel/Broadcom/AristaMicrosoft/Arista/Emulex/
Dell/HPVMware(formerly Nicira)Linux Integration (kernel 3.7 or
later) (no implementation exists) (Niciras Open vSwitch is
required)22 23. VXLAN Terminology VXLAN Virtual eXtensible Local
Area Network VXLAN Segment VXLAN Layer 2 overlay network over which
VMs communicate VTEP VXLAN Tunnel End Point an entity which
originates and/or terminates VXLAN tunnels VNI VXLAN Network
Identifier (or VXLAN Segment ID) VXLAN Gateway an entity which
forwards traffic between VXLAN and non-VXLAN environments23 24.
VXLAN segment format 0Outer Ethernet Header: - FCS is newly
calculated, inner FCS is omitted. Outer IP header - If Inner dst
MAC is unicast MAC and local VTEP knows remote VTEP for the MAC
address, dst IP set to remote VTEPs IP address. - If not, packet
will be sent out to multicast group associated with VNI. - The VTEP
will use (*,G) joins. Outer UDP header - Source port: It is
recommended to be calculated from inner Ether Header, for ECMP
purpose. - Destination port: 4789 - Checksum: SHOULD be 0. Or
correct value VXLAN header - VNI has a 24-bit field From current
draft: (IPv4 case)
http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-062013-04-17:
IANA assigned udp/4789 for VXLAN port
http://www.iana.org/assignments/service-names-portnumbers/service-names-port-numbers.xml1
2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
Outer Ethernet Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Outer Destination MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Outer Destination MAC Address | Outer Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Outer Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|OptnlEthtype = C-Tag 802.1Q | Outer.VLAN Tag Information |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Ethertype = 0x0800 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Outer IPv4
Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Time to Live |Protocl=17(UDP)| Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Outer Source IPv4 Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Outer Destination IPv4 Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Outer UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Source Port = xxxx | Dest Port = VXLAN Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
UDP Length | UDP Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
VXLAN Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|R|R|R|I|R|R|R| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
VXLAN Network Identifier (VNI) | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
(continuing to Inner Ethernet header, abbrev.)24 25. CloudStack
Integration -NETWORK CONCEPTS IN CLOUDSTACK-25 26. Network concepts
in CloudStack Guest Network Virtual network VMs are connected
Isolated each other There are two type of Guest networks Isolated
network Traffic from VMs go out to Public Network through Virtual
Router. VR is created per Isolated network. Shared network Traffic
from VM go out directly. VPC In VPC, Virtual Router can have
multiple Isolated Networks (It is called VPC tier). In VPC, routing
between tier is configurable. 26 27. Network concepts in CloudStack
(cont.) Isolation method Method to isolate Guest Networks each
other. Typical isolation method is VLAN. VXLAN need to be
implemented as isolation method. Physical Network Underlay network
of Guest network Isolation method of guest network is specified
while defining physical network createPhysicalNetwork API has
isolationmethods parameter.
http://cloudstack.apache.org/docs/api/apidocs4.2/root_admin/createPhysicalNetwork.html27
28. Advanced Zone Network Overview Internet Public NetworkKVMVR for
VPC* VR for VPC will be created per VPCKVMVRVPC
tierVMVMVMVMVMVMGuest NetworkVM User can create multiple VPCs
(depending on settings, up to 20 by default) VPC can have multiple
tiers (depending on settings, up to 3 by default) 28 29. Step by
Step VXLAN base zone setup procedure29 30. VXLAN base zone setup
procedure(1) - login30 31. VXLAN base zone setup procedure(2) -
welcome wizardClick skip bottom31 32. VXLAN base zone setup
procedure(3) - infrastructure tabClick Infrastructure tab32 33.
VXLAN base zone setup procedure(4)33 34. VXLAN base zone setup
procedure(5)Select Advanced type34 35. VXLAN base zone setup
procedure(6)Fill zone wizard. KVM is the only supported
hypervisor.35 36. VXLAN base zone setup procedure(7)Management
network/Public network/Storage network are not supported by
VXLANSelect VLANSelect VXLAN Guest network is supported by VXLANSet
underlay network I/F name (ex. eth0) to traffic type36 37. VXLAN
base zone setup procedure(8)Fill zone wizard. There is no VXLAN
specific concern.37 38. VXLAN base zone setup procedure(9)Fill zone
wizard. There is no VXLAN specific concern.38 39. VXLAN base zone
setup procedure(10)You can use 0-16777215 as VNI39 40. VXLAN base
zone setup procedure(11)Fill zone wizard. There is no VXLAN
specific concern.40 41. VXLAN base zone setup procedure(12)Fill
zone wizard. There is no VXLAN specific concern.41 42. VXLAN base
zone setup procedure(13)Fill zone wizard. There is no VXLAN
specific concern.42 43. VXLAN base zone setup procedure(14)Fill
zone wizard. There is no VXLAN specific concern.43 44. VXLAN base
zone setup procedure(15)Fill zone wizard. There is no VXLAN
specific concern.44 45. VXLAN base zone setup procedure(16)Click
Launch zone button45 46. VXLAN base zone setup procedure(17)46 47.
VXLAN base zone setup procedure(18)Click Yes button47 48. VXLAN
base zone setup procedure(19)Add Instance wizard. There is no VXLAN
specific concern.48 49. VXLAN base zone setup procedure(20)Add
Instance wizard. There is no VXLAN specific concern.49 50. VXLAN
base zone setup procedure(21)Add Instance wizard. There is no VXLAN
specific concern.50 51. VXLAN base zone setup procedure(22)Add
Instance wizard. There is no VXLAN specific concern.51 52. VXLAN
base zone setup procedure(23)Add Instance wizard. There is no VXLAN
specific concern.52 53. VXLAN base zone setup procedure(24)Add
Instance wizard. There is no VXLAN specific concern.53 54. VXLAN
base zone setup procedure(25)Add Instance wizard. There is no VXLAN
specific concern.54 55. VXLAN base zone setup procedure(26)55 56.
VNI validationIf your VNI range is invalid, This error will pop
up.56
