Modern Datacenter - Architecture Patterns - Infrastructure · Cloud Platform Integration Framework i Modern Datacenter - Architecture Patterns - Infrastructure Prepared by Kunal Bhatia,

Cloud Platform Integration Framework

i

Modern Datacenter - Architecture Patterns - Infrastructure

Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow

“Hybrid Networking”

Hybrid Networking

25-Mar-15

Version 1.0

Prepared by

Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow

Modern Datacenter

Architecture Patterns -

Infrastructure


ii




Revision and Signoff Sheet

Change Record

Date Author Version Change Reference

11/26/2014 Microsoft 1.0 Release


iii




Table of Contents

1 Overview .................................................................................................................................................. 4

1.1 Technology Background ........................................................................................................................... 4

1.2 Pattern Requirements................................................................................................................................ 6

1.2.1 Connecting On-Premises Networks to Azure ........................................................... 7

1.2.2 Extending Hybrid Networks Across Azure Virtual Networks .................................... 9

1.2.3 Extending Hybrid Networks Across Azure Datacenter Regions ............................... 9

1.2.4 Extending Hybrid Networks Across Azure Subscriptions ........................................ 10

1.2.5 Providing developers remote access to the hybrid network ................................... 10

1.2.6 Additional recommended requirements .................................................................. 10

2 Architecture Pattern ........................................................................................................................... 11

2.1 Pattern Dependencies ............................................................................................................................. 14

2.2 Azure Services .......................................................................................................................................... 14

2.3 Pattern Considerations ............................................................................................................................ 14

3 Interfaces and End Points ................................................................................................................ 15

4 Availability and Resiliency ................................................................................................................ 16

5 Scale and Performance ..................................................................................................................... 17

6 Cost .......................................................................................................................................................... 17

6.1 Cost Factors .............................................................................................................................................. 18

6.2 Cost Drivers .............................................................................................................................................. 18

7 Operations ............................................................................................................................................. 20

8 Architecture Anti-Patterns ............................................................................................................... 23


Page 4




1 Overview

The Hybrid Networking design pattern details the Azure features and services required to deliver

network functionality that can provide predictable performance and high availability across

geographic boundaries. A full list of Microsoft Azure regions and the services available within

each is provided within the Microsoft Azure documentation site. This document provides an

overview of Microsoft Azure networking capabilities for hybrid environments. Microsoft Azure

Virtual Networking enables you to create logically isolated networks in Azure and securely

connect them to your on-premises datacenter over the Internet or using a private network

connection. In addition, individual client machines can connect to an isolated Azure network

using an IPsec VPN connection.

The Hybrid Networking architecture pattern includes the following focus areas:

Connecting on premises networks to Azure

Extending Azure virtual networks across regions

Extending Azure virtual networks between subscriptions

Providing developers remote network access

1.1 Technology Background

In order to understand the Hybrid Networking Architectural Pattern, a basic knowledge of

Microsoft Azure networking is required. The following table defines terms that will be used in

the Hybrid Networking architectural pattern.

Technology Description

Virtual

Network

An address space that can be established in Azure that can be subdivided to logically

separate virtual machines and cloud services.

Local

Network Site

One or more address spaces that represent on-premises routable subnets that you

want to communicate with across a gateway

Subnet A defined set of addresses from the Azure virtual network address pool that you want

to logically separate for other addresses within the same network ID

http://azure.microsoft.com/en-us/regions/#services

http://msdn.microsoft.com/en-us/library/azure/jj156007.aspx






Page 5




DNS Server Server that provides domain name resolution to machines in Azure virtual networks.

One or more DNS servers can be added to an Azure subscription and then assigned to

a virtual network. The DNS server will be assigned to the virtual machines on the virtual

network through DHCP during the boot process.

Azure

Gateway

This is a virtual VPN gateway device that is created on request in an Azure virtual

network. It is designed to support routable connectivity between virtual networks, on

premises locations, and single remote access VPN client machines. The gateway comes

in three configurations: Dynamic, Static and Dedicated. Gateway uses IPSec and pre-

shared keys to establish connectivity and encrypt data in motion

On Premises

Gateway

This is a physical gateway that is established at a customer on premises location. In

addition, this must be a supported device to be used with Microsoft Azure.

vNet-to-vNet

connectivity

This is the capability to have one or more virtual networks in a routable configuration.

This is accomplished using IPSec based gateways and pre-shared keys. To accomplish

vNet-vNet routing across the Azure backbone, you must configure vNet-to-vNet multi-

hop routing.

vNet Multi-

hop Routing

Virtual Network multi-hop routing requires the Azure Gateway device to be configured

with vNet-to-Local Network Site routing where the local network site definition has all

of the routable subnets on the other side of the connection.

Daisy Chain A virtual network multi-hop routing configuration where the virtual networks connect

in an end-to-end fashion

Hub and

Spoke

A virtual network multi-hop routing configuration where the virtual networks connect

to a central hub virtual network and multiple hubs connect together for expansion.

vNet1 vNet2 vNet3

vNet-vNet vNet-vNet

vNet4

vNet-vNet

vNet5

vNet-vNet

Hub

vNet2 vNet3 vNet4 vNet5

Hub2

vNet-vNet

vNet9 vNet X






http://msdn.microsoft.com/library/azure/jj156075.aspx

http://msdn.microsoft.com/en-us/library/azure/dn690122.aspx



Page 6




Full Mesh A Virtual network routing configuration where the virtual networks connect to every other virtual network over a single hop connection. While this allows routing access to every subnet, it limits scalability due to the virtual network connection limit for an Azure gateway

Site-to-Site

VPN

Azure Virtual Network to on premises connection utilizing Azure Dynamic or Static

Routing Gateway and On Premises Gateway to establish a routable communications

path between subnets on both sides. The connection is an IPsec tunnel over the

Internet using pre-shared keys to establish the connection and perform encryption.

Point-to-Site

VPN

Client computer to Azure virtual network connection utilizing an Azure Gateway and a

VPN client. The connection is established as an IPSec VPN using certificates.

ExpressRoute Azure Virtual Network to on premises connection utilizing an Azure Dedicated Routing

Gateway and a private network path to the on premises environment. No traffic flows

across the Internet. Implementation of the ExpressRoute connection can be over a

multi-protocol label switching (MPLS) circuit or dedicated connection.

1.2 Pattern Requirements

The following are general requirements of the Hybrid Networking Infrastructure architectural

pattern:

An active Microsoft Azure account

An active Microsoft Azure Subscription within the account

An active Internet connection to access the Microsoft Azure portal or the service

management API to configure Microsoft Azure subscription networking

IP address pool for establishing Virtual Networks in Microsoft Azure that is non-

overlapping with on premises

IP Subnets from the IP Address pool for point-to-point connectivity: A non-overlapping

IP address block (/28 subnet) to configure IP addressing for the Site-to-Site or

ExpressRoute circuits

vNet1 vNet2

vNet3

vNet-vNet

vNet4

vNet-vNet

http://azure.microsoft.com/en-us/documentation/articles/azure-subscription-service-limits/#networkinglimits







Page 7




Once these requirements are satisfied, the following actions can be performed to provide

network access to the organization’s Azure subscription:

1. Connecting on-premises networks to Azure

2. Extending on-premises networks across Azure Virtual Networks

3. Extending networks across Azure datacenter regions

4. Extending networks across Azure subscriptions

5. Providing developers remote access to the Azure network

The following sections will describe each of the scenarios outlined above.

1.2.1 Connecting On-Premises Networks to Azure

Establishing a hybrid networking connection to Microsoft Azure can be accomplished using two

approaches. The first involves utilizing an IPsec based Virtual Private Network (VPN) between

Microsoft Azure and an on-premises network infrastructure and is often referred to as a “Site-to-

Site” (S2S) connection. This approach uses the Internet as the communication medium and

therefore is subject to random changes in latency and bandwidth availability. The second

approach is to use a dedicated network connection between Microsoft Azure and an on-

premises network infrastructure and is referred to as an ExpressRoute connection. This approach

uses dedicated private circuits or private network clouds for the connection. Unlike site-to-site

connections, these private network paths do not utilize the Internet and are not subject to the

random changes in latency and bandwidth of an Internet based connection.



Page 8




IPsec VPN Connectivity

Site-to-Site communications utilize an IPsec tunnel established between two gateway devices

(one hosted within Azure and one on-premises). The IPsec tunnel uses pre-shared key approach

for establishing the connection and encrypting the data flowing over the connection. This allows

data to flow across the Internet in a secure manner. The gateway within Azure is a virtual

gateway implemented as a small virtual machine. When the gateway is created, you must select

either a dynamic routing or a static routing gateway. Note that the on-premises gateway device

must be implemented using either a supported model or compatible with a supported model.

Details about the supported on premises gateways can be found in the article About VPN

Devices for Virtual Network.

To establish the Site-to-Site connection between the two gateways, you must provide a pre-

shared key and configure both gateways to utilize the key. The pre-shared key can be generated

by the Azure portal, or it can be created and set on the connections using PowerShell. A single

gateway may now establish multiple site-to-site connections to different on-premises locations.

This enables redundancy for site-to-site communications. In this configuration, a separate on-

premises gateway is required to establish the alternate connection.

ExpressRoute Connectivity

ExpressRoute allows direct site-to-site connectivity between an on-premises datacenter and

Microsoft Azure, thus bypassing traditional, shared network connections through the Internet. It

allows for flexibility of choice in the network performance desired. ExpressRoute allows two

types of private connectivity to Azure - Through an ExpressRoute location (Internet Exchange

Provider facility or IXP) or directly from an existing MPLS VPN network provided by a network

service provider (NSP).

The following summarizes the requirements of the Hybrid Networking Infrastructure

architectural pattern for connecting on-premises networks to Microsoft Azure:

S2S Internet connectivity

Azure routing gateway (Static or dynamic)

On-premises routing gateway device that is on the Microsoft approved list of

gateway devices for Site-to-Site connections.

Potential secondary on-premises routing devices if the pattern requires multiple on-

premises site connections

Pre-shared key for each connection path for establishing connection and encryption

of data





Page 9




ExpressRoute Connectivity

Routing gateway device that is on the Microsoft approved list of gateway devices for

ExpressRoute connections.

An existing business relationship with an Internet Exchange Provider (IXP) or a

Network Service Provider (NSP) who are on Microsoft's supported list of vendors for

facilitating ExpressRoute connectivity

Connectivity to an IXP: Network connectivity from customer datacenter to a

supported IXP Datacenter (e.g. Equinix), along with an Ethernet cross-connect to IXP's

Layer 2 peering infrastructure

Connectivity to an NSP: MPLS Layer 3 VPN connectivity through at least one of

customer sites/datacenters to a supported NSP network (e.g. - AT&T NetBond)

1.2.2 Extending Hybrid Networks Across Azure Virtual Networks

Hybrid networks can be established from an on-premises environment to Azure virtual networks.

By default, these connections only allow the on-premises network to talk to the virtual network

that the connection is established with (the first hop virtual network). In order to extend the

connection from the first hop virtual network to additional virtual networks in Azure, you make

configuration changes to establish multi-hop routing.

Establishing a hybrid network connection across Azure Virtual Networks requires an Azure

gateway to be established and connected to the on-premises gateway. In the Site-to-Site

scenario, an Azure static or dynamic routing gateway is required. If multi-hop virtual network

routing is desired, then the dynamic routing Azure gateways must be utilized. In the

ExpressRoute scenario, an Azure dedicated gateway is required. It is not possible to combine

multi-hop routing and ExpressRoute connections currently. To obtain access to multiple Azure

virtual networks in the ExpressRoute scenario, a separate ExpressRoute connection must be

established with each virtual network.

1.2.3 Extending Hybrid Networks Across Azure Datacenter Regions

Hybrid network connections can span Azure datacenter regions to allow traffic to route between

regions. This can be accomplished either using virtual networks in different regions connected

using IPsec tunnels or ExpressRoute connections independently established to virtual networks

in separate regions. These two methods cannot be combined.


Page 10




Establishing a hybrid network connection to Azure Virtual Networks in different regions requires

an Azure gateway be established in each virtual network. These gateways will then connect to

each other to establish the link between Azure Virtual Networks. An on-premises gateway is

required to establish a hybrid network connection to an on-premises datacenter. In the Site-to-

Site scenario, an Azure dynamic routing gateway and multi-hop routing is required for cross

region connections. In the ExpressRoute scenario, an Azure dedicated gateway is required. For

ExpressRoute to obtain access to Azure virtual networks in different regions, a separate

ExpressRoute connection must be established with each virtual network.

1.2.4 Extending Hybrid Networks Across Azure Subscriptions

Hybrid network connections can span Azure subscriptions to allow traffic to route between

virtual networks in each subscription. This can only be accomplished using multi-hop routing of

virtual networks in different subscriptions connected using IPsec tunnels. ExpressRoute

connections currently are not supported across subscriptions.

Establishing a hybrid network connection to Azure Virtual Networks in different subscriptions

requires an Azure Gateway be established in each virtual network and connected. In the Site-to-

Site scenario, an Azure dynamic routing gateway and multi-hop routing is required.

1.2.5 Providing developers remote access to the hybrid network

In scenarios where a user does not have access to the corporate network where a Site-to-Site or

ExpressRoute hybrid network connection can provide routable access to Azure virtual networks,

it is possible to establish a temporary connection to the hybrid network by creating a Point-to-

Site (P2S) connection directly from the a Windows-based system to an Azure virtual network. To

accomplish this, an Azure virtual network must be configured to support Point-to-Site

connections (remote access VPN client connections), an address range must be established for

the machines connecting, a certificate must be created and uploaded for IPsec encryption, and

the client machine must download a VPN client to establish the connection.

1.2.6 Additional recommended requirements

The following are not required to establish hybrid networking, but are recommended for an

operational environment:

Accessible DNS servers that can be specified in Microsoft Azure for automatic

assignment to virtual machines placed on virtual networks




Page 11




An Enterprise Certificate Authority to generate trusted certificates for Point-to-Site

connections

2 Architecture Pattern

The hybrid networking architecture pattern is complex due to the possible number of scenarios

that can be created. This architectural pattern will focus on the following four scenarios:

Site-to-Site hybrid networking with Multi-hop virtual network routing within a single

subscription and region

Site-to-Site hybrid networking with multi-hop virtual network routing across

subscriptions and regions

ExpressRoute hybrid networking using MPLS connectivity

ExpressRoute hybrid networking using IXP connectivity

Site-to-Site Hybrid Networking with Multi-Hop Virtual Network Routing within a Single

Subscription and Region

Azure Subscription w/Multihop vNet routing

Virtual NetworkVirtual Network Virtual Network

Site-to-Site

Dynamic Routing Gateway

Internet

Enterprise Datacenter

On Premises Gateway


On Premises Gateway


Page 12




Site-to-Site Hybrid Networking with Multi-Hop Virtual Network Routing across

Subscriptions and Regions

Azure Subscription A East US Region

Virtual Network Virtual Network

Site-to-Site


Internet


On Premises Gateway


On Premises Gateway

Azure Subscription B West US Region



Azure Subscription A West Europe Region



vNet-vNet across Subscription

and region


Page 13




ExpressRoute Hybrid Networking using MPLS connectivity

This option uses a Network Service Provider and is for customers who already use MPLS VPN

services through a supported telecommunications provider. In this option, Azure becomes

another "branch office" extension to the customer's existing MPLS network. This option is

relatively simple to deploy as the customer's carrier is responsible for configuring and managing

the ExpressRoute connection. This option is suitable for customers who already use MPLS VPN

services through a supported Telco. In this option, Azure becomes another "branch office"

extension to the customer's existing MPLS network. This option is relatively simple to deploy as

the customer's carrier is responsible for configuring and managing the ExpressRoute connection.

This is illustrated in the diagram below:

MPLS VPN Network

ExpressRoute Hybrid Networking using IXP Connectivity

This option uses a Network Service Provider and allows customers to directly peer with Azure by

collocating customer premises equipment at an IXP (e.g. - Equinix). This option provides more

granular routing control to the customer and also enables higher bandwidth connections to

Azure. It is suitable for enterprises that require a high-throughput connection to Azure or run


Page 14




low latency hybrid workloads, Media services, storage/backup in Azure. This option requires the

customer to have in-house capabilities for configuring and managing this network

infrastructure. This option is illustrated in the diagram below:

Customer DatacenterExpressRoute Partner Location

2.1 Pattern Dependencies

Being an infrastructure pattern, there are no dependencies on other patterns, however

dependencies with the customer’s on-premises network environment and the network service

provider used within the organization.

2.2 Azure Services

The following Azure Services are required for this architecture pattern as outlined above:

Virtual Networks

Virtual Network Gateways

2.3 Pattern Considerations

For each component of Microsoft Azure, a series of subscription and service limits are defined

by the service. These limits are subject to change and are published at the following link.

Microsoft Azure limits fall into the categories of default and maximum limits. Default limits are

those which exist on every Azure subscription and can be increased through a request to

Microsoft support whereas maximum limits define the upper boundary of a given service or

capability within Azure. Limits can be raised by contacting Microsoft support as outlined

http://azure.microsoft.com/en-us/documentation/articles/azure-subscription-service-limits/


Page 15




through the Azure portal as outlined in this article, however the request cannot exceed the

maximum limits outlined for each Azure service listed above. The following considerations must

be taken into account during architectural planning of the Hybrid Networking pattern:

Currently there are hard and soft limits on the number of Virtual Networks per Azure

subscription. The default limit is 10, while the maximum limit is 100.

Azure currently supports up to 3000 customer routes per ExpressRoute BGP peering.

Quality of Service (QoS):

- Customer traffic inbound to Azure: The NSP may provide QoS treatment of traffic

traversing over its network towards Azure; this is dependent on the service Offering

procured from the carrier.

- Customer traffic outbound from Azure: This traffic is provided default QoS treatment

as Microsoft routers today do not mark DSCP/CoS (Class of Service) on this traffic.

A virtual network in Azure can have a maximum of a single routing gateway.

A routing gateway can only be in a single state at a time (static or dynamic).

A dynamic routing gateway in Azure has a maximum number of connections (site-to-site

or vNet-to-vNet) as identified in the Azure Subscription and Service Limits, Quotas, and

Constraints product documentation.

A Point-to-Site VPN requires dynamic routing gateway.

A Site-to-Site VPN requires a dynamic or static routing gateway.

An ExpressRoute connection requires a dedicated routing gateway.

By default a vNet-to-vNet connection can only route across a single vNet.

ExpressRoute connection cannot utilize vNet-to-vNet routing to pass traffic across the

Azure backbone.

3 Interfaces and End Points

As outlined earlier, the Hybrid Networking pattern utilizes different types of end points to

establish connectivity between the on-premises network and the Azure network infrastructure

for each subscription. For the purposes of hybrid networking, these endpoints can be divided

into physical and logical end points.

http://azure.microsoft.com/blog/2014/06/04/azure-limits-quotas-increase-requests/

http://en.wikipedia.org/wiki/Border_Gateway_Protocol




Page 16




The following table outlines these end points by type:

End Point Type Description

Physical

IXP based connectivity - Ethernet cross connect setup between customer &

Microsoft equipment collocated in IXP facility

NSP based connectivity - MPLS based connectivity to Azure is extended from

within existing customer WAN

Logical

BGP session # 1 for connectivity to Azure Compute services within a Vnet - VM's

(IaaS) and Cloud Services (PaaS)

BGP session # 2 for connectivity to Azure Public Services - SQL, Storage, Media

Services etc

4 Availability and Resiliency

Microsoft provides clearly defined Service Level Agreements (SLAs) for each service provided

within Azure. Each architectural pattern is comprised of one or more Azure services and details

about each individual Azure service can be found on the Microsoft Azure Service Level

Agreement website. For the Hybrid Networking architectural pattern, the Azure services

required carry the following SLAs:

Azure Service Service Level Agreement

Virtual Network Gateway 99.9%

Virtual Networks 99.9%

ExpressRoute 99.9%

The composite Service Level Agreement (SLA) of the Hybrid Networking architectural pattern is

99.9% based on the services outlined above. The details behind the SLA can differ based on the

http://azure.microsoft.com/en-us/support/legal/sla/



Page 17




type of connectivity utilized. For ExpressRoute connections, the composite SLA is comprised of

the ExpressRoute SLA offered by Microsoft combined with the underlying infrastructure which is

provided by a third party service provider. For Site-to-Site connections, the composite SLA is

comprised of the VPN SLA offered by Microsoft is around the Virtual Gateway and not the

connection itself.

5 Scale and Performance

A key consideration for the performance and scale of the Hybrid Networking architectural

pattern is highly dependent on the network connection itself, which is defined by the type of

connection used. In general, ExpressRoute provides higher scale and performance (higher

throughput, lower latency, lower jitter) in comparison to an Internet-based IPSec VPN

connection (site-to-site or point-to-site). The differences in the type of connection and

expected speeds are outlined in the table below:

Type of Connection Speed

Network Service Provider 10Mbps, 50Mbps, 100Mbps, 500Mbps & 1Gbps

Exchange Provider 200Mbps, 500Mbps, 1Gbps & 10Gbps

Site-to-Site VPN Default: 100Mbps or High Performance (200Mbps)

Point-to-Site VPN 100Mbps

6 Cost

An important consideration when deploying any Solution within Microsoft Azure is the cost of

ownership. Costs related to on-premises cloud environments typically consist of up-front

investments in compute, storage and network resources, while costs related to public cloud

environments such as Azure are based on the granular consumption of the services and

resources found within them. Costs can be broken down into two main categories: cost factors

and cost drivers. Cost factors consist of the specific Microsoft Azure services which have a unit

http://go.microsoft.com/fwlink/?linkid=391425&clcid=0x409



Page 18




consumption cost and are required to compose a given architectural pattern. Cost drivers are a

series of configuration decisions for these services within a given architectural pattern that can

increase or decrease costs. Microsoft Azure costs are divided by the specific service or capability

hosted within Azure and continually updated to keep pace with the market demand. Costs for

each service are published publicly on the Microsoft Azure pricing calculator. It is recommended

that costs be reviewed regularly during the design, implementation and operation of this and

other architectural patterns.

6.1 Cost Factors

Cost factors within the Hybrid Networking architectural pattern include the choice in

ExpressRoute providers and connectivity options (Site-to-Site vs. ExpressRoute). For Site-to-Site

connections (VPN) there is a minimal charge for the Gateway itself. The connection runs over

the open Internet. Customers can establish the ExpressRoute Service in two ways – Exchange

Provider or Network Service Provider. The associated charges for both these options differ

accordingly. ExpressRoute service using an Exchange Provider is charged based on a monthly

dual-port fee for two physical Ethernet ports (on two routers). Inbound transfer (e.g. - Customer

to Azure) is included. A finite outbound transfer (e.g. - Azure to Customer) is included with the

port speed selected. Any outbound transfer beyond that is charged based on bandwidth

consumed. Alternatively, ExpressRoute using a Network Service Provider is charged based on a

monthly dual-port fee for two physical Ethernet ports (on two routers), and there are no

additional costs related to all inbound and outbound data transfer.

6.2 Cost Drivers

As stated earlier, cost drivers consist of the configurable options of the Azure services required

when implementing an architectural pattern which can impact the overall cost of the Solution.

These configuration choices can have both a positive or negative impact on the cost of

ownership of a given Solution within Azure, however they may also potentially impact the

overall performance and availability of the Solution depending on the selections made by the

organization. Cost drivers can be categorized by their level of impact (high, medium and low).

http://azure.microsoft.com/en-us/pricing/calculator/

https://azure.microsoft.com/en-us/pricing/details/expressroute/

https://azure.microsoft.com/en-us/pricing/details/expressroute/

https://azure.microsoft.com/en-us/pricing/details/expressroute/#network

https://azure.microsoft.com/en-us/pricing/details/expressroute/#network


Page 19




Cost drivers for the Hybrid Network architectural pattern are summarized in the table below.

Level of

Impact

Cost Driver Description

High

Connectivity from customer

location to Azure

While considering the cost of connecting to Azure

using ExpressRoute, it must be kept in mind that

Network service providers/Exchange providers may

levy additional charges/fees for their services in

addition to the ExpressRoute charges levied by

Microsoft.

Cost of egress of data using

IPsec VPN connection to

Azure

An IPsec VPN based connection to Azure incurs

charges only for egress of customer data outbound

from Azure. There are no cost impacts for ingress

(inbound) of data into Azure, irrespective of the

amount of data transferred.

Medium

Data transfer over

ExpressRoute within

US/Europe Azure regions

Once a customer has established an ExpressRoute

connection to a Microsoft location within US/Europe,

data can be sent to/from any other Azure region

within that continent. For example, a US customer can

send or receive data to/from any Azure region within

US without additional charges from Microsoft. This

however, is not applicable to Asia-Pacific

Data transfer over vNet-to-

vNet connection across Azure

zones

In a VNet-to-VNet Scenario, an egress charge will be

assessed if those VNet’s are in different Azure zones.

Low Cancellation of ExpressRoute

Service

Prorated billing is applied in case the ExpressRoute

service is cancelled during a month such that the

billing would only be for the hours used and the

actual data transfer incurred.


Page 20




7 Operations

Cloud Platform Integration Framework (CPIF) extends the operational and management

functions of Microsoft Azure, System Center and Windows Server to support managed cloud

workloads. As outlined in CPIF, Microsoft Azure architectural patterns support deployment,

business continuity and disaster recovery, monitoring and maintenance as part of the operations

of the Hybrid Networking architectural pattern.

Deployment of this pattern can be achieved through multiple methods. In addition to using the

Management portal, ExpressRoute deployment currently requires PowerShell-based

configuration tasks which need to be performed through the Service Management REST API.

The following three broad steps that need to be carried out in a proper order for a circuit to be

fully provisioned. Detailed configuration steps can be found in the article Configure an

ExpressRoute Connection through an Exchange Provider.

1. Creation of ExpressRoute circuits

2. Route configuration

3. Linking Virtual network to ExpressRoute circuit

http://msdn.microsoft.com/library/azure/dn606306.aspx

http://msdn.microsoft.com/library/azure/dn606306.aspx


Page 21




These steps are illustrated in the diagram below:

Figure 1: ExpressRoute Deployment Workflow for Exchange Provider based connection


Page 22




Azure Virtual Networks can also be created through the portal and Azure PowerShell. When

dealing with VNet-to-VNet configuration and multi-site VPN configurations, customers will be

required to author a .netcfg xml configuration file for deployment.

Monitoring of this pattern and associated resources depends on the connection type. An

ExpressRoute circuit can be managed and monitored through the Service Provider portal. A Site-

to-Site VPN connection can be managed and monitored through the Service Portal. From this

view a customer will only see the External interfaces from a connection perspective. With a Site-

to-Site VPN connection the only side which is visible in the portal is the Azure side of the

connection. In order to look at verbose logging of the connection access to the on-premises

router will be required. For VNet-to-VNet monitoring, the gateways and the connection can be

monitored solely in the Azure Portal.

Depending on the nature of the task involved, maintenance of an ExpressRoute connection can

be performed using a combination of Azure Management Portal, Service Provider portal and/or

PowerShell through the Service Management API.

From a business continuity and disaster recovery perspective, the availability constructs outlined

earlier should help make sure that solutions deployed using the Hybrid Networking architectural

pattern are designed to support continuity of operations, however it is important to understand

how this differs between ExpressRoute and Site-to-Site VPN connections.

ExpressRoute connections are typically terminated at both ends using two physical Ethernet

ports on two separate routers. This allows for a pair of simultaneously active physical

connections to carry data traffic to/from an ExpressRoute location. While Microsoft offers a

99.9% uptime SLA on the ExpressRoute service itself, it is up to the customer to appropriately

architect the network connectivity to an ExpressRoute location. This SLA will not be applicable if

the circuit is not built in an active-active configuration. Site-to-Site VPN connections are

associated to a single Azure Virtual Gateway that is redundant, but that connection is not

redundant. In the event of a lost connection on a Gateway a customer will have to establish

another connection.

Operational changes to an ExpressRoute circuit generally require modifications to both

Microsoft Azure and the underlying Service provider’s infrastructure. While Microsoft allows

customers to perform operational changes through the Service Management REST API, the

service provider generally offer a portal through which operational tasks can be carried out.

Operational changes to a Site-to-Site VPN Connection are very limited. Once the network is in

use, most changes are restricted. If you need to delete a gateway or change a gateway and



https://manage.windowsazure.com/

http://www.microsoft.com/en-sg/download/details.aspx?id=42674

http://msdn.microsoft.com/en-us/library/dn606310.aspx


Page 23




there are virtual machines on the Virtual Network. They will have to be removed in order to

make that modification.

8 Architecture Anti-Patterns

Establishing a Hybrid Network to Microsoft Azure where multi-hop routing will be used in a

Site-to-Site VPN scenario is not recommend. It is not recommended to utilize the “full mesh”

approach (outlined above) to perform multi-hop virtual network routing. When multi-hop virtual

network routing is configured using the full mesh routing approach, the supported number of

virtual network connections per gateway will limit the scale of the hybrid networking

infrastructure in Azure based on the gateway type selected.

The ExpressRoute connection is meant to be a Layer 3 (routed) connection between the

customer and Azure. As such, no Layer 2 connectivity is allowed over an ExpressRoute circuit.

Documents

Modern Datacenter - Architecture Patterns - Infrastructure · Cloud Platform Integration Framework i Modern Datacenter - Architecture Patterns - Infrastructure Prepared by Kunal Bhatia,