QCB Reference Architecture - Exertis Hammer · Azure Pack Deployment Scope Windows Azure Pack has four different deployment architectures: express, basic, minimal and scaled. In QCB100,

QCB Reference Architecture

QCB RA White Paper

2

CONTENTS

QCB Solution Introduction ............................................................................. 3

What is QCB?.................................................................................................................. 3

Logical Infrastructure ..................................................................................................... 3

Network Topology .......................................................................................................... 3

Storage Spaces Storage Solution ................................................................................... 4

Network Connection Redundancy ................................................................................. 6

Storage Network Redundancy ....................................................................................... 7

Azure Pack Self Service Platform ................................................................................... 8

QCB Software Infrastructure ........................................................................ 11

List of Management VMs ............................................................................................. 11

Service Accounts and Groups ...................................................................................... 12

Azure Pack Deployment Scope .................................................................................... 12

QCB Features Demonstration ....................................................................... 16

BMC Remote Management ......................................................................................... 16

vDC Virtual DC .............................................................................................................. 17

SOFS（Scale-Out File Server） .................................................................................... 17

Infra / WAP SQL............................................................................................................ 18

VMM Management Studio .......................................................................................... 19

SPF（Service Provider Foundation） .......................................................................... 20

Windows Azure Pack Admin Portal.............................................................................. 21

Windows Azure Pack Tenant Portal ............................................................................. 21

ADFS（AD Federation Services） ................................................................................ 22

QCB BOM ..................................................................................................... 23

QCB100......................................................................................................................... 23

QCB300......................................................................................................................... 23

QCB600......................................................................................................................... 25

About QCT .................................................................................................... 28

REVISIONS

Version Date Description Authors

0.1 12/16/2015 First publication Weithenn Wang

0.1Eng 12/22/2015 Translated Publication

Jack Hsu

3

QCB Solution Introduction

What is QCB? QCB is an acronym for Quanta Cloud in a Box. It is a private cloud solution offered by QCT that leverages Microsoft System Center and the Windows Azure Pack self-service user portal to provide a manageable and easy-to-maintain private cloud solution.

In QCB, the user self-service portal helps to provision and maintain VM resources automatically. This improves efficiency; less effort is needed to maintain the Virtual Machines, because the user can now provision and maintain their own VM resources without the intervention of IT personnel. VMs can now be provisioned within an hour or less. This helps to speed up project development time by improving resource request efficiency.

Logical Infrastructure The QCB private cloud platform is available in three different offerings: QCB100 (2U), QCB300 (8U) and QCB600 (14U). Figure 1 below illustrates the QCB 300 software and hardware infrastructure. The hardware requires 8U of rack space.

Figure 1. QCB Hardware & Software Topology

Network Topology Figure 2 below depicts the QCB network topology. There are two network switches in this QCB infrastructure. Switch Redundancy Protocol is used to avoid SPOF events (e.g., switch failure, power failure, cable failure).

Every node within the QCB solution has four 10Gbs network ports. Two ports are used for server management, VM migration and VM networks. With these ports, Windows 2012 R2 NIC Teaming is used for NIC load balancing and high availability.

4

The other two ports (SFP+) are dedicated to RDMA storage network bandwidth. With these ports, SMB Multichannel technology is used for load balance and high availability.

Figure 2. Network Topology

Storage Spaces Storage Solution QCB taps Windows Server 2012 R2 Storage Spaces technology as a storage solution, using a Windows Management Instrumentation (WMI) API (Management API, SMAPI) with corresponding PowerShell command. This storage API is a combination of a WMI object model and other related properties. This enables storage partners to expand the API with SNIA, SMI-S and SMP. There are many storage devices that already support SMI-S. In fact, since the release of Windows Server 2012, SMAPI is able to manage other storage devices without the need of added software.

Leveraging the concept of storage pool in Storage Spaces, QCB enables the creation of a large storage pool by attaching JBODs to the storage server; then, a scale-out file server is leveraged for high availability. An SMB3.0 share is created out of the storage pool as VM storage for the compute nodes (Hyper-V 3.0).

5

Figure 3. Storage Space Illustration

Storage Spaces technology supports different kinds of disk interfaces, such as SAS, Near Line SAS, SATA and SCSI. All of these can be in the same storage pool. Also, Storage Spaces supports different kinds of disk redundancy that are similar to RAID levels: Simple (RAID-0), Mirror (RAID-1) and Parity (RAID-5). There is also a hot spare option. The virtual disk created can be fixed or thin provisioned format.

In Storage Spaces, if Windows detects a mixture of SSD and spindle disk, there will be a tiered storage option. This option allows Storage Spaces to separate out the more often accessed data (hot data) and the less often accessed data (cold data). The hot data is stored in the SSD while the cold data gets written back to the spindle disks. Combining this with write-back cache technology, the IOPS of the storage are hugely increased.

Figure 4. Tiered Storage

6

Network Connection Redundancy In QCB there are two separate NICs. The cards are used for host management, VM migrations and VM networks. Windows 2012 R2 built-in NIC Teaming technology is used for load balancing, failover and LBFO. Not only does this provide greater throughput, but also it prevents network outage in the event of NIC failure or SPOF malfunction.

Figure 5. NIC Teaming with VLAN Support

Figure 6. Illustration of Network Redundancy

7

Storage Network Redundancy QCB Storage Network leverages RDMA (SMB Direct). SMB Multichannel technology is leveraged for storage network redundancy and load balance. RDMA can effectively offload the TCP/IP overhead on the CPU while driving down the network latency.

Figure 7. SMB Multichannel

Figure 8. RDMA in Effect

The QCB solution leverages Windows scale-out file server technology (SOFS) with SMB transparent failover. In SOFS, if a node within the cluster were to fail, applications or VMs running on top would not be affected by SOFS node failures, because SOFS provides continuity of service.

8

Figure 9. SOFS Continuity of Service

Azure Pack Self Service Platform QCB integrates with Microsoft System Center and Azure Pack to provide a highly flexible private cloud platform that offers IT administrators and tenants a self-service portal infrastructure.

Figure 10. Azure Pack Self-Service Platform

9

QCB leverages System Center Operation Manager to assist IT administrators in monitoring the underlying hardware platform, services that are running, and alerts that are generated when an availability, performance, configuration or security situation is identified.

Figure 11. SCOM Monitoring Hardware

QCB provides a separate hardware monitoring platform: the Quanta System Manager (QSM). QSM monitors hardware status—such as fan speed, power consumptions, and more—and is able to categorize the servers according to the datacenter’s topology.

.

Figure 12. QCB Dashboard

10

Figure 13. Datacenter Monitoring Platform

11

QCB Software Infrastructure

List of Management VMs There are many VM roles and services in QCB. Listed below are the VM roles and their corresponding running services:

Roles Application/Services

DC Domain Controller DNS DHCP

Infra SQL VMM SQL Database SPF SQL Database SCOM SQL Database

SCVMM Library Server SPF Service Provider Foundation

WAP SQL Windows Azure Pack Management Database SQL Cloud Database

WAP

Windows Azure Pack Admin API Tenant API / Tenant Public API Management Portal for Admin / Tenants Admin / Tenant Authentication Site

ADFS AD Federation Service SCOM Operations Manager

The list below provides the minimum VM requirements for QCB 100. In QCB 300 and QCB 600, the resource requirements scale accordingly.

Roles HostName vCPU vRAM （GB） vDisk （GB）

DC QCB-DC 2 4 / 8 / Dynamic 127 （Dynamic）

Infra SQL QCB-InfraSQL 4 16 / Static 300 （Fixed）

SCVMM QCB-VMM 4 16 / Static 127 （Dynamic）

SPF QCB-SPF 2 8 / Static 127 （Dynamic）

WAP SQL QCB-WAPSQL 4 16 / Static 300 （Fixed）

WAP QCB-WAP 4 16 / Static 127 （Dynamic）

ADFS QCB-ADFS 2 4 / 8 / Dynamic 127 （Dynamic）

SCOM QCB-SCOM 4 16 / Static 127 （Dynamic）

12

Service Accounts and Groups

Security Groups The list below details the security groups used in QCB along with the corresponding permissions.

Group Name Permissions/Description

QCB-SQLGroup Group members include QCB-SQL-gMSA

QCB-ADFSGroup Group members include QCB-ADFS-gMSA

SPF_Admin

Group members include QCB-SPF-SVC SPF_Provider

SPF_Usage

SPF_VMM

Service Accounts Below are the list of service accounts in QCB and their roles. QCB creates gMSAs (Group Managed Service Accounts) and DKM (Distributed Key Management) service accounts.

Account Name Permission/Account Roles

QCB-SQL-gMSA SQL service account (gMSA).

QCB-SQL-SVC SQL instance administrator. This provides read/write permission to the Infra SQL database.

QCB-VMM-SVC SCVMM administrator account. This account starts the VMM service and manages the VMM host infrastructure.

QCB-VMM-HV Hyper-V service account. This is the account used to manage the compute node by SCVMM.

QCB-VMM-DKM VMM data, database, product key, object encryption DKM

QCB-SPF-SVC SPF service account. This account needs to be a member of SPF_Admin, SPF_Provider, SPF_Usage and SPF_VMM groups.

QCB-ADFS-gMSA ADFS service account (gMSA)

Azure Pack Deployment Scope Windows Azure Pack has four different deployment architectures: express, basic, minimal and scaled. In QCB100, express and basic deployment are recommended. In QCB300, minimal deployment is recommended. If using QCB 600, scaled deployment is recommended.

Express Deployment Method Below is the express deployment of QCB. All Azure Pack services are running on one VM.

13

Figure 14. Windows Azure Pack Express Deployment

Basic Deployment Method Figure 15 below is the Windows Azure Pack basic deployment method. All of Windows Azure Pack services are installed on eight different VMs.

Figure 15. Windows Azure Pack Basic Deployment

14

Minimal Deployment Method Figure 16 below illustrates the Windows Azure Pack minimal deployment method. All of the Azure Pack services are installed on two management hosts with two separate VMs. Windows NLB service is then used to load balance the services on the hosts.

Figure 16. Windows Azure Pack Minimal Deployment

Scaled Deployment Method This is the operational framework for large deployment of Windows Azure Pack, which installs all operating elements on each host group to provide services. In addition, in order to maintain high availability of services, each host group will establish a NLB (Network Load Balancer) mechanism, and as for repository, failover clustering mechanisms are integrated to protect database services.

15

Figure 17. Windows Azure Pack Scaled Deployment

16

QCB Features Demonstration

BMC Remote Management In QCB, all management, compute and storage nodes use the BMC remote management utility for hardware monitoring.

Figure 18. BMC Remote Management

Figure 19. BMC Monitoring

17

vDC Virtual DC Windows 2012 R2 bootstrapping protocol with cached credentials is used in QCB. Hence, in the event of power failure, the cluster will use the cached credential to start its cluster services. This can avoid the chicken-and-egg issue of not being able to start the cluster without the DC.

Figure 20. Bootstrap Sequence

SOFS（Scale-Out File Server）

QCB uses a SOFS (Scale-Out File Server) high availability mechanism to communicate storage data through SMB 3.0 protocol, thus enabling the Hyper-V host to operate high availability VMs.

Figure 21. SOFS（Scale-Out File Server）

18

Figure 22. SOFS（Scale-Out File Server) / SMB3.0 Network Share

Infra / WAP SQL All the databases used by the Infra VMS (SCVMM, SPF, SCOM, etc.) are stored within the QCB-InfraSQL VM, and all the databases used for Azure Pack (WAP config, portal, usage, etc.) are stored within the WAPSQL host.

Figure 23. Infra SQL Management

19

Figure 24. Windows Azure Pack SQL Management

VMM Management Studio QCB integrates System Center VMM virtualization management platform to manage the Hyper-V virtual environment and the Windows Azure Pack self-service portal.

Figure 25. VMM Management Studio

20

Figure 26. VMM Network Management

SPF（Service Provider Foundation）

Service Provider Foundation (SPF) acts as a communication bridge between Windows Azure Pack and System Center to allow users to log in via the self-service portal to provision, manage and delete VMs.

Figure 27. Connecting Azure Pack to SPF

21

Windows Azure Pack Admin Portal The Windows Azure Pack admin portal allows administrators to create different subscriptions and plans according to the various hardware profiles they wish to offer to users. The different plans can be set to “public,” which are open to all users, or to “private,” which are accessible by invitation only.

The admin portal also provides the current total hardware usage report, status and user account management.

Figure 28. Windows Azure Pack Admin Portal

Windows Azure Pack Tenant Portal The Windows Azure Pack Tenant portal is a self-service portal for users to provision their own machines based on the subscription planned allowed and provided by the IT administrators.

Users can create accounts on the fly and request for private subscriptions to be used. Once the request is approved by IT admin, the private subscription plan becomes usable to the particular user.

Users may use Windows Azure Pack to provision VMs, SQL databases and websites and to see the status of hardware resources.

22

Figure 29. Windows Azure Pack Tenant Portal

ADFS（AD Federation Services） ADFS (Active Directory Federation Services) is used to allow single sign-on into the user portal. QCB supports ADFS3.0 (Windows Server 2012 R2) and ADFS2.0 (Windows Server 2008R2).

Figure 30. Windows Azure Pack Integration with ADFS

23

QCB BOM

QCB100 QCB 100 is the minimal SKU. It has compute, network and storage on the same 2U 2Node enclosure to form a hyper-converged infrastructure.

Software List Type Software

OS Windows Server 2012 R2 Datacenter

Management Software Microsoft System Center 2012 R2

Management Software Microsoft Azure Pack

Database Microsoft SQL Server 2012 SP2 Standard

Hardware List

Hardware Roles Quantity Server Model

Management Compute Storage 1 QuantaPlex T21SR-2U （S2SR）

Hardware Role Hardware Spec Quantity

MGMT / COMP / STOR

（2U – 2Nodes）

QuantaPlex T21SR-2U （S2SR） 1

CPU： Intel Xeon E5-2620v3 4

RAM： 16GB DDR4 32

Boot Disk： SATA SSD 240GB 4

HBA： Quanta LSI 3008 2

SSD Disk： SAS SSD 400GB 4

HDD Disk： SATA 4TB 8

NIC Card： Intel x540T2 2

QCB300 QCB 300 is the mid-scale SKU of QCB, having storage and compute on different enclosure, thus forming a converged architecture.

Software List Typle Software





24

Hardware List Hardware Roles Quantity Server Model

Management 1 QuantaPlex T21SR-2U （S2SR）

Compute 1 QuantaPlex T41S-2U （S2S）

Storage 1 QuantaPlex T21SR-2U （S2SR）

Network Switch 2 QuantaMeash T3040-LY3

Hardware Spec Management Node Server: This holds all management VMs, both storage and compute (vCPU / vRAM / vStorage).

Hardware Roles Hardware Spec Quantity

MGMT

（2U – 2Nodes）



RAM： 16GB DDR4 16




HDD Disk： SAS 600GB 10,000 RPM 10


RDMA Card： Quanta OCP Mezz CX3 2

Compute Node: This holds all tenant VMs（vCPU / vRAM).


COMP

（2U – 4Nodes）

QuantaPlex T41S-2U （S2S） 1


RAM： 16GB DDR4 64

Boot Disk： SATA 500GB 8



25

Storage Node: This holds all the tenant VM VHDs（vStorage).


MGMT / Network / STOR

（2U – 2Nodes）



RAM： 16GB DDR4 16








Network Switch

QuantaMeash T3040-LY3 2

40 Ports： 100/1000/10G BaseT

8 Ports： 1/10GbE SFP+

Switching Capacity： 960 Gbps

Latency： < 3us

10G SFP+ DAC Cable

10G SFP+ DAC Cable - 3m 18

10G Cat 6A - 2m 18

1G Cat5e CABLE - 2m 14

QCB600 QCB 600 is the larger SKU of the QCB solution family. It has manage, compute, storage and network on different enclosures to form a converged solution.

Software List Typle Software





26

Hardware List Hardware Roles Quantity Server Model

Management 1 QuantaPlex T21SR-2U （S2SR）

Compute 1 QuantaPlex T41S-2U （S2S）

Storage 1 QuantaPlex T21SR-2U （S2SR）

Network Switch 2 QuantaMeash T3040-LY3 1 QuantaMeash T1048-LY4A

Hardware Specifications Hardware Roles Hardware Spec Quantity

MGMT

（2U – 2Nodes）



RAM： 16GB DDR4 32




HDD Disk： SAS 600GB 10,000 RPM 10

NIC Card： Intel 82599ES SFP+ 2



COMP

（2U – 4Nodes）

QuantaPlex T41S-2U （S2S） 2


RAM： 16GB DDR4 128

Boot Disk： SATA 500GB 16




MGMT / Network / STOR

（2U – 2Nodes）



RAM： 16GB DDR4 32







27

JBOD

JBOD MESOS M4600H （JB9T） 1


MiniSAS 2


Network Switch

QuantaMeash T3048-LY2R 2

48 Ports： 1G/10GbE SFP+

4 Ports： 40GbE QSFP+

Switching Capacity： 1.28 Tbps

Latency： < 1us

MGMT Switch

QuantaMeash T1048-LY4A 1

48 Ports： 1GBase-T

2 Ports： 1G/10GbE SFP+

Switching Capacity： 136 Tbps

10G SFP+ DAC Cable

10G SFP+ DAC Cable - 3m 71

10G Cat 6A - 2m 6

1G Cat5e CABLE - 2m 23

40G QSFP+ DAC Cable - 1m 2

28

United States

China

QCT LLC., Silicon Valley office 1010 Rincon Circle, San Jose, CA 95131 TOLL-FREE: 1-855-QCT-MUST TEL: +1-510-270-6111 FAX: +1-510-270-6161 Support: +1-510-270-6216 QCT LLC., Seattle office 13810 SE Eastgate Way, Suite 190, Building 1, Bellevue, WA 98005 TEL: +1-425-633-1620 FAX: +1-425-633-1621

云达科技, 北京办公室（Quanta Cloud Technology）

北京市朝阳区东三环中路 1 号，环球金融中心东楼 1508 室 Room 1508, East Tower 15F, World Financial Center No.1, East 3rd Ring Zhong Rd., Chaoyang District, Beijing, China TEL: +86-10-5920-7600 FAX: +86-10-5981-7958

云达科技, 杭州办公室（Quanta Cloud Technology）

浙江省杭州市西湖区古墩路浙商财富中心 4 号楼 303 室

Room 303，Building No.4，ZheShang Wealth Center No. 83 GuDun Road, Xihu District, Hangzhou, Zhejiang , China TEL: +86-571-2819-8660

Japan

Taiwan

Other regions

Quanta Cloud Technology Japan 株式会社

日本国東京都港区芝大門二丁目五番八号

牧田ビル３階 Makita Building 3F, 2-5-8, Shibadaimon , Minato-ku, Tokyo 105-0012, Japan TEL: +81-3-5777-0818 FAX: +81-3-5777-0819

雲達科技（Quanta Cloud Technology）

桃園市龜山區文化二路 211 號 1 樓 1F, No. 211 Wenhua 2nd Rd., Guishan Dist., Taoyuan City 33377, Taiwan TEL: +886-3-286-0707 FAX: +886-3-327-0001 Quanta Cloud Technology No. 211 Wenhua 2nd Rd., Guishan Dist., Taoyuan City 33377, Taiwan TEL: +886-3-327-2345 FAX: +886-3-397-4770

About QCT

QCT (Quanta Cloud Technology) is a global datacenter solution provider extending the power of hyperscale datacenter design in standard and open SKUs to all datacenter customers.

Product lines include servers, storage, network switches, integrated rack systems and cloud solutions, all delivering hyperscale efficiency, scalability, reliability, manageability, serviceability and optimized performance for each workload.

QCT offers a full spectrum of datacenter products and services from engineering, integration and optimization to global supply chain support, all under one roof.

The parent of QCT is Quanta Computer Inc., a Fortune Global 500 technology engineering and manufacturing company.

http://www.QCT.io

All specifications and figures are subject to change without prior notice. Actual products may look different from the photos.

QCT, the QCT logo, Rackgo, Quanta, and the Quanta logo are trademarks or registered trademarks of Quanta Computer Inc.

All trademarks and logos are the properties of their representative holders.

Copyright © 2014-2016 Quanta Computer Inc. All rights reserved.

Documents

QCB Reference Architecture - Exertis Hammer · Azure Pack Deployment Scope Windows Azure Pack has four different deployment architectures: express, basic, minimal and scaled. In QCB100,