HPE Reference Configuration for Docker Datacenter on Bare Metal with Persistent Docker Volumes Docker Datacenter on HPE ProLiant DL360, Hyper Converged 380, and 3PAR StoreServ 8200

Technical white paper

Technical white paper

Contents Executive summary ................................................................................................................................................................................................................................................................................................................................ 3 Introduction ................................................................................................................................................................................................................................................................................................................................................... 4 Solution overview ..................................................................................................................................................................................................................................................................................................................................... 5 Solution components ............................................................................................................................................................................................................................................................................................................................ 6

Solution hardware ............................................................................................................................................................................................................................................................................................................................. 6 Solution management software ........................................................................................................................................................................................................................................................................................... 8

Docker Datacenter best practices on HPE Hyper Converged 380 and HPE ProLiant DL360 ................................................................................................................................... 13 Docker Datacenter configuration best practices .............................................................................................................................................................................................................................................. 15 Utilizing HPE OneView and HPE ICsp to deploy bare-metal UCP swarm workers .......................................................................................................................................................... 15

Using HPE Docker Volume Plugin for persistent Docker volumes ........................................................................................................................................................................................................ 16 Example of persistent volumes ......................................................................................................................................................................................................................................................................................... 17 Example of dedup volumes .................................................................................................................................................................................................................................................................................................. 17

Automated Docker monitoring with HPE Operations Bridge ..................................................................................................................................................................................................................... 19 Key highlights of the OMi Management Pack for Docker ........................................................................................................................................................................................................................ 19

Summary ...................................................................................................................................................................................................................................................................................................................................................... 21 Appendix A: Bill of materials ...................................................................................................................................................................................................................................................................................................... 21 Appendix B: Deploying Docker Datacenter management nodes on HPE Hyper Converged 380 ........................................................................................................................... 24

Configure Docker Datacenter Management components ....................................................................................................................................................................................................................... 25 Appendix C: Using Docker UCP etcd for the HPE Docker Volume Plugin...................................................................................................................................................................................... 26 Resources and additional links ................................................................................................................................................................................................................................................................................................ 27

Technical white paper Page 3

Executive summary Organizations building new generations of cloud native applications are looking for an environment that optimizes container resource efficiency and supports persistent container storage. This paper provides architectural guidance for deploying, scaling, and monitoring a Docker® environment using Docker Datacenter and a combination of HPE ProLiant servers, Hyper Converged systems, 3PAR all-flash storage, and Operations Bridge (OpsBridge) software.

This Reference Configuration expands on the HPE Reference Configuration for Docker Datacenter on HPE Hyper Converged 380 guide to provide architectural guidance on how organizations can take advantage of a hybrid virtualized and bare-metal deployment strategy and address the needs for persistent storage and monitoring of containers. The Docker cluster nodes will be deployed as bare-metal hosts on HPE ProLiant servers while the HPE 3PAR all-flash array will serve as the backend data store. The management plane and all its tools will be deployed as virtualized instances hosted by the HPE Hyper Converged 380.

The benefits of this hybrid architecture are to:

• Provide a high performance Docker environment and meet the persistent data needs of stateful container applications such as databases

• Enable management and monitoring of both Docker and non-Docker environments from a central console across the enterprise

• Enhance existing investments in virtualized infrastructure and applications by leveraging the HPE Hyper Converged 380 as a multi-purpose platform to host Docker management tools, other enterprise management tools or any other general purpose virtualized workloads.

• Support the Docker DevOps lifecycle end-to-end with the option to create VM sandboxes to test or prototype Docker applications as well as bare-metal Docker hosts to deploy Docker applications into production.

Docker Datacenter provides an integrated technology suite that enables development and IT operations teams to build, ship, and run distributed applications anywhere. Docker Datacenter brings container management and deployment services to enterprises with a production-ready Containers-as-a-Service (CaaS) platform. Inclusion of Docker Datacenter in this Reference Configuration enables a Docker native container management experience and on-premises CaaS platform, allowing organizations to save time and seamlessly take applications from development to production.

The combination of Docker Datacenter with HPE ProLiant servers, Hyper Converged infrastructure, 3PAR storage, and Operations Bridge monitoring allows organizations to build a resource efficient, storage I/O performance oriented, and highly manageable container platform.

Target audience: This document is intended for IT architects, systems integrators, and partners who are planning to deploy an enterprise grade Containers-as-a-Service platform with Docker Datacenter on Hewlett Packard Enterprise infrastructure.

Document purpose: The purpose of this document is to describe a best practice scenario for deploying Docker containers to a bare-metal environment with support for persistent storage and centralized event and health monitoring. Readers can use this document to achieve the following goals:

• Gain insight into the value proposition for running Dockerized workloads in a hybrid virtual and bare-metal environment leveraging the strengths of the HPE Hyper Converged 380 and HPE ProLiant DL360 servers

• Learn how to deploy the HPE Docker Volume Plugin for supporting container applications requiring data persistency and high performance I/O.

• Use HPE Operations Bridge software to monitor containers and the applications inside them, correlate with the status of their dependent physical infrastructure, and then manage detect-to-correct actions based on their impact on business users and targets.

Technical white paper Page 4

Introduction This Reference Configuration aims to define an architecture that is capable of supporting organizations across common use cases for containerization on their path to adopting containers and microservices as shown in Figure 1. This flexibility to support organizations at many stages of the journey is achieved by building on the scalable hyperconverged virtualization solution from the HPE Reference Configuration for Docker Datacenter on HPE Hyper Converged 380 guide and expanding it to include support for bare-metal host use cases as well. The ability to scale both the virtualized and bare-metal parts of the solution enables organizations to adjust the configuration to match their workloads.

Figure 1: Use case examples supported by this Reference Configuration

IT organizations are looking to deploy Docker containers on bare metal to optimize performance and resource efficiency of server hosts. By running bare metal and not relying on the hypervisor, the container stack needs fewer compute and storage resources than running in VMs. Consequently, more containers can run on a host than would be possible if these containers are running inside VMs. This consolidation can lead to reduced capex costs by minimizing virtualization and operating system instances licensing costs. Applications built using microservices and requiring high performance data stores are highly suitable to run on bare metal because they are highly distributed, scale horizontally, need quick instantiation, and don’t need hypervisor level features to be highly available. Deploying them on bare metal allows efficient usage of existing capacities on demand, higher server density, and more reactive systems.

While the cost and performance advantages of bare-metal hosts are quite attractive, the Docker Datacenter management components such as UCP managers and DTR replicas (see Figure 2) and other management tools are best deployed as VMs instead of individual physical instances to optimize resource usage, assure high availability of load balancers, and simplify maintenance. The hyperconverged portion of this Reference Configuration is intended to address this use case to deploy the Docker management components as VMs as well as providing a platform for hosting other virtualized workloads, both related and not related to containers. This provides a single, on-premises, infrastructure solution to act as a bridge for organizations adopting containers and microservices, allowing them to compose and scale applications where some tiers or services can be running as VMs while others are running as containers.

There are two additional deployment considerations for Docker containers: data persistency and container monitoring and management. Enterprise container applications needs data persistency as the container is moved from host to host or when the container is destroyed (intentionally or unintentionally). Docker developers need the ability to create a container and a data volume that will persist beyond the life of the container and the container host. The container data volumes should also be able to follow the container no matter where they are scheduled within a cluster of servers.

As organizations deploy container environments with increasing frequency and scale, monitoring the critical errors and the overall health of the containers becomes a challenge. They will require monitoring software that is aware of not only the Docker container, but also the services and applications within this container and the operating system and physical infrastructure below the container to determine and govern business impact. In addition, IT will need to manage both containerized and non-containerized environments. This can lead to potential management

Technical white paper Page 5

complexities where IT has to deal with multiple management consoles and collating and analyzing events from multiple sources. A centralized approach to logging and monitoring clusters of both containerized and non-containerized applications is critical to enable rapid root cause analysis.

Solution overview This Reference Configuration provides a solution architected for Docker Datacenter on hybrid bare-metal and virtualized HPE infrastructure and integrations to provide persistent container storage, automated server provisioning, and comprehensive monitoring and management as shown in Figure 2. This creates a scalable and highly available platform for a Docker Datacenter deployment, supports persistent Docker Volumes that leverage the strengths of the HPE 3PAR StoreServ 8200 All-Flash Starter Kit, and utilizes HPE Operations Bridge to monitor and maintain the solution.

Figure 2: Solution overview: Docker Datacenter utilizing HPE 3PAR Persistent Docker Volumes and HPE Operations Bridge

This solution includes a Docker Datacenter deployment consisting of Docker Universal Control Plane (UCP), Docker Trusted Registry (DTR), and Docker Commercially Supported Engine. The Docker management nodes will be running as virtual machines on a VMware® vSphere cluster as shown in Figure 2. For HA and load balancing purposes, Docker recommends multiple hosts for Docker UCP, DTR, and load balancers. Running these components initially in virtual instead of physical hosts will enable rapid deployment of the Docker management plane as well as reduce overall hardware requirements and promote better resource utilization. The HPE Hyper Converged 380 provides a scalable virtualization platform for the VMware vSphere environment. The VMware Distributed Resource Scheduler (DRS) clusters will provide resource management and high availability for the Docker Datacenter virtual machines. The Docker UCP cluster nodes are running on bare-metal HPE ProLiant DL360 Gen9 servers and can be efficiently scaled out by adding additional servers to the solution.

Technical white paper Page 6

The combination of the HPE 3PAR StoreServ 8200 All-Flash Starter Kit and the HPE Docker Volume Plugin provides a persistent storage solution to meet the needs of “stateful” containers, such as databases, that require high performance I/O and data persistency even when the container goes away. HPE 3PAR StoreServ has a unique architecture that allows the system to truly leverage flash capabilities as the system is built to be both massively parallel and efficient – a key requirement to persisting massive scale containerized application payload.

Solution components The following components were utilized in this Reference Configuration.

Solution hardware The following hardware components were utilized in this Reference Configuration as listed in Table 1.

Table 1: Hardware components

Component Purpose

HPE Hyper Converged 380 v1.1 Docker management and general virtualization hosts

HPE ProLiant DL360 Gen9 Bare-metal Docker swarm worker hosts

HPE 3PAR StoreServ 8200 All-Flash Starter Kit Storage for persistent Docker volumes

HPE Hyper Converged 380 For customers who are looking for a configurable, scalable, agile and highly available hyperconverged virtualization system, the HPE Hyper Converged 380 delivers a simple solution stack with extended flexibility and manageability. It builds on the powerful, industry-standard, HPE ProLiant DL380 Gen9 server platform, HPE StoreVirtual VSA software-defined storage, and is combined with VMware vSphere. HPE StoreVirtual VSA adds flexible scale, data mobility, and enterprise resiliency – all without adding complexity to the solution. The system utilizes the HPE OneView User Experience (UX) which integrates virtual machine management and vending, live automated server firmware updates and operations analytics in a single pane of glass. The HPE Hyper Converged 380 delivers a turn-key virtualization solution for medium-sized businesses, enterprises, and IaaS providers.

Designed from the ground up for the software-defined data center, the HPE Hyper Converged 380 enables a standardized approach to virtual server deployment, available in three workload configurations: General Virtualization, Cloud-In-a-Box and a Virtual Desktop Infrastructure (VDI). Unlike many hyperconverged systems, the HPE Hyper Converged 380 can be customized at the time of order, ships from the factory ready for virtualized workloads, and includes an on-site initial configuration utility to get up and running in a few simple clicks. After the initial installation, IT administrators manage their virtualized environment within HPE OneView User Experience and VMware vCenter Server.

HPE ProLiant DL360 Gen9 The Hewlett Packard Enterprise leading server for dense general-purpose computing, the HPE ProLiant DL360 Gen9 server delivers increased performance as compared to past generations with the best memory and I/O expandability packed in a 1U dense rack design. Reliability, serviceability and always on availability, backed by a comprehensive warranty, make it ideal for the most space constrained server workloads when you need:

• High performance dense server for space-constrained data centers

• Increased performance from Intel® Xeon® E5-2600 v4 processors (as compared to E5-2600 v3 processors) ranging up to 22 cores

• Faster DDR4 HPE SmartMemory with support for up to 3.0 TB at 2400 MHz

HPE 3PAR StoreServ 8200 All-Flash Starter Kit with iSCSI adapters The HPE 3PAR StoreServ 8200 All-Flash Starter Kit is an all-flash version of the HPE 3PAR StoreServ 8200 that provides all-flash acceleration at entry-level price. The kit includes the HPE 3PAR StoreServ 8200 Storage System Base equipped with 8 x 480GB SFF non-AFC SSD drives, the OS Suite, and Virtual Copy software. It also includes 3 year 24x7 Proactive Care Support. The HPE 3PAR StoreServ 8200 All-Flash Starter Kit has the same Drive LTUs cap (48) and maximum number of SSDs (120) as the HPE 3PAR StoreServ 8200. For this Reference Configuration iSCSI adapters were added as the HPE Docker Volume Plugin only supports iSCSI connections.

HPE 3PAR is built on a modern architectural design that includes multi-controller scalability, a highly virtualized data layer, system-wide striping, a highly specialized ASIC, and numerous flash innovations; optimizing it to deliver speed and enterprise data resiliency at an affordable price point.

Technical white paper Page 7

Some of the characteristics that make HPE 3PAR an ideal platform for containerized workloads include:

• Performance acceleration – optimizing the entire I/O path HPE 3PAR optimizes the entire I/O path from host ports, through the controller, firmware and even drive connections. It allows a volume to be active on every drive, every controller and every host port at all times. This design provides extremely high performance as the load is balanced across all the components.

• Mixed workload support Varying application I/O patterns can create stress on the performance of some storage arrays. With HPE 3PAR StoreServ, the metadata and data traffic are separated and processed in parallel greatly improving the responsiveness to I/O requests. Therefore, containers running transaction-intensive workloads (OLTP) are not held up behind workloads in containers running throughput-intensive workloads (e.g., media streaming) which makes 3PAR an ideal platform for running mixed workloads in multi-tenant environments.

• Eliminate noisy neighbors with Guaranteed Quality of Service (QoS) Quality of service (QoS) is an essential component for delivering modern, highly scalable multi-tenant storage architectures. HPE 3PAR Priority Optimization software enables service levels for applications and workloads as business requirements dictate, enabling administrators to provision storage performance in a manner similar to provisioning storage capacity. This allows the creation of differing service levels to each containerized application workload by assigning a minimum goal for I/O per second and bandwidth, and by assigning a latency goal so that performance for a specific tenant or application is assured. It is also possible to assign maximum performance limits on workloads with lower service-level requirements to make sure that high-priority applications receive the resources they need to meet service levels.

• Data Mobility through Storage Federation Often times it is required to migrate the data volumes around to optimize the cost of the workload on the right storage tier or during a technology refresh cycle when the array is replaced with the newer generation. HPE 3PAR Peer Motion offers bidirectional data mobility between 3PAR arrays without complex planning or dependency on external tools. Regardless of whether you are running a microservices-based architecture or a traditional application inside a container, the performance needs of each application can be matched to the volume and the array to maintain the balance of cost and performance requirement of containerized workloads.

HPE FlexFabric 5930 2QSFP+ 2-slot switch The HPE FlexFabric 5930 switch series is a family of high-density, ultra-low-latency, top-of-rack (ToR) switches that is part of the Hewlett Packard Enterprise FlexNetwork architecture's HPE FlexFabric solution.

Ideally suited for deployment at the aggregation or server access layer of large enterprise data centers, the HPE 5930 switch series is also powerful enough for deployment at the core layer of medium-sized enterprises.

With the increase in virtualized applications and server-to-server traffic, customers now require spine and ToR switch innovations that will meet their needs for higher-performance server connectivity, convergence of Ethernet and storage traffic, the capability to handle virtual environments, and ultra-low-latency all in a single device – the HPE FlexFabric 5930 switch series.

Technical white paper Page 8

Solution management software The following software components were utilized in this Reference Configuration as listed in Table 2 and Table 3.

Table 2: Docker Datacenter subscription components

Component Version

Docker Universal Control Plane (UCP) 2.0.0

Docker Trusted Registry (DTR) 2.1.0

Docker CS Engine 1.12.3-cs4

Table 3: HPE solution management software

Component Version

HPE Hyper Converged 380 Management UI 1.00.02

HPE OneView 2.00.07

HPE Insight Control server provisioning (ICsp) 7.5.1

HPE StoreServ Management Console (SSMC) 2.4.1

HPE Docker Volume Plugin https://github.com/hpe-storage/python-hpedockerplugin/tree/v1.1.0

HPE Operations Bridge Suite 2016.05

HPE Operations Manager i (OMi) 10.11

HPE OMi Management Pack for Docker 2.10

Docker Datacenter Docker Datacenter (DDC) provides container management and deployment orchestration services to enterprises with a production-ready platform supported by Docker and hosted locally behind the firewall.

Docker Datacenter delivers a secure software supply chain at enterprise scale. Docker’s integrated technology platform spans across the application lifecycle with tooling and support for both developers and IT operations.

Docker Datacenter features include:

• Built-in clustering and orchestration

• Declarative application services

• Content security with built-in image signing and verification enabling secure software lifecycle management

• Container centric networking with features such as service discovery, built-in load balancing and network routing mesh

• Secure access control with granular Role-Based Access Control (RBAC)

• CLI and API support

Docker Datacenter is built on the Docker Engine to deliver a universal, platform agnostic container runtime with built-in orchestration, networking and volumes for container-based applications. It offers open APIs for automation, extensibility and integrations into existing systems like LDAP/AD, monitoring, logging and more. Docker Datacenter comes with technical product support with dedicated SLAs.

This Reference Configuration includes production grade deployments of the core components of the Docker Datacenter subscription which are listed in Table 2 above.

Technical white paper Page 9

The Docker Datacenter UI dashboard will serve as the primary means of monitoring the Docker Datacenter deployment described in this Reference Configuration. As shown in Figure 3, the dashboard provides high-level status of the nodes of the DDC cluster, services and containers running in the cluster, as well as providing some quick action highlights for preforming common maintenance tasks such as adding nodes to the cluster, managing users, and configuring Docker Content Trust to restrict the DDC deployment to only run services with container images from trusted publishers.

Figure 3: Docker Datacenter dashboard

To achieve high availability, both Docker UCP and Docker DTR are deployed in a cluster with multiple nodes, each running the Commercially Supported (CS) Docker Engine as shown in Figure 4.

Figure 4: Docker Datacenter architecture

Technical white paper Page 10

Docker Datacenter consists of three primary types of nodes that can be deployed on a mixture of virtual machines and physical servers:

• UCP swarm managers: The swarm managers schedule tasks to run on worker nodes, perform cluster orchestration and management functions, and host the UCP UI and API.

• DTR Replica workers: Docker Trusted Registry nodes provide a highly available image repository as well as hosting the DTR UI.

• UCP swarm workers: These nodes run container workloads.

The Docker swarm manager orchestrates and schedules containers on the entire cluster. Docker Datacenter supports the use of three, five, or seven UCP swarm manager nodes for failover and state preservation.

HPE Hyper Converged 380 Management User Interface The HPE Hyper Converged 380 provides a Management User Interface that is designed for quick deployment and monitoring of virtual machines as shown in Figure 5. In this solution, we will leverage this Management User Interface to quickly deploy virtual machines from a preconfigured virtual machine template; the VMs will be used in the management plane. These virtual machine templates have Commercially Supported Docker Engine preinstalled.

Figure 5: HPE Hyper Converged 380 Management User Interface

HPE OneView HPE OneView is a comprehensive unifying platform designed from the ground up for converged infrastructure management. A unifying platform increases the productivity of every member of the internal IT team across servers, storage, and networking. By streamlining processes, incorporating best practices, and creating a new holistic way to work, HPE OneView provides organizations with a more efficient way to work. It is designed for open integration with existing tools and processes to extend these efficiencies.

HPE OneView is instrumental for the deployment and management of HPE servers and enclosure networking. It collapses infrastructure management tools into a single resource-oriented architecture that provides direct access to all logical and physical resources of the solution. Logical resources include server profiles and server profile templates, enclosures and enclosure groups, and logical interconnects and logical interconnect groups. Physical resources include server hardware blades and rack servers, networking interconnects, and computing resources.

The HPE OneView converged infrastructure platform offers a uniform way for administrators to interact with resources by providing a RESTful API foundation. The RESTful APIs enable administrators to utilize a growing ecosystem of integrations to further expand the advantages of the integrated resource model that removes the need for the administrator to enter and maintain the same configuration data more than once and keep all versions up to date. It encapsulates and abstracts many underlying tools behind the integrated resource model, so the administrator can operate with new levels of simplicity, speed, and agility to provision, monitor, and maintain the solution. HPE OneView is integrated with HPE Operations Bridge to ensure correlation of hardware and infrastructure issues with workloads running on the infrastructure (e.g., the Docker containers, the applications they contain and the business services being delivered).

Technical white paper Page 11

HPE Insight Control server provisioning (ICsp) Insight Control server provisioning is designed to streamline server provisioning administrative tasks. It simplifies the process of deploying operating systems on HPE ProLiant bare-metal servers as well as virtual machines.

HPE Insight Control server provisioning allows the administrator to perform the following tasks:

• Install Microsoft® Windows®, Linux®, VMware vSphere, and Microsoft Hyper-V on HPE ProLiant servers

• Deploy to target servers with, or without, PXE

• Run deployment jobs on multiple servers simultaneously

• Customize HPE ProLiant deployments with an easy-to-use, browser-based interface

HPE StoreServ Management Console (SSMC) The HPE 3PAR StoreServ Management Console is a converged management platform that can be used for all HPE 3PAR StoreServ arrays to provide a user interface with a modern look and consistent feel to the HPE OneView and ICsp UIs. The console provides an intuitive graphical interface to provision block, file, and object storage from one management interface serving diverse workloads for increased agility. The SSMC, provides all the information you need at a glance with customizable reporting capabilities and removes the need for add-on software tools as well as diagnosis and troubleshooting that require professional services.

HPE Docker Volume Plugin The HPE Docker Volume Plugin integrates with standalone Docker hosts as well as Docker swarm and Docker UCP clusters as shown in Figure 6. When used in clustered deployments, the plugin uses etcd instances to cache storage mappings, allowing orchestrated access to persistent data as the stateful container is migrated between hosts. Additionally, as Docker UCP clusters also use etcd, the plugin can be configured to use the existing UCP instances of etcd – thus reducing complexity. The Docker Data Volume and the mapping to the stateful container are portable within the Docker cluster allowing persistent data to follow the stateful container. This capability can be key in allowing IT operations the means to containerize and deploy applications such as databases which require data persistence in production.

Figure 6: HPE Docker Volume Plugin use in a Docker UCP cluster

One of the hurdles to enterprise adoption we address is enabling customers to get unique data from Docker containers to a fast, reliable storage backend as quickly and easily as possible. The HPE Docker Volume Plugin allows users of Docker to utilize HPE 3PAR storage arrays and HPE StoreVirtual VSA software-defined storage using Docker containers. The plugin is packaged as a Docker container to simplify the installation process down to assuring there is a copy of the container running on each Docker cluster host. Configuration and usage of the plugin is straightforward and allows for users familiar with Docker to quickly begin using volumes with HPE storage backends via iSCSI connections.

Technical white paper Page 12

Note Only iSCSI connections were supported by the initial release of the plugin. Be sure to order a pair of iSCSI adapters for the HPE 3PAR StoreServ 8200 as per entries in Appendix A or check the HPE Docker Volume Plugin documentation to verify support of other connection types.

The plugin supports the following capabilities via the --opt parameter of the “docker volume create” command:

• Size – specifies the desired size in GB of the volume.

• Provisioning – specifies the type of provisioning to use (thin, full, dedup).

• Flash-cache – specifies whether flash cache should be used or not (True, False). 3PAR Adaptive Flash Cache allows SSDs to act as Level 2 read cache holding random read data for spinning media that has aged out of DRAM read cache. Adaptive flash cache reduces application response time for read intensive I/O workloads and can improve write throughput in mixed-workload environments.

HPE Operations Bridge HPE Operations Bridge (OpsBridge) is a simplified, unified and automated IT operations management software. OpsBridge provides the ability to sense, analyze and adapt to manage business and IT services that support digital business. With advanced event correlation, log intelligence, Big Data based and predictive analytics plus automation you can detect and correct issues across all your technologies to prioritize business targets. OpsBridge senses data across 100+ new technologies including Docker using agent-based management packs and agentless methods, and integrates data from your existing tools using Operations Connectors. This coverage is further augmented with out-of-the-box HPE and partner developed correlation rules and domain specific report packs which can easily be customized.

OpsBridge model-based IT operations management brings a single pane of glass to operations. With automated discovery, monitoring and analysis of traditional, public and private cloud services, and Docker containers plus out-of-the-box integration with HPE OneView, IT operations gains a Metal to App, top to bottom view of infrastructure and its business impact. As an example, the impact of a fan or power supply issue can be correlated with the IT and business services it would impact, and remediation can be automated to switch the applications in question to a different Docker system before any impact is felt.

The OpsBridge solution incorporates three editions to suit customer requirements. This Reference Configuration validated the HPE Operations Bridge Express edition, but it should be noted organizations can upgrade progressively to the next edition as needed. All editions are based on the run-time service model detailed above, the industry’s first and only dynamic model of the IT landscape.

OpsBridge and Docker As the Docker platform is adopted to host more mission-critical applications and microservices, it is increasingly important to incorporate monitoring of their health and lifecycle. Using the HPE OMi Management Pack for Docker, OpsBridge automates discovery of the hosts and the containers they support, and can even detect and automatically initiate monitoring of popular applications running inside containers such as MySQL and MongoDB. Subsequently, monitoring is automatically activated based on best practice based policies. OpsBridge provides end-to-end correlation of IT data spanning from the underlying infrastructure up to the services the Docker based applications are a part of. This “service impact” based management accelerates operations responsiveness with as much as 72x improved mean time to repair1, and up to 75% IT event reduction being measured by some of our customers.

OMi Management Pack for Docker features OMi Management Pack for Docker, Version 2.10 features include:

• Docker Engine support up to version 1.12.1

• Automatic discovery of swarm mode, monitoring of swarm health, and topology views for Docker swarm

• Discovers Docker containers with MySQL and MongoDB relational databases and monitors those applications

• Enriched monitoring of the performance of Docker containers with performance dashboards for Docker hosts and containers

• Real-time streaming of Docker performance metrics

• Enhanced Docker topology discovery with image and Docker daemon

1 72x MTTR improvement measured from real use case by HPE IT leveraging the analytics capabilities of OpsBridge Ultimate

Technical white paper Page 13

• Monitoring Docker container log files to generate events for container specific errors and failures

• Integrated tools to act directly on containers for troubleshooting and remediation

OpsBridge version 10.11 or later must be used to manage Docker, see our User Guide for more details and read this blog for a step-by-step guide to using the Management Pack for Docker.

Docker Datacenter best practices on HPE Hyper Converged 380 and HPE ProLiant DL360 This Reference Configuration outlines how to configure a hybrid deployment of Docker Datacenter using a combination of virtual machines and physical servers as shown in Figure 7. This hybrid configuration provides an optimized solution for hosting the management portion of the solution using a hyperconverged virtualization platform as well as a highly efficient solution to host and scale Dockerized applications on physical servers.

This solution is ideal for organizations who are developing cloud native applications using containers and want to maximize resource, performance, and cost efficiencies. The Docker UCP swarm workers can take advantage of the performance of the HPE DL360 and HPE 3PAR all-flash array hardware, while the Docker management components can run as VMs instead of dedicated physical hosts.

Docker Datacenter Management Components (HA VMs on the HPE Hyper Converged 380)

Docker UCP Swarm Workers (HPE ProLiant DL360 Servers)

Docker UCPLoad

Balancer

UCP Swarm Manager

UCP Swarm Manager

UCP Swarm Manager

Docker DTRLoad

Balancer

DTR Replica DTR Replica DTR Replica

Bare-Metal Worker 1

Bare-Metal Worker 2 . . . Bare-Metal

Worker N

Figure 7: Hybrid virtual and physical node Docker Datacenter logical diagram

Technical white paper Page 14

The HPE Hyper Converged 380 can support from two to sixteen nodes in a single deployment with up to three 8-disk storage blocks per node. For this Reference Configuration we are using a three-node configuration with two 4.98 TB hybrid storage blocks for mixed use per node as shown in Figure 8. Each of these hybrid 8-disk blocks consist of six 900GB SAS HDD drives and two 480GB mixed use SSD drives as documented in the HPE Hyper Converged 380 Installation Guide. This configuration provides a total of 9.96 TB of usable storage per node. Depending on your intended workload, a different quantity or type of storage block may be used as long as the same type and quantity of storage blocks are used in each node of the cluster.

Figure 8: HPE Hyper Converged 380 (three nodes with two blocks of storage)

This configuration provides high availability of storage using HPE StoreVirtual VSA that enables all of the storage in each Hyper Converged 380 node to be presented as software-defined storage to both of the solution’s VMware vSphere clusters as well as to other hosts on your network. In this configuration we selected three physical nodes to provide full high availability to the vSphere cluster that can in turn be used to provide additional redundancy for critical virtual machines running key Docker Datacenter roles.

While having multiple Docker UCP swarm managers and Docker DTR replica nodes does provide high availability of Docker UCP and DTR respectively, HPE does not recommend this as the sole means of making Docker UCP and DTR highly available. HPE recommends using the minimal number of manager and replica nodes (three each) needed to achieve high availability from a pure Docker Datacenter perspective as VMware vSphere high availability will already be protecting key virtual machines such as the Docker UCP managers, DTR replicas, and load balancers to assure they are restarted immediately in the event of a physical node failure. This leverages the strength of the HPE Hyper Converged 380 platform, minimizes licensing costs, and avoids potential performance degradation of the Docker Datacenter deployment.

The HPE ProLiant DL360 Gen9 servers were used as the Docker UCP swarm worker nodes that will be used to host all Docker application containers as shown in Figure 7. These servers were selected as they provide excellent general-purpose compute and memory performance for a 1U, two-socket form factor. Offering exceptional density, flexibility, and energy efficiency, these servers are an ideal match for dynamic workloads as mixed-use Docker swarm workers that will be tasked with running many different Dockerized applications. Additionally, the two-socket form factor is optimal for Docker Datacenter per node licensing which supports up to two sockets per license. Using the HPE 3PAR StoreServ array to meet the persistent storage needs of the Dockerized workloads allows organizations to centrally manage the storage capacity of the Docker Datacenter cluster, and organizations can efficiently add additional HPE ProLiant DL360 servers to meet the memory and compute requirements of their applications.

Technical white paper Page 15

As per Figure 9, this Reference Configuration devoted two 10 GbE ports each to the Docker production network used for all Dockerized application network traffic and to the Docker storage network used for iSCSI volumes presented from the HPE 3PAR array. While beyond the scope of this document to provide specific configuration guidance, HPE recommends that the two production ports should be configured as bonded network interfaces to increase redundancy or throughput as per your best practices. The two storage ports should not be configured as bonded as the multipath features of the HPE Docker Volume Plugin will simplify the process of utilizing both network interfaces to attach multipath enabled iSCSI volumes.

Item Description Recommended use

1 Embedded 4x1GbE network adapter Use 1 or more ports for datacenter management network for connectivity of HPE OneView and HPE ICsp appliances to deploy servers

2 PCIe 3.0 expansion slot with HPE Ethernet 10Gb 2-port 560SFP+ adapter

Use 1 port each for Docker production (Ethernet) network and Docker storage (iSCSI) network

3 FlexibleLOM bay with HPE Ethernet 10Gb 2-port 560FLR-SFP+ adapter

Use 1 port each for Docker production (Ethernet) network and Docker storage (iSCSI) network

Figure 9: HPE ProLiant DL360 Gen9 network interfaces

Docker Datacenter configuration best practices This section outlines several best practices validated by HPE as part of this Reference Configuration. HPE recommends following Docker best practices for Docker Datacenter deployments including:

• Configure all Docker hosts (swarm managers, swarm workers, and DTR replicas) to use a “direct-lvm” devicemapper storage driver configuration. The “loop-lvm” configuration that is set up by default is not recommended for performance reasons.

• Utilize the HPE Hyper Converged 380 to create highly available load balancer VMs for use with Docker UCP and Docker DTR. In addition to the obvious benefits of balancing UI and API requests across the three manager nodes and DTR replicas this also effectively provides external load balancing to any applications hosted in your Docker Datacenter instance.

• Use the VMware Distributed Resource Scheduler (DRS) to keep the management nodes (Docker UCP managers and DTR replicas) on separate physical hosts to prevent a host failure from impacting your Docker Datacenter deployment.

• Create a backup and recovery plan for both DTR images and configurations, Docker UCP configuration data, and Docker volume data. Specific guidance in this area is beyond the scope of this Reference Configuration; however, Docker has published guidelines at:

– https://docs.docker.com/datacenter/ucp/2.0/guides/high-availability/backups-and-disaster-recovery

– https://docs.docker.com/datacenter/dtr/2.1/guides/high-availability/backups-and-disaster-recovery

For more details on these best practices as well as configuration and deployment guidance, see Appendix B.

Utilizing HPE OneView and HPE ICsp to deploy bare-metal UCP swarm workers This configuration includes four HPE ProLiant DL360 Gen9 servers with two local 300GB disks serving as bare-metal Docker UCP swarm workers as shown in Figure 7. The number of swarm workers, and therefore number of HPE ProLiant DL360 servers, should be adjusted as appropriate based on the number of Dockerized applications that you intend to host in the cluster.

To simplify and improve the reliability of deploying the bare-metal swarm workers, HPE recommends using a combination of HPE OneView and HPE Insight Control server provisioning (HPE ICsp) to configure and install the HPE ProLiant DL360 Gen9 servers.

Technical white paper Page 16

By using the Server Profile Template feature within HPE OneView you can specify and maintain a single configuration for the system firmware, BIOS, and boot-order at time of initial deployment as well as orchestrate updates to that configuration as needed. This provides a location to centrally manage and update configuration settings, such as system firmware, and provides assurance that each server is running with the same configuration and has event and health data being exposed up to the HPE OneView and OpsBridge dashboards.

For OS provisioning, HPE recommends using a customized version of the out-of-the-box “ProLiant OS – RHEL 7.2 x64 Scripted Install” OS build plan. This provides an HPE standardized starting point for the base operating system install that can be further customized to include Docker CS Engine with the direct-lvm configuration and other recommended Docker best practices.

An overview of the process for deploying new bare-metal Docker UCP swarm workers is provided below:

1. Physically set up the server and set up the four network connections to the Docker production (Ethernet) network and Docker storage (iSCSI) network as per Figure 9.

2. Register the server with HPE OneView and deploy it with a Server Profile Template to configure the system firmware, BIOS, and boot-order.

3. Register the server with HPE ICsp and deploy it with a customized OS build plan to deploy the base OS, configure networking, and perform the following additional custom steps to configure the host as a UCP swarm worker:

a. Configure firewall for Docker

b. Install Docker CS Engine

c. Join the swarm cluster as a worker

d. Install the HPE Docker Volume Plugin

For more detailed examples of how to use HPE OneView and HPE ICsp to deploy bare-metal Docker hosts, see the HPE Reference Architecture for Docker Datacenter on HPE CA700 with HPE Helion CloudSystem 9.0 Update 2 guide for examples on how to add customized steps to HPE ICsp OS build plans. The examples from the HPE CA700 guide should be combined with information in Appendix B to create an OS build plan to deploy the same version of Docker and Red Hat® Enterprise Linux (RHEL) 7.2 as used on the Docker management virtual machines running on the HPE Hyper Converged 380.

Using HPE Docker Volume Plugin for persistent Docker volumes The HPE Docker Volume Plugin will be installed on the cluster nodes to support container storage requests using an HPE 3PAR StoreServ 8200 all-flash storage array. These nodes should already be joined to the Docker UCP cluster and appear on the Nodes pages within the Docker UCP web user interface.

This Reference Configuration validated the plugin version available as an open-source project at https://github.com/hpe-storage/python-hpedockerplugin/tree/v1.1.0. Refer to the HPE Docker Volume Plugin documentation for usage documentation. Make sure to follow the quick-start instructions for configuring and installing the containerized version of the plugin using Docker Compose.

Tip Use an anti-affinity rule to quickly deploy plugin containers to each node of your Docker UCP cluster by specifying an environment setting in either your run command, your docker-compose.yml file, or your daemon settings:

affinity:container!=hpevolumedockerquickstart_hpedockerplugin_*

See Appendix C for more details on simplifying plugin deployments in Docker UCP.

Technical white paper Page 17

Example of persistent volumes We executed a sequence of commands to illustrate how volumes created with the plugin persist and the data written to them is preserved beyond the life of the container. The example below shows a file written to a 1 GiB thinly provisioned persistent volume mounted at /tmp of container b702af886c74; it can be seen after exiting from that container and mounting the same volume to the new container 67ecbb4d58b0. Conversely, the notPermanent file, written to a different location (outside of the mounted volume), does not persist and is shown to not exist in the second container:

[user@node1 ~]# docker volume create -d hpe --name persistentDemo -o size=1 -o provisioning=thin persistentDemo [user@node1 ~]# docker run -it -v persistentDemo:/tmp ubuntu bash root@b702af886c74:/# echo "this is saved on a persistent volume" > /tmp/demo root@b702af886c74:/# echo "this is will be destroyed with the container > /notPermanent root@b702af886c74:/# exit [user@node1 ~]# docker run -it -v persistentDemo:/tmp ubuntu bash root@67ecbb4d58b0:/# cat /tmp/demo this is saved on a persistent volume root@67ecbb4d58b0:/# cat /notPermanent

cat: /notPermanent: No such file or directory

Because the data is saved to the HPE 3PAR volume, it can be attached to Docker containers running anywhere in the UCP cluster. While not illustrated here, a similar example to the above could have been executed with a swarm scheduling constraint to assure the two containers were launched on different physical hosts.

Example of dedup volumes We executed a sequence of commands to create a dedup volume and then fill it up with 50 copies of a 20 MB tar file using different names (sample1.tar – sample50.tar) to highlight the benefit of the deduplication support in the plugin:

[user@node1 quick-start]# sudo docker volume create -d hpe --name dedupDemo3PAR -o size=1 -o provisioning=dedup [user@node1 ~]# sudo docker run -it -v dedupDemo3PAR:/tmp --name 3parDemo ubuntu bash <Then run commands inside the container to fill up the volume> root@44279207e625:/# tar -cf /tmp/sample.tar /lib tar: Removing leading `/' from member names root@44279207e625:/# du -h /tmp/* 20M /tmp/sample.tar root@44279207e625:/tmp# for i in `seq 1 50`; do cp /tmp/sample.tar /tmp/sample$i.tar ; done cp: error writing '/tmp/sample50.tar': No space left on device root@44279207e625:/tmp# du -h /tmp 1007M /tmp root@44279207e625:/tmp# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/docker-253:0-134509569-f93d6d5a8f1706499ae80d2f83c9d7f5ac1e048621c50182820090a647d0bd99 10G 155M 9.9G 2% / tmpfs 63G 0 63G 0% /dev tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sdc 1008M 1008M 0 100% /tmp /dev/mapper/rhel-root 50G 7.1G 43G 15% /etc/hosts shm 64M 0 64M 0% /dev/shm After the volume was full, we exited from the container which then unmounts the volume automatically and makes it available to attach to any container running anywhere else within the Docker UCP cluster, thus providing the option of persistent data mobility across the cluster.

Technical white paper Page 18

To see the actual storage used as part of this Docker volume we used the HPE StoreServ Management Console (SMMC) and were able to quickly locate the dedupDemo3PAR volume we created as shown in Figure 10.

Figure 10: Searching for Docker volume name in the HPE 3PAR StoreServ Management Console

Tip It is best to use unique Docker volume names when creating volumes with the plugin. Unique names will increase the chance of only matching a single volume when using the SSMC “Search” feature as shown in Figure 10. Additionally, when doing searches, it will typically be necessary to select to search “All” to locate the Docker volume.

By clicking on the link in the search screen, the SSMC will navigate to the report for our Docker volume. As we were interested in the dedup savings in this test, we selected the Capacity report for the volume as shown in Figure 11. The report shows a 98% savings where the 1007 MB of data we wrote to the volume only consumed 0.02 GiB of actual physical storage in the array.

Figure 11: Volume Capacity report in HPE 3PAR StoreServ Management Console

Technical white paper Page 19

Automated Docker monitoring with HPE Operations Bridge OpsBridge enables you to automatically and dynamically monitor the availability and performance of your Docker infrastructure using the HPE OMi Management Pack for Docker. The Management Pack automatically discovers Docker topology and populates them into Run Time Service Model (RTSM) instantly. Automated monitoring by this Management Pack generates alerts on containers starving for resources, network performance, and workload errors, etc. It also continuously monitors the performance and availability metrics which are stored in HPE Operations Agent data store with further use of that data for analytics, reporting and graphing solutions.

Key highlights of the OMi Management Pack for Docker Enhanced automated Docker topology discovery – OMi Management Pack for Docker is comprised of out-of-the-box automated discovery for the entire Docker infrastructure, including clusters, images, daemons, containers and applications, and creation of respective elements in RTSM to provide a topology map as shown in Figure 12. Automated discovery enables any Docker container instance to be instantly discovered and automated monitoring configuration to be activated.

Figure 12: HPE OMi automated Docker topology map

Enriched automated monitoring and alerting of the performance of Docker infrastructure – OMi Management Pack for Docker monitors the critical aspects of Docker Host and Container health which includes – container CPU throttling, memory usage limit, and disk I/O operations as well as container network statistics including: Network In/Out Bytes, Network In/Out Packets and Network failures like Errors and Drops. The management pack comes with out-of-the-box templates and threshold configurations that can be easily customized and fine-tuned per container or per container tags

Automated Docker Log and Error monitoring – OMi Management Pack for Docker notifies the Event Console on detecting container specific errors and failures in Docker container logs based on customizable predefined patterns. Event management will automatically execute correlation of the event with respect to dependent objects to ascertain root cause. Subsequent execution of simple tasks or complex workflows, can be configured to apply best practice ITSM processes and operator actions to remediate issues and keep business targets on track. The OMi Management Pack for Docker includes specific customizable correlation rules to accelerate deployment.

Technical white paper Page 20

Performance Dashboards for Docker Hosts and Docker Containers – OMi Management Pack for Docker comes equipped with an out-of-the-box Performance Dashboard that helps you visualize the performance of Docker hosts and each container at a glance and looks for performance patterns even with the past data. The dashboard for Docker Host provides a summary view of the containers and the health of Docker host as shown in Figure 13. On the other hand, the dashboard for Docker Containers provides container’s availability and resource usage overview. Users can customize or create their own dashboards.

Figure 13: HPE OMi Performance Dashboard for Docker Hosts

Real time streaming of Docker performance metrics – OMi Management Pack for Docker also provides an option to stream Docker Host and Container performance metrics in real time to the performance dashboards to get the pulse of Docker container performance exactly at a given point in time as shown in Figure 14 and accelerate remedial actions.

Figure 14: HPE OMi individual container performance dashboards

Technical white paper Page 21

Inspect and Interact with Container – OMi Management Pack for Docker includes tools which help trigger actions on containers in the context of alerts or events in the OMi Event console or from the performance dashboard of any container. You can inspect the container or you can look at the container logs. Any particular container can be stopped and started again via the tools. You can also interact with a container by running commands inside the container to troubleshoot the issue or to rectify the issue.

Summary This document has described how to architect and deploy Docker Datacenter on HPE Hyper Converged 380 and HPE ProLiant DL360 with HPE OneView provisioning to quickly install and scale a Docker Datacenter deployment. This solution combines the ease-of-use of the hyperconverged virtual machine management for Docker management components with the cost savings and performance efficiency of bare-metal Docker swarm workers.

To address the need for persistent container storage, the configuration included the HPE 3PAR StoreServ 8200 All-Flash Starter Kit along with HPE Docker Volume Plugin to provide cost-effective high-performance storage for containers and allow container mobility across the cluster.

Finally, the configuration leveraged the operation management abilities of HPE Operations Bridge to optimally manage the Docker platform in production by providing automated monitoring and remediation services to maintain the health and performance of the Docker environment, containers, and applications.

Appendix A: Bill of materials The following BOMs contain electronic license to use (E-LTU) parts. Electronic software license delivery is now available in most countries. HPE recommends purchasing electronic products over physical products (when available) for faster delivery and for the convenience of not tracking and managing confidential paper licenses. For more information, please contact your reseller or an HPE representative.

Note Part numbers are at time of publication and subject to change. The bill of materials does not include complete support options or other rack and power requirements. If you have questions regarding ordering, please consult with your HPE Reseller or HPE Sales Representative for more details. hpe.com/us/en/services/consulting.html

Table 4: Bill of materials

Qty Part number Description

Rack and network infrastructure

1 BW908A HPE 42U 600x1200mm Enterprise Shock Rack

4 H8B50A HPE Mtrd Swtchd 4.9kVA/L6-30P/NA/J PDU

1 BW930A HPE Air Flow Optimization Kit

1 BW909A HPE 42U 1200mm Side Panel Kit

1 JG505A HPE 59xx CTO Switch Solution

1 JG510A HPE 5900AF 48G 4XG 2QSFP+ Switch

2 JD096C HPE X240 10G SFP+ SFP+ 1.2m DAC Cable

2 JC680A HPE 58x0AF 650W AC Power Supply

2 JC682A HPE 58x0AF Bck(pwr) Frt(prt) Fan Tray

2 JG505A HPE 59xx CTO Switch Solution

2 JH379A HPE 5930 2-slot 2QSFP BF AC Bdl

4 JG326A HPE X240 40G QSFP+ QSFP+ 1m DAC Cable

4 JH180A HPE 5930 24p SFP+ and 2p QSFP+ Mod

22 JD097C HPE X240 10G SFP+ SFP+ 3m DAC Cable

Technical white paper Page 22

Qty Part number Description

Three (3) node Hyper Converged 380 for virtualization with two 4.98 TB Hybrid storage blocks per node

3 P9D74A HPE HC380 Cluster Node

3 P9D74A 001 HPE HC380 General Virtualization SW

3 817959-L21 HPE DL380 Gen9 E5-2690v4 FIO Kit

3 817959-B21 HPE DL380 Gen9 E5-2690v4 Kit

24 805351-B21 HPE 32GB 2Rx4 PC4-2400T-R Kit

12 816985-B21 HPE 480GB 6G SATA MU-3 SFF SC SSD

36 785069-B21 HPE 900GB 12G SAS 10K 2.5in SC ENT HDD

3 719073-B21 HPE DL380 Gen9 Secondary Riser

3 665243-B21 HPE Ethernet 10Gb 2P 560FLR-SFP+ Adptr

3 749974-B21 HPE Smart Array P440ar/2G FIO Controller

3 726897-B21 HPE Smart Array P840/4G Controller

6 783009-B21 HPE DL380 Gen9 8SFF SAS Cable Kit

3 786092-B21 HPE DL380 Gen9 8SFF H240 Cable Kit

3 758959-B22 HPE Legacy FIO Mode Setting

3 666988-B21 HPE 2U Security Bezel Kit

3 733660-B21 HPE 2U SFF Easy Install Rail Kit

6 720620-B21 HPE 1400W FS Plat Pl Ht Plg Pwr Spply Kit

3 768900-B21 HPE DL380 Gen9 Sys Insght Dsply Kit

3 733664-B21 HPE 2U CMA for Easy Install Rail Kit

3 P9D85A HPE HC380 Base SW Image 6.0 FIO Kit

1 H1K90A3 HPE 3Y Proactive Care NBD Service

6 BD715AAE VMw vSphere EntPlus 1P 3yr E-LTU

1 P9U41AAE VMw vCenter Server Std for vSph 3y E-LTU

Four (4) HPE ProLiant DL360 servers – bare-metal Docker hosts

4 755258-B21 HPE DL360 Gen9 8SFF CTO Server

4 Q0J88AAE HPE Docker Eng Server Bundle 9x5 3Yr Sub

4 Q0K92AAE HPE Docker CS Eng Upg-DDC 3yr 9x5 E-LTU

4 818176-L21 HPE DL360 Gen9 E5-2640v4 FIO Kit

4 818176-B21 HPE DL360 Gen9 E5-2640v4 Kit

32 805351-B21 HPE 32GB 2Rx4 PC4-2400T-R Kit

8 759208-B21 HPE 300GB 12G SAS 15K 2.5in SC ENT HDD

4 665243-B21 HPE Ethernet 10Gb 2P 560FLR-SFP+ Adptr

4 761872-B21 HPE Smart Array P440/4G FIO Controller

4 665249-B21 HPE Ethernet 10Gb 2P 560SFP+ Adptr

4 734807-B21 HPE 1U SFF Easy Install Rail Kit

8 720478-B21 HPE 500W FS Plat Ht Plg Pwr Supply Kit

Technical white paper Page 23

Qty Part number Description

One (1) HPE 3PAR StoreServ 8200 All-Flash Starter Kit with iSCSI adapters and accessories

1 K2Q36A HPE 3PAR StoreServ 8200 2N Fld Int Base

1 M0S95A HPE 3PAR StoreServ 8200 AF Starter Kit

2 H6Z10A HPE 3PAR 8000 2-pt 10Gb iSCSI/FCoE Adptr

8 K2P89A HPE 3PAR 8000 1.92TB SFF SSD

1 L7B45A HPE 3PAR 8200 OS Suite Base LTU

8 L7B46A HPE 3PAR 8200 OS Suite Drive LTU

1 L7B57A HPE 3PAR 8200 Virtual Copy Base LTU

8 L7B58A HPE 3PAR 8200 Virtual Copy Drive LTU

1 BD362A HPE 3PAR StoreServ Mgmt/Core SW Media

1 BD363A HPE 3PAR OS Suite Latest Media

4 JG081C HPE X240 10G SFP+ SFP+ 5m DAC Cable

Docker Datacenter licenses for management VMs

6 Q0K90AAE HPE Docker Dtr 3yr 9x5 E-LTU

HPE Operations Bridge Express Edition licenses

5 M4D54AAE HPE OB Ste EE 250+ Nd 50 Nd Pk SW E-LTU

2 M4D56AAE HPE OBSC EE AO 5+ OSI 5 OSI Pk SW E-LTU

Alternate solution components

Additional network adapters to add to the three (3) node Hyper Converged 380 for virtualization, if additional bandwidth is desired

3 665249-B21 HPE Ethernet 10Gb 2P 560SFP+ Adptr

Alternate Docker licensing options (pick one) for the HPE ProLiant DL360 servers

Option 1

4 Q0J87AAE HPE Docker Eng Server Bundle 9x5 1Yr Sub

4 Q0K91AAE HPE Docker CS Eng Upg-DDC 1yr 9x5 E-LTU

Option 2

4 Q0J93AAE HPE Docker CS Eng Serv Bndl 7x24 1Yr Sub

4 Q0K97AAE HPE Docker CS Eng Upg-DDC 1yr 7x24 E-LTU

Option 3

4 Q0J94AAE HPE Docker CS Eng Serv Bndl 7x24 3Yr Sub

4 Q0K98AAE HPE Docker CS Eng Upg-DDC 3yr 7x24 E-LTU

Alternate Docker licensing options (pick one) for the Docker management virtual machines

Option 1

6 Q0K87AAE HPE Docker Ddc 1yr 9x5 E-LTU

Option 2

6 Q0K93AAE HPE Docker Ddc 1yr 7x24 E-LTU

Option 3

6 Q0K94AAE HPE Docker Ddc 3yr 7x24 E-LTU

Technical white paper Page 24

Note Each server or VM installed with the Docker CS Engine requires a Docker Datacenter license. At minimum, 10 Docker Datacenter licenses are required for the UCP deployment as outlined in this Reference Configuration, as six Docker Datacenter licenses are required for the management VMs (UCP swarm managers and DTR replicas) and four licenses are required for the four UCP swarm workers.

Appendix B: Deploying Docker Datacenter management nodes on HPE Hyper Converged 380 Follow the process described in the HPE Reference Configuration for Docker Datacenter on HPE Hyper Converged 380 guide to prepare to deploy the Docker Datacenter management nodes on the HPE Hyper Converged 380. Follow the guidance from that Reference Configuration including creating a shared datastore, creating an external production network, and creating a Docker image to use as a template.

Important To achieve best performance, configure all nodes to use the “direct-lvm” devicemapper storage driver configuration. This can most easily be done before running any containers, and thus, should be part of the VM image used as a template to deploy the Docker UCP swarm manager and Docker DTR nodes.

To achieve the “direct-lvm” configuration, a new partition should be added to an existing VM disk image so the partition can be used as the block device used in the direct-lvm configuration described in the Docker Device Mapper storage driver instructions. The following steps describe a simple process to add a new partition to an existing VM image:

1. Create a larger “Size” in the HPE Hyper Converged 380 User Interface that reflects how large the VM disk should be to account for the existing image size as well as the new partition.

2. Vend a VM with the existing image using your newly defined size. This will grow the VM disk to be larger than the existing image and leave some unpartitioned space on /dev/sda.

3. Log in to the VM and use fdisk /dev/sda to create a new “Linux LVM” type primary partition using the unpartitioned space. For typical RHEL 7.2 installs this will create the /dev/sda3 partition, but this may vary if the existing image used a custom disk partitioning scheme.

4. Follow the Docker Device Mapper storage driver instructions to configure direct-lvm mode on the newly created partition.

5. Follow the instructions from the HPE Hyper Converged 380 User Guide to export an OVA file for the VM and then import it as a new OVA image to make it available to vend Docker VMs as a template.

Use the HPE Hyper Converged 380 User Interface to provision or “vend” the virtual machines that will be used to provide the Docker Datacenter services. To vend new virtual machines, follow the process described in the HPE Reference Configuration for Docker Datacenter on HPE Hyper Converged 380 guide to provision the virtual machines listed in Table 5 from the Docker image. The virtual machine and hostnames used in this configuration are provided as examples only.

Table 5: VMs to vend for initial Docker Datacenter setup

Hostname Node type DRS rules (recommended) Notes

ucp HAProxy load balancer None Use your preferred Linux image with minimal packages as appropriate

dtr HAProxy load balancer None Use your preferred Linux image with minimal packages as appropriate

ucpManager1 UCP manager Keep separated from other ucpManager VMs Use your Docker Image

ucpManager2 UCP manager Keep separated from other ucpManager VMs Use your Docker Image

ucpManager3 UCP manager Keep separated from other ucpManager VMs Use your Docker Image

DTR1 UCP worker / DTR replica Keep separated from other DTR VMs Use your Docker Image

DTR2 UCP worker / DTR replica Keep separated from other DTR VMs Use your Docker Image

DTR3 UCP worker / DTR replica Keep separated from other DTR VMs Use your Docker Image

Technical white paper Page 25

Configure Docker Datacenter Management components In this section we will provide a high-level overview of the Docker Datacenter installation process on the six Docker and two load balancer virtual machines. For specific installation and configuration processes, refer to the official Docker documentation and installation guides. In our Reference Configuration, there will be three Docker Universal Control Plane manager nodes and three Docker Trusted Registry UCP worker nodes running as VMs as shown in Figure 7.

Installing Docker Universal Control Plane Management components Once the VM vending process is complete, use the vSphere Web Client or ssh client to open a session to each VM to configure them for their specific Docker Universal Control Plane (UCP) roles. Use Table 5 to note the recommended role for each node. All nodes in the solution except for the load balancers need to be joined to the UCP cluster.

Start with the load balancers (as at least the DTR load balancer must be up and functional before installing DTR). After the load balancers are up, and pre-configured with the expected addresses of the other nodes, install the Docker nodes starting with a Docker UCP manager node, ucpManager1 in this Reference Configuration. Next, vend the VMs for the remaining UCP manager nodes and join them as managers. Finally, deploy and join the DTR nodes as UCP workers.

Important Please read the full list of HPE recommendations below. While this Reference Configuration is similar, there have been updates since the HPE Reference Configuration for Docker Datacenter on HPE Hyper Converged 380 was published.

Additionally, as the HPE ProLiant DL360 servers will not be managed by the Hyper Converged 380, the cluster will not have any true worker nodes until the steps in the Utilizing HPE OneView and HPE ICsp to deploy bare-metal UCP swarm workers section are followed.

Refer to the Docker UCP installation documentation for detailed UCP manager and worker installation instructions. HPE recommends following the Docker best practices for the Docker UCP deployment including:

• To enable load balancing, the IP address and/or domain name of the UCP load balancer should be specified as a Subject Alternative Name (SAN) using the --san option when installing the UCP controller and replicas.

• The swarm scheduler should be configured to prevent users from scheduling containers on the UCP controller nodes. This can be accomplished in the Docker Universal Control Plane by selecting Settings Scheduler under Scheduler Settings and clearing the checkbox in “Allow Users to deploy containers on UCP controllers or nodes running DTR”.

Installing Docker Trusted Registry Docker Trusted Registry will be installed on cluster nodes DTR1, DTR2, and DTR3 as per Table 5. These nodes should already be joined to the Docker UCP cluster and appear as workers on the Nodes pages within the Docker UCP web user interface.

Refer to the official Docker Trusted Registry documentation for full detailed instructions for installing and configuring Docker Trusted Registry. HPE recommends following the Docker best practices for the Docker DTR deployment including:

• To enable load balancing, the IP address or domain name of the DTR load balancer node should be specified for the --dtr-external-url option when installing DTR. Additionally, as that address will be probed as part of the DTR installation process, the DTR load balancer should be installed and configured prior to installing DTR on the first node.

• Create a DTR backup and disaster recovery plan as per the Docker documentation.

Note If you are working in an environment that requires an HTTP proxy, note that the DTR install command does not read such settings from environment values of your shell or the Docker daemon configuration. The DTR install command requires proxy settings such as --http-proxy, --https-proxy, and --no-proxy to be provided directly. Failure to provide such settings may result in errors pulling images, failure communicating to UCP, or in DTR containers failing to start even though no errors are reported from the install command.

Technical white paper Page 26

UCP and DTR load balancing Two virtual machines (ucpLoadbalancer and dtrLoadbalancer) were created to act as load balancers for the Docker Universal Control Plane controller nodes and the Docker Trusted Registry cluster nodes. In our example we are using HA Proxy to provide load-balancing services. Other load balancers may be used to provide load-balancing services. For more information on configuring Docker Universal Control Plane and Docker Trusted Registry, refer to the High-availability sections for those products at https://docs.docker.com.

Appendix C: Using Docker UCP etcd for the HPE Docker Volume Plugin As per Figure 6 the HPE Docker Volume Plugin utilizes etcd to share information about HPE 3PAR backed Docker volumes across all nodes of the Docker swarm and UCP clusters. One of the advantages of UCP is that it already has instances of etcd running on the UCP swarm manager nodes with each node running a container named ucp-kv that is based on the docker/ucp-etcd image.

To enable the plugin to utilize the ucp-etcd instance of etcd, the following line should be added to the “volume:” entries in the docker-compose.yml file used to launch the plugin:

- ucp-node-certs:/root/hpedocker/python-hpedockerplugin/hpedockerplugin/ssl

The docker-compose.yml file will be included as part of cloning the plugin github repo. See https://github.com/hpe-storage/python-hpedockerplugin/blob/containerize/quick-start/docker-compose.yml. This added line mounts the UCP client certificates for etcd into a location that the plugin can utilize to authenticate into the UCP etcd instance.

Additionally, the following lines should be specified in the /etc/hpedockerplugin/hpe.conf file to enable the plugin to target the UCP etcd instance and utilize the certificates from the volume mounted by the above addition to the docker-compose.yml file:

# IP Address and port # of the etcd instance # to use for storing volume meta data host_etcd_ip_address = <IP address of a Docker UCP manager or a load balancer if configured> host_etcd_port_number = 12379 # Client certificate and key details for secured # etcd cluster host_etcd_client_cert = /root/hpedocker/python-hpedockerplugin/hpedockerplugin/ssl/cert.pem host_etcd_client_key = /root/hpedocker/python-hpedockerplugin/hpedockerplugin/ssl/key.pem host_etcd_ca_cert = /root/hpedocker/python-hpedockerplugin/hpedockerplugin/ssl/ca.pem

Note For information about the etcd instance in your cluster, you can use the Docker inspect command or click on a ucp-kv container from the Containers page in the Docker UCP UI. In particular, you can examine the value used for the --advertise-client-urls parameter to identify what value to use for the host_etcd_port_number setting in the hpe.conf file.

For demo and development use cases, the IP address of any UCP swarm manager node can be used for the host_etcd_ip_address setting in the hpe.conf file. HPE recommends configuring an additional load balancer for the etcd port (e.g., 12379) as the general UCP load balancer will only be listening for UCP UI and API traffic on port 443. Specifying the load balancer address for this setting will keep the etcd cluster available to the plugin in the event one of the manager nodes is unavailable.

Technical white paper Page 27

Sign up for updates

© Copyright 2016 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. Red Hat is a registered trademark of Red Hat, Inc. in the United States and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Java is registered trademark of Oracle and/or its affiliates.

4AA6-8758ENW, November 2016

Resources and additional links HPE Hyper Converged systems, hpe.com/info/hyperconverged

HPE Docker Volume plugin, https://github.com/hpe-storage/python-hpedockerplugin

HPE Operations Bridge, hpe.com/software/opsbridge

Docker Datacenter product information, docker.com/products/docker-datacenter

Manually installing Docker CS Engine, http://docs.docker.com/cs-engine/install

Docker Trusted Registry documentation, https://docs.docker.com/datacenter/dtr/2.1/guides

Docker Universal Control Plane documentation, https://docs.docker.com/datacenter/ucp/2.0/guides

Plan a Production Docker UCP Installation,

• https://docs.docker.com/datacenter/ucp/2.0/guides/installation/plan-production-install

• https://docs.docker.com/datacenter/ucp/2.0/guides/configuration/integrate-with-dtr

Docker Service Discovery and Load Balancing, docker.com/sites/default/files/RA_UCP%20Load%20Balancing-Feb%202016_0.pdf

HA Proxy, haproxy.org

HPE Reference Architectures, hpe.com/info/ra

HPE Servers, hpe.com/servers

HPE Storage, hpe.com/storage

HPE Networking, hpe.com/networking

HPE Technology Consulting Services, hpe.com/us/en/services/consulting.html

To help us improve our documents, please provide feedback at hpe.com/contact/feedback