Dell EMC Ready Architecture for Red Hat OpenShift ... · Dell EMC Ready Architecture for Red Hat OpenShift Container Platform v3.11 7 Deployment Guide For more information about OpenShift

Dell EMC Ready Architecture for Red Hat OpenShift Container Platform v3.11 April 2019

H17702

Deployment Guide

Abstract

This deployment guide describes how to create a Red Hat OpenShift Container Platform environment on Dell EMC infrastructure for a highly available production deployment. The guide includes sample configurations and modifiable Ansible scripts.

Dell EMC Solutions

Copyright

2 Dell EMC Ready Architecture for Red Hat OpenShift Container Platform v3.11

Deployment Guide

The information in this publication is provided as is. Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable software license.

Copyright © 2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. Other trademarks may be trademarks of their respective owners. Published in the USA 04/19 Deployment Guide H17702.

Dell Inc. believes the information in this document is accurate as of its publication date. The information is subject to change without notice.

Contents

3 Dell EMC Ready Architecture for Red Hat OpenShift Container Platform v3.11 Deployment Guide

Contents

Chapter 1 Introduction 5

Solution overview .................................................................................................. 6

Document purpose ................................................................................................ 6

Audience ............................................................................................................... 7

We value your feedback ........................................................................................ 7

Chapter 2 Before You Start 8

Introduction ........................................................................................................... 9

Requirements and assumptions ............................................................................ 9

Cabling ................................................................................................................ 10

Preinstallation steps ............................................................................................ 11

Chapter 3 Configuring the Switches 14

Introduction ......................................................................................................... 15

Customizing sample files..................................................................................... 15

Customizing the Dell EMC Networking switches ................................................. 17

Chapter 4 Installing and Preparing the Bastion Node 20

Introduction ......................................................................................................... 21

Preparing the installation ..................................................................................... 21

Installing Red Hat Enterprise Linux 7.6 on the bastion node .............................. 22

Configuring the bastion node .............................................................................. 25

Chapter 5 Preparing Cluster Servers for OpenShift Installation 26

Introduction ......................................................................................................... 27

Preparing the cluster nodes ................................................................................ 27

Chapter 6 Deploying the OpenShift Cluster 29

Introduction ......................................................................................................... 30

Preparing for deployment .................................................................................... 30

Deploying the cluster ........................................................................................... 32

Chapter 7 Creating OpenShift User Accounts 33

Introduction ......................................................................................................... 34

Logging in to OpenShift ....................................................................................... 34

Creating user accounts and setting cluster-admin roles ...................................... 34

Chapter 8 Creating the Default Storage Class 37

Introduction ......................................................................................................... 38

Contents


Creating the default storage class ....................................................................... 38

Chapter 9 Validating the OpenShift Container Platform Deployment 39

Introduction ......................................................................................................... 40

Validating the deployment ................................................................................... 40

Chapter 10 Validating OpenShift Operation 47

Introduction ......................................................................................................... 48

Deploying a sample application: ruby-ex ............................................................. 48

Deploying a sample application: Lab-Guide-Workshop ....................................... 51

Appendix A Sample Switch Configuration Files 57

Introduction ......................................................................................................... 58

Downloading the sample files .............................................................................. 58

Modifying the deployment control file .................................................................. 61

Switch configuration file examples ...................................................................... 65

Appendix B All-In-One Installation 100

Introduction ....................................................................................................... 101

Deploying OpenShift All-in-One using a single server ....................................... 101

Installing Red Hat Enterprise Linux 7.6 ............................................................. 102

Creating Docker storage ................................................................................... 104

Installing OpenShift 3.11 AIO on a PowerEdge R640 server ............................. 105

Deploying a sample Kubernetes application ...................................................... 110

Chapter 1: Introduction


Chapter 1 Introduction

This chapter presents the following topics:

Solution overview ............................................................................................... 6

Document purpose ............................................................................................. 6

Audience .............................................................................................................. 7

We value your feedback ..................................................................................... 7



Solution overview

Red Hat OpenShift Container Platform is an open-source application deployment platform

that is based on Kubernetes container orchestration technology. Containers are

standalone processes that run within their own environment, independent of the operating

system and underlying infrastructure. Red Hat OpenShift Container Platform helps you

develop, deploy, and manage container-based applications.

As part of Red Hat OpenShift Container Platform, Kubernetes manages containerized

applications across a set of containers or hosts and provides mechanisms for the

deployment, maintenance, and scaling of applications. The container run-time engine

packages, instantiates, and runs containerized applications.

A Kubernetes cluster consists of one or more masters and a set of nodes. Kubernetes

allocates to each pod an IP address from an internal network so that all containers within

the pod behave as if they were on the same host. Giving each pod its own IP address

means that pods can be treated like physical hosts or virtual machines for port allocation,

networking, naming, service discovery, load balancing, application configuration, and

migration. Dell EMC recommends creating a service that enables your pods to interact

rather than letting the pods talk to one another directly using their IP addresses.

A fully functioning Domain Name System (DNS) is crucial in the deployment and operation

of your container ecosystem. Red Hat OpenShift Container Platform has a built-in DNS so

that the services can be found through DNS switched virtual circuit (SVC) entries or

through the service IP/port registrations.

The Dell EMC Ready Architecture for Red Hat OpenShift Container Platform is a proven

design to help companies accelerate their container deployments and cloud-native

adoption. Dell EMC delivers tested, validated, and documented design guidance to help

customers rapidly deploy Red Hat OpenShift on Dell EMC infrastructure by minimizing

time and effort. For details, see the architecture guide on the OpenShift Container

Platform Info Hub for Ready Solutions.

Document purpose

This deployment guide describes the required infrastructure for deploying and operating

Red Hat OpenShift Container Platform and the validated deployment process for a

production-ready OpenShift Container Platform cluster. It provides the information that

you need to facilitate readiness for day 2 operations. Topics include:

• Containers and images

• Pods and services

• Projects and users

• Builds and image streams

• Deployments

• Routes

• Templates

https://docs.openshift.com/container-platform/3.11/architecture/core_concepts/pods_and_services.html#services

https://community.emc.com/docs/DOC-71309




For more information about OpenShift Container Platform, see the Overview on the

OpenShift website.

This guide provides a reference design for deploying version 3.11 of Red Hat OpenShift

Container Platform on Dell EMC PowerEdge servers and Dell EMC Networking switches.

Dell EMC strongly recommends that you complete the validation steps that are described

in this guide and ensure that you are satisfied that your application will operate smoothly

before proceeding with development or production use.

Audience

This deployment guide is for system administrators and system architects. Read the

solution reference architecture guide to familiarize yourself with the solution architecture

and design before planning your deployment. Some experience with Docker and Red Hat

OpenShift Container Platform technologies is recommended.

Appendix B All-in-One Installation provides basic instructions for installing a single-

node deployment of Red Hat OpenShift Container Platform. Red Hat refers to this

single-node deployment as “All-in-One” (AIO). These instructions are provided to

enable you to test-drive the deployment before starting the process of installing a full

high availability (HA) cluster.

We value your feedback

Dell EMC and the authors of this document welcome your feedback on the solution and

the solution documentation. Contact the Dell EMC Solutions team by email or provide

your comments by completing our documentation survey.

Authors: Michael Tondee, Scott Powers, John Terpstra, Aighne Kearney

Note: OpenShift Container Platform Info Hub for Ready Solutions on the Dell EMC Communities

website provides links to additional documentation for this solution.

https://docs.openshift.com/container-platform/3.11/architecture/core_concepts/index.html

mailto:[email protected]?subject=Feedback:%20Red%20Hat%20OpenShift%20Container%20Platform%20v3.11%20Deployment%20Guide%20(H17702)

https://www.surveymonkey.com/r/SolutionsSurveyExt


Chapter 2: Before You Start


Chapter 2 Before You Start


Introduction ......................................................................................................... 9

Requirements and assumptions ........................................................................ 9

Cabling ............................................................................................................... 10

Preinstallation steps ......................................................................................... 11



Introduction

This chapter provides information about the typographical conventions that are used in

this guide and describes the planning and preparation that you must complete before you

deploy the Red Hat OpenShift Container Platform cluster.

This guide uses certain typographical conventions to mark commands and command

output:

• Information that is specific to your environment is placed inside <> symbols and

presented in italics. For example: OS10(config)# hostname <hostname>

• Command output is presented with a grey background.

Before you begin, ensure that you have reviewed the following questions:

• Do you have the correct Dell EMC servers and switches and are they equipped with

the resources necessary to support your production workload?

Note the following requirements:

▪ Use only Dell EMC PowerEdge R640 or PowerEdge R740xd server nodes. Images

are tagged during Red Hat OpenShift Container Platform deployment, and the

deployment scripts cannot differentiate between mixed systems for a given node

type.

▪ Manually install and update the image streams. Image streams and templates

are not installed by default or updated when you upgrade the system.

▪ Install on-premises PowerEdge servers. Do not attempt to install OpenShift

Container Platform on nodes from a cloud provider.

• How many pods do you need to deploy and run concurrently within the cluster?

See Planning your installation on the OpenShift website.

• How many server nodes will you require in the cluster?

See Planning your installation on the OpenShift website.

• How do you plan to monitor your cluster?

See Prometheus Cluster Monitoring on the OpenShift website.

Before starting the deployment, read the following documents:

• Dell EMC Red Hat Intel OpenShift Architecture and Deployment Guide (includes

hardware design and ordering information)

• Relevant sections of the OpenShift Container Platform 3.11 Documentation

Requirements and assumptions

The customer environment must have:

▪ Red Hat Enterprise Linux Server subscriptions with OpenShift-3.11 Enterprise

▪ RHEL-7.6-20181010.0-Server-x86_64-dvd1.iso

Typographical

conventions

Installation and

configuration

checklist

Essential

reading

https://docs.openshift.com/container-platform/3.11/install/index.html

https://docs.openshift.com/container-platform/3.11/install/index.html

https://docs.openshift.com/container-platform/3.11/install_config/prometheus_cluster_monitoring.html#prometheus-cluster-monitoring


https://docs.openshift.com/container-platform/3.11/welcome/index.html



▪ Eleven PowerEdge R640 servers with integrated Dell Remote Access Controller

(iDRAC) Enterprise licenses

▪ Four PowerEdge R740xd servers with iDRAC Enterprise licenses

▪ Static IP addresses set for iDRAC NICs and cabled to the Dell EMC Networking

S3048-ON switch that is uplinked to the data switches

▪ Between two and four 10 GbE or 25 GbE NICs

Note: This Ready Architecture uses four 25 GbE NICs in bond mode=4.

▪ Switches supporting LACP port channels with untagged and tagged VLANs on the

same interface

The Dell EMC Ready Architecture for Red Hat OpenShift Container Platform uses Dell

EMC Networking S5232F-ON switches. These switches are referred to as S5232F-1

and S5232F-2 in this guide.

▪ DNS wildcard entries for the *.DomainX that resolve to the openshift_cluster_ip

address in /etc/ansible/hosts

▪ Internet access to download packages and updates

The deployment procedures in this guide assume that:

• Servers and networking switches are racked, and power and network cabling has

been completed in accordance with the instructions in the architecture guide.

• Static IP addresses are set on the iDRAC NICs, and the NICs are cabled to the

S3048-ON switch.

Cabling

Cable the servers to switches, spreading a port from each dual-port adapter across the

network switches. The following table lists the recommended cabling between the NICs on

the server network daughter card and the switches.

Table 1. Server/switch wiring recommendations for integrated NICs

Server S5232F-1 S5232F-2 Switch

Ethernet port

Switch port

channel

Switch untagged

VLAN

Switch tagged VLAN

bastion em1 em2 1/1/1:1 1 20 10, 421

master1 em1 em2 1/1/1:2 2 20

master2 em1 em2 1/1/1:3 3 20

master3 em1 em2 1/1/1:4 4 20

infra1 em1 em2 1/1/2:1 5 20 421

infra2 em1 em2 1/1/2:2 6 20 421

infra3 em1 em2 1/1/2:3 7 20 421

app1 em1 em2 1/1/2:4 8 20



Server S5232F-1 S5232F-2 Switch

Ethernet port

Switch port

channel

Switch untagged

VLAN

Switch tagged VLAN

app2 em1 em2 1/1/3:1 9 20

app3 em1 em2 1/1/3:2 10 20

app4 em1 em2 1/1/3:3 11 20

stor1 em1 em2 1/1/3:4 12 20

stor2 em1 em2 1/1/4:1 13 20

stor3 em1 em2 1/1/4:2 14 20

stor4 em1 em2 1/1/4:3 15 20

The following table shows the recommended cabling for the PCIe NICs.

Table 2. Server/switch wiring recommendations for add-in NICs

Server S5232F-1 S5232F-2

Switch

Ethernet port

Switch port

channel

Switch untagged

VLAN

Switch tagged

VLAN

bastion p2p1 p2p2 1/1/9:1 1 20 10, 421

master1 p2p1 p2p2 1/1/9:2 2 20

master2 p2p1 p2p2 1/1/9:3 3 20

master3 p2p1 p2p2 1/1/9:4 4 20

infra1 p2p1 p2p2 1/1/10:1 5 20 421

infra2 p2p1 p2p2 1/1/10:2 6 20 421

infra3 p2p1 p2p2 1/1/10:3 7 20 421

app1 p2p1 p2p2 1/1/10:4 8 20

app2 p2p1 p2p2 1/1/11:1 9 20

app3 p2p1 p2p2 1/1/11:2 10 20

app4 p2p1 p2p2 1/1/11:3 11 20

stor1 p7p1 p7p2 1/1/11:4 12 20

stor2 p7p1 p7p2 1/1/12:1 13 20

stor3 p7p1 p7p2 1/1/12:2 14 20

stor4 p7p1 p7p2 1/1/12:3 15 20

Preinstallation steps

Before starting the installation, complete these steps:

1. From this github repository, download the sample scripts that Dell EMC has

provided for the installation.

https://github.com/dell-esg/openshift-bare-metal/tree/master/examples



2. Modify the S5232F-1, S5232F-2, and S3048 files to match your networks and

infrastructure, and save the modified files.

3. Configure each of the switches using the modified files that you saved in the

preceding step.

4. Install the bastion node by using iDRAC remote console with Virtual Media

attached to RHEL-7.6-20181010.0-Server-x86_64-dvd1.iso.

5. On the bastion node:

Register Red Hat Enterprise Linux 7.6 and attach the Red Hat pool ID with

OpenShift subscriptions.

Install the packages and Red Hat Enterprise Linux 7.6 updates, and then

reboot and prepare RHEL 7.6 iso.

Run the following command:

# cd /root

# git clone https://github.com/dell-esg/openshift-bare-

metal/

6. Modify examples/hosts.example.com to match your infrastructure, and then

copy the file to the bastion node file /etc/ansible/hosts.

Note: Your Dell Service Tags are required in the updated hosts file.

7. From the bastion node, run the Ansible scripts that install Red Hat Enterprise

Linux 7.6 and OpenShift 3.11 onto the other 14 PowerEdge servers:

# ansible-playbook src/prerequisites/nodes_setup.yaml -k

# ansible-playbook src/keepalived-

multimaster/keepalived.yaml

# ansible-playbook /usr/share/ansible/openshift-

ansible/playbooks/prerequisites.yml

# ansible-playbook /usr/share/ansible/openshift-

ansible/playbooks/deploy_cluster.yml

# ansible-playbook src/ocs-pool/storage-class.yml

8. Log in as user system:admin, and then:

Create new OpenShift users.

Set htpasswd passwords for the users.

Add the users to the cluster-admin role.

9. Deploy a sample application and validate that the application routes are

accessible on the Red Hat OpenShift Container Platform public network.



To uninstall OpenShift dependencies, run the following command:

ansible-playbook /usr/share/ansible/openshift-

ansible/playbooks/adhoc/uninstall.yml

To clean glusterfs disks using values from /etc/ansible/hosts, run the following

command:

ansible-playbook -i /etc/ansible/hosts -e

"openshift_storage_glusterfs_wipe=True" -e

"openshift_storage_glusterfs_registry_wipe=true"

/usr/share/ansible/openshift-ansible/playbooks/openshift-

glusterfs/uninstall.yml

Troubleshooting

Chapter 3: Configuring the Switches


Chapter 3 Configuring the Switches


Introduction ....................................................................................................... 15

Customizing sample files ................................................................................. 15

Customizing the Dell EMC Networking switches ............................................ 17



Introduction

This chapter describes how to customize the Dell EMC-provided sample configuration

files for your environment.

The sample configuration files and Ansible scripts that we used to validate the deployment

are available in a github repository. These files enable you to easily configure multiple

complex OpenShift 3.11 components. Before a production installation, you can modify

these files and scripts, in consultation with Dell EMC and Red Hat, to meet your

requirements.

CAUTION: If you use different hardware or need different configurations, you must modify the

scripts to ensure proper solution installation and functioning.

Customizing sample files

To customize sample files:

1. Browse to the github repository and select Current Branch: 3.11.

2. Look for the following directory:

/root/openshift-bare-metal/examples/

The /root/openshift-bare-metal/examples/example.xlsx file

contains the server and switch details that are used to document changes in your

installation.

3. Download the sample files and modify them as necessary to match your

environment.

Alternatively, you can clone the github.com repository from an existing Linux

system by running the following command:

cd /root

git clone https://github.com/dell-esg/openshift-bare-metal

4. Save the modified files.

Note:

▪ Kickstart examples are modified and used to install Red Hat Enterprise Linux

7.6 on the bastion and AIO nodes.

▪ Switch examples are modified, copied to running-configuration, and saved to

each switch, enabling you to quickly configure the three Dell EMC Networking

switches for the example.com deployment.

The following table on the Inventory tab in the example.xlsx file shows the

names, IP addresses, and switch details for your deployment.

https://github.com/dell-esg/openshift-bare-metal/


https://github.com/dell-esg/openshift-bare-metal



Table 3. Ready Architecture deployment servers

Name Hardware iDRAC VLAN-10 192.168.10.0/24 gw:192.168.10.1

Internal VLAN-20 192.168.20.0/24 gw:192.168.20.1

Public VLAN-421 100.82.42.0/24 gw:100.82.42.1

DNS: 100.82.42.8

W2008R2 VM 192.168.10.8 192.168.20.8 100.82.42.8

Existing Linux system

Any system 100.82.42.9

RHEL01 Bastion

R640#1 192.168.10.11 192.168.20.11 100.82.42.11

RHEL02 master1

R640#2 192.168.10.12 192.168.20.12

RHEL03 master2

R640#3 192.168.10.13 192.168.20.13

RHEL04 master3

R640#4 192.168.10.14 192.168.20.14

RHEL05 infra1

R640#5 192.168.10.15 192.168.20.15 100.82.42.15

RHEL06 infra2

R640#6 192.168.10.16 192.168.20.16 100.82.42.16

RHEL07 infra3

R640#7 192.168.10.17 192.168.20.17 100.82.42.17

RHEL08 app1

R640#8 192.168.10.18 192.168.20.18

RHEL09 app2

R640#9 192.168.10.19 192.168.20.19

RHEL10 app3

R640#10 192.168.10.20 192.168.20.20

RHEL11 app4

R640#11 192.168.10.21 192.168.20.21

RHEL12 stor1

R740xd#1 192.168.10.22 192.168.20.22

RHEL13 stor2

R740xd#2 192.168.10.23 192.168.20.23

RHEL14 stor3

R740xd#3 192.168.10.24 192.168.20.24

RHEL15 stor4

R740xd#4 192.168.10.25 192.168.20.25

S5232F-1 S5232F-ON vrrp: 192.168.10.1 vrrp: 192.168.20.1

S5232F-2 S5232F-ON vrrp: 192.168.10.1 vrrp: 192.168.20.1

5-S3048 S3048-ON



The following table shows the recommended processor model, memory capacity, and

firmware versions that we used in the installation.

Table 4. Recommended firmware versions

CPU model (all nodes)

Memory at 2,666 MHz

Dell EMC BIOS version

Dell EMC iDRAC and LifeCycle versions

Mellanox NIC firmware version

Dell EMC BOSS firmware version

2 x Intel Xeon Gold 6138

384 GB 1.6.12 3.21.26.22 14.23.10.20 2.5.13.3016

Customizing the Dell EMC Networking switches

To modify the sample files:

1. Update the S5232F-1 and S5232F-2 values to match your VLANs.

2. Place the modified S5232F-1 and S5232F-2 files on an existing Linux server that

can be reached on the switch out-of-band (OOB) management network.

3. Change the public VLAN 421 and network CIDR to a routable network in your

infrastructure that has Internet access.

4. Modify the switch OOB management network VLAN and CIDR to match the

switch OOB management network at your site.

The following table shows the networks and settings that we used in the ocp.example.com

installation.

Table 5. Sample installation networks and settings

VLAN number

Network Purpose Gateway location

10 192.168.10.0/24 iDRAC S5232F via VRRP= 192.168.10.1

20 192.168.20.0/24 Internal S5232F via VRRP= 192.168.20.1

421 100.82.42.0/24 Public External= 100.82.42.1

33 100.82.33.0/24 Switch management

network

External= 100.82.33.1

(management route 0.0.0.0/0 for each Dell EMC Switch)

The initial switch configuration settings are:

• Dell Networking serial port baud-rate speed: 115200

• Dell Networking OS10 Enterprise default passwords switch login:

▪ Username: admin

▪ Password: admin

Modifying the

sample files

Configuring the

switches



To perform the configuration:

1. Copy the updated switch configuration files to each switch and save the

configuration.

2. Power on the switches, connect to the serial debug port, and set a static IP

address.

3. Log in to each Dell EMC Networking switch, set the static IP address for the

switch OOB management port, copy the switch examples to running-

configuration, and save the switch configuration.

The following code sample shows switch S5232F-1:

S5232F-1# configure terminal

S5232F-1(config)# interface mgmt 1/1/1

S5232F-1(conf-if-ma-1/1/1)# no shutdown

S5232F-1(conf-if-ma-1/1/1)# no ip address dhcp

S5232F-1(conf-if-ma-1/1/1)# ip address 100.82.33.81/24

S5232F-1(conf-if-ma-1/1/1)# exit

S5232F-1(config)# management route 0.0.0.0/0 100.82.33.1

S5232F-1# copy scp://root@hostip/root/openshift-bare-

metal/examples/S5232F-1 running-configuration

S5232F-1# write memory

S5232F-1# reload

S5232F-2 switch

The following code sample shows how to configure the S5232F-2 switch:

S5232F-2# configure terminal

S5232F-2(config)# interface mgmt 1/1/1

S5232F-2(conf-if-ma-1/1/1)# no shutdown

S5232F-2(conf-if-ma-1/1/1)# no ip address dhcp

S5232F-2(conf-if-ma-1/1/1)# ip address 100.82.33.82/24

S5232F-2(conf-if-ma-1/1/1)# exit

S5232F-2(config)# management route 0.0.0.0/0 100.82.33.1

S5232F-2# copy scp://root@hostip/root/openshift-bare-

metal/examples/S5232F-2 running-configuration

S5232F-2# write memory



S5232F-2# reload

S3048 switch

The following code sample shows how to configure the S3048 switch:

S3048# configure terminal

S3048(config)# interface mgmt 1/1

S3048(conf-if-ma-1/1/1)# no shutdown

S3048(conf-if-ma-1/1/1)# ip address 100.82.33.83/24

S3048(conf-if-ma-1/1/1)# exit

S3048(config)# management route 0.0.0.0/0 100.82.33.1

S3048# copy scp://root@hostip//root/openshift-bare-

metal/examples/S3048 running-configuration

S3048# write memory

S3048# reload

Chapter 4: Installing and Preparing the Bastion Node


Chapter 4 Installing and Preparing the Bastion Node


Introduction ....................................................................................................... 21

Preparing the installation ................................................................................. 21

Installing Red Hat Enterprise Linux 7.6 on the bastion node ........................ 22

Configuring the bastion node .......................................................................... 25



Introduction

This chapter describes how to install Red Hat Enterprise Linux 7.6 on the bastion node

using automated kickstart files and the iDRAC remote console and how to configure the

bastion node for deployment.

Preparing the installation

Before you begin installing the Red Hat OpenShift Container Platform on your cluster, you

must install and configure the bastion node, which launches the next steps in the

deployment process. Choose either of the following options to use kickstart with the

iDRAC remote console.

To prepare and use an image file for use with the iDRAC remote console:

1. From an existing Red Hat Enterprise Linux 7.6 system, create a USB image by

running the following command:

mkfs.vfat -C ks_usb.img 1024

2. Mount the image by running the following command:

mount -o loop ks_usb.img /mnt

3. Place the customized bastion.ks file into the image by running the following

command:

cp bastion.ks /mnt

4. Sync and unmount the image by running the following command:

sync; umount /mnt

5. Copy ks_usb.img to the machine that will log in to the server’s iDRAC.

During the installation, you will attach this file to removable media in the iDRAC

remote console’s Virtual Media.

To prepare a USB key for use with the iDRAC remote console:

1. From an existing Red Hat Enterprise Linux 7.6 system, format a USB key by


mkfs.ext3 /dev/sdb

2. Mount the USB key:

mount /dev/sdb /mnt

3. Place the customized sample bastion.ks file onto the USB key:

cp bastion.ks /mnt


sync; umount /mnt

Option 1: Using

an image file

Option 2: Using

a USB key



5. Plug the USB key into the bastion server before you power it on, and then log in

to the server’s iDRAC.

During the operating system installation, you will select the USB location to the

bastion.ks on the USB key in the iDRAC remote console.

Installing Red Hat Enterprise Linux 7.6 on the bastion node

To install Red Hat Enterprise Linux 7.6 on the bastion node:

1. Log in to the iDRAC web interface of the R640 PowerEdge server and launch a

remote console.

2. On the Virtual Media tab, connect virtual media and select Map CD/DVD.

3. Browse to the location of the Red Hat Enterprise Linux Server 7.6 installation

media and select Map Device.

4. Use either the ks_usb.img file (Option 1) or the USB key you prepared earlier

(Option 2):

▪ Option 1: In the iDRAC remote console, on the Virtual Media tab, select Map

Removable Disk. Browse to the location of the ks_usb.img file and select

Map Device.

▪ Option 2: Insert the USB key into a USB port on the R640 server.

sda = BOSS RAID1 as OS drive

sdb = ks_usb_img mounted to iDRAC

5. Select Next Boot, set Virtual CD/DVD/ISO as the next boot option.

6. Power on the system and click OK on the remote console.

7. From the Red Hat Enterprise Linux 7.6 installation menu, select Install.

8. Do not press the [Enter] key. Press the [e] key.

9. Move the cursor to the end of the line that begins with vmlinuz, and append the

following code to the end of the line:

ks=hd:sdb:/bastion.ks

Note: If only one physical RAID, SAS, or SATA disk is presented to the server, use the

device name sdb for the kickstart location in the Red Hat Enterprise Linux 7 installer.

Note: NVMe disks are enumerated as nvme0n1, nvme1n1, nvme2n1, and nvme3n1.

The following figure shows the installation command.



Figure 1. Installation command

10. Press [CTRL+X] to start the installation.

11. Reboot the server and accept the license.

12. Log in and verify that the bonds and networks are functioning.

13. Start an SSH session to the bastion server.

14. To register with Red Hat, attach the pool ID with OpenShift subscriptions, and

install the supporting packages, run the following commands:

subscription-manager register --username Your_RH_account --

password RH_password --force

subscription-manager attach --

pool=xxxxxxxxxxxxxxxxxxxxxxxxxxxx

subscription-manager repos --enable="rhel-7-server-rpms" --

enable="rhel-7-server-extras-rpms" --enable="rhel-7-server-

ose-3.11-rpms" --enable="rhel-7-server-ansible-2.6-rpms"

yum -y install wget git net-tools bind-utils iptables-

services bridge-utils bash-completion kexec-tools sos psacct

yum -y install openshift-ansible

yum -y install docker-1.13.1



If partitions with previous data exist on the disks, clean the disks by running the following

command:

dd if=/dev/zero of=/dev/nvmeXn1 bs=1M count=1

Replace X with the target drive number such as 0, 1, 2, 3.

Example: dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=1

Start an SSH session to the bastion node, and then follow these steps:

1. Run the following commands as root:

cd /etc/sysconfig/

vi docker-storage-setup

2. Edit this file as follows:

cat /etc/sysconfig/docker-storage-setup

VG=docker-vg

WIPE_SIGNATURES=true

3. Save your changes.

4. Configure Docker storage by running the following command:

docker-storage-setup

Rounding up size to full physical extent 1.49 GiB

Thin pool volume with chunk size 512.00 KiB can address at

most 126.50 TiB of data. Logical volume "docker-pool"

created. Logical volume docker-vg/docker-pool changed.

Enable and start Docker by running the following commands:

systemctl enable docker

systemctl start docker

systemctl is-active docker

5. Log in to the Docker registry and install atomic by running the following

commands:

docker login -u Your_RH_account -p Your_RH_password

docker login https://registry.redhat.io

yum -y install atomic

yum -y install atomic-openshift-clients

mkdir /etc/dnsmasq.d

6. Install the Red Hat Enterprise Linux updates and reboot the bastion server:

yum clean all

yum makecache

yum -y update

reboot

Preparing the

node for

OpenShift

installation

Creating Docker

storage for the

bastion node



Configuring the bastion node

The Dell EMC Ready Architecture for the Red Hat OpenShift Container Platform uses Red

Hat Enterprise Linux (Red Hat Enterprise Linux) as the bare-metal operating system for all

nodes. Setting up the bastion node involves obtaining the Red Hat Enterprise Linux 7.6

ISO image from the Red Hat customer portal and transferring it to the bastion node. For

automated provisioning, copy the image content to the appropriate directories.

Follow these steps:

1. Copy the Red Hat Enterprise Linux 7.6 ISO image to the bastion /root

directory, mount ISO, and then copy the contents to /tftp/rhel/media/ by

running the following commands:

cd /root/

mkdir -p /mnt/media

mount -o loop rhel-server-7.6-x86_64-dvd.iso /mnt/media

mkdir -p /tftp/rhel/media

cd /mnt/media

cp -r * /tftp/rhel/media/

2. On the bastion node, log in and git clone the github repo to /root/as follows:

cd /root

git clone https://github.com/dell-esg/openshift-bare-metal/

3. From the bastion node, run the Ansible script that creates iPXE, dnsmasq, and

the tftp containerized resources used to install Red Hat Enterprise Linux 7.6 on

the remaining servers:

cd /root/openshift-bare-metal

ansible-playbook src/ipxe-deployer/ipxe.yml

4. Run the docker ps command to verify that two Docker containers are running,

which indicates that the setup was successful.

5. Edit /etc/ansible/hosts file, remove the variable

openshift_hostname=xxxx, and then run the remaining Ansible scripts for the

remainder of the deployment.

Note: The Ansible script ipxe.yml requires the variable <openshift_hostname>.

However, the remaining Ansible scripts do not accept the <openshift_hostname>

variable in [NODES] section of the /etc/ansible/hosts file.

https://github.com/dell-esg/openshift-bare-metal

Chapter 5: Preparing Cluster Servers for OpenShift Installation


Chapter 5 Preparing Cluster Servers for OpenShift Installation


Introduction ....................................................................................................... 27

Preparing the cluster nodes ............................................................................. 27



Introduction

Afer you have configured the bastion node, you can prepare to install Red Hat Enterprise

Linux L 7.6 on all the cluster nodes and then deploy Red Hat OpenShift Container

Platform to each node.

This section describes how to install Red Hat Enterprise Linux 7.6 by pre-boot execution

(PXE) booting the servers. The process involves opening iDRAC remote consoles and

validating that:

• Each server completes the Red Hat Enterprise Linux 7.6 installation process

• The network interfaces are set up correctly

Preparing the cluster nodes

Prepare the cluster nodes as follows:

1. For each server, log in to the iDRAC website and open an iDRAC remote console.

2. Set Next boot to PXE.

3. Power on each server and monitor the progress of the automated Red Hat

Enterprise Linux 7.6 PXE installation on each server.

After the PXE installation is complete, the server reboots. Each server returns to

the login prompt and all network interfaces are configured, as shown in the

following figure.

Figure 2. Red Hat Enterprise Linux 7.6 post-installation page



4. After the Red Hat Enterprise Linux 7.6 installation is complete, ping each server

on the internal IP address from the bastion node to verify that the installation was

successful.

1. If a server fails to complete the PXE installation, reboot the server and retry the

installation.

2. If failure persists, check your settings.

3. If you find mistakes in your hosts file, re-run ipxe.yml to update the kickstart

files on the bastion node.

You can find the kickstart files for each node on the bastion node at

/tftp/rhel/media/.

Troubleshooting

Chapter 6: Deploying the OpenShift Cluster


Chapter 6 Deploying the OpenShift Cluster


Introduction ....................................................................................................... 30

Preparing for deployment ................................................................................. 30

Deploying the cluster ........................................................................................ 32



Introduction

This chapter describes how to prepare your systems for the OpenShift Platform

deployment and start deploying the cluster.

Preparing for deployment

Prepare for the deployment as follows:

1. From the bastion node, run the Ansible nodes_setup.yaml script:

cd /root/openshift-bare-metal/

ansible-playbook src/prerequisites/nodes_setup.yaml -k

2. When prompted, enter the root password for the nodes in your

/etc/ansible/hosts file.

The following figure shows the resulting output.

Figure 3. Ansible nodes_setup.yaml script start

3. From the bastion node, to enable the SSH prompt on all nodes, run the

sshyestoall.sh command:


chmod 755 *.sh

./sshyestoall.sh

You can now perform SSH from the bastion node to all nodes without SSH

prompting.



Red Hat recommends using an external load balancer such as F5. For simplicity, the Dell

EMC Ready Architecture uses keepalived or haproxy on infra nodes. Consult Dell

EMC and Red Hat about any additional considerations affecting your environment.

To set up HA:

1. On the bastion node, in the Red Hat Ansible Automation inventory file at

/etc/ansible/hosts, define the following variables:

external_interface=bond0

external_vlan=421

internal_interface=bond0

openshift_master_cluster_ip=192.168.20.99

openshift_master_cluster_public_ip=100.82.42.99

2. From the bastion node, run the Ansible script keepalived.yaml as follows:

cd /root/openshift-bare-metal/ansible-playbook

src/keepalived-multimaster/keepalived.yaml

The following figure shows the completed keepalived.yaml with putty to infra1 tail

and -f /var/log/messages.

Figure 4. Completed keepalived.yaml script

The expected output is:

keepalived.sh exited with status 7

Note: The keepalived daemon will start working during cluster deployment.

Setting up high

availability



3. From the bastion node, run prerequisites.yml to install OpenShift

prerequisites to all the nodes:



ansible/playbooks/prerequisites.yml

Deploying the cluster

From the bastion node, run deploy_cluster.yml to automatically deploy Red Hat

OpenShift Container Platform:



ansible/playbooks/deploy_cluster.yml

As shown in the following figure, the scripts create OpenShift 3.11 resources using the Ansible playbooks and values from your /etc/ansible/hosts file.

Figure 5. OpenShift 3.11 deployment script

Chapter 7: Creating OpenShift User Accounts


Chapter 7 Creating OpenShift User Accounts


Introduction ....................................................................................................... 34

Logging in to OpenShift ................................................................................... 34

Creating user accounts and setting cluster-admin roles ............................... 34



Introduction

Default login files are created on the master nodes at /root/.kube/config during the

installation. You can log in and create new OpenShift users and then add the new users to

groups or cluster roles using the system: admin account.

Logging in to OpenShift

From the bastion node, start an SSH session to any master node and log in to OpenShift:

ssh master1

cd /root/.kube/

oc login -u system:admin

The following output appears:

Logged into "https://r5a.local:8443" as "system:admin" using

existing credentials.

You have access to the following projects and can switch

between them with 'oc project <projectname>':

app-storage

* default

infra-storage

kube-public

kube-service-catalog

kube-system

management-infra

openshift

openshift-ansible-service-broker

openshift-console

openshift-infra

openshift-logging

openshift-metrics-server

openshift-monitoring

openshift-node

openshift-sdn

openshift-template-service-broker

openshift-web-console

Using project "default".

Creating user accounts and setting cluster-admin roles

Create user accounts as follows. The examples in these steps show how to create

accounts for dellemc1, kubeflow, alice, mike, and john. You can substitute any valid

OpenShift user names.

1. Run the following command:

oc create user dellemc1

oc create user kubeflow

oc create user alice

oc create user mike



oc create user john

The output displays the user accounts that you created:


user.user.openshift.io/alice created

create user dellemc1

user.user.openshift.io/dellemc1 created

2. To obtain a list of OpenShift users, run the following command:

oc get users

The output displays the user details:

NAME UID FULL NAME

alice 9b20286-48d0-11e9-95d5-98039b58ae12

dellemc1 5c95e3a9-48d0-11e9-95d5-98039b58ae12

john 6d23c350-48d0-11e9-95d5-98039b58ae12

kubeflow 6a59b12d-48d0-11e9-95d5-98039b58ae12

mike 6049b237-48d0-11e9-95d5-98039b58ae12

3. To designate some users as administrators, run the following command.

CAUTION: This command assigns the cluster-admin role to all the accounts included in

the command. Use it with care.

kubectl create clusterrolebinding permissive-binding --

clusterrole=cluster-admin --user=alice --user=kubeflow --

user=dellemc1 --user=mike --group=system:serviceaccounts

In this example, John remains a regular user while the other users are assigned a

cluster-admin role.


clusterrolebinding.rbac.authorization.k8s.io/permissive-

binding created

4. Set a password for the new users by running htpasswd on the three masters:

cd /etc/origin/master/

htpasswd -b htpasswd dellemc1 Password1

htpasswd -b htpasswd alice Password2

htpasswd -b htpasswd kubeflow Password3

htpasswd -b htpasswd mike Password4

htpasswd -b htpasswd john Password5

The following output appears from master1:

Adding password for user dellemc1

Adding password for user alice

Adding password for user kubeflow

Adding password for user mike



Adding password for user john

5. From the bastion node, log in as a new user with the cluster-admin role:

oc login https://ocp.example.com:8443 -u dellemc1 -p

Password1 --insecure-skip-tls-verify=true

6. To view the status of all OpenShift nodes, run the following command:

oc get nodes


NAME STATUS ROLES AGE VERSION

app1.r5a.local Ready compute 33m v1.11.0+d4cacc0




infra1.r5a.local Ready infra 33m v1.11.0+d4cacc0



master1.r5a.local Ready master 41m v1.11.0+d4cacc0



stor1.r5a.local Ready compute 33m v1.11.0+d4cacc0




Note: To prevent containers from running on the storage nodes, storage nodes 1–4 can be

configured for applications as do not schedule.

7. To view all pods in all namespaces, run the following command:

oc get pods --all-namespaces

8. To view all routes, run the following command:

oc get routes --all-namespaces

Chapter 8: Creating the Default Storage Class


Chapter 8 Creating the Default Storage Class


Introduction ....................................................................................................... 38

Creating the default storage class ................................................................... 38

Chapter 8: Creating the Default Storage Class


Introduction

To create a default storage class, run this procedure from the bastion node with a user

that has been assigned the “cluster-admin” role. Because the Dell EMC ready architecture

for Red Hat OpenShift Container Platform uses NVMe SSD drives, we named the storage

volumes glusterfs-nvme.

Creating the default storage class

To create the default storage class:

1. From the bastion node, log in to OpenShift as a user that has been assigned the

cluster-admin role.





ansible-playbook src/ocs-pool/storage-class.yml


oc get sc

The output shows the new default storage class:

NAME PROVISIONER AGE

glusterfs-nvme (default) kubernetes.io/glusterfs 2m

glusterfs-registry-block gluster.org/glusterblock 34m

Chapter 9: Validating the OpenShift Container Platform Deployment


Chapter 9 Validating the OpenShift Container Platform Deployment


Introduction ....................................................................................................... 40

Validating the deployment ................................................................................ 40



Introduction

This chapter provides guidelines for validating that the OpenShift Container Platform

software is properly deployed by showing examples of the OpenShift Web Console user

interface and explaining how to verify that resources are being displayed as expected.

Note: ocp.example.com is not publicly available on the Internet.

Validating the deployment

The OpenShift Web Console is the web interface of an OpenShift cluster. Use the Web

Console to validate that the OpenShift Container Platform software was properly

deployed:

1. Log in by entering the following URL into your browser, using the hostname you

generated in your DNS:

https://ocp.example.com:8443/

The Service Catalog appears, as shown in the following figure.

Figure 6. Service Catalog

You can use the Service Catalog to easily download and deploy applications, create a

new project, and view the infrastructure.



2. In the upper right, click Create Project and enter the following information:

Create the project: dellemc1

The following page appears.

Figure 7. Creating a project

3. In the upper left of the Web Console, next to OPENSHIFT CONTAINER

PLATFORM, select Cluster Console from the drop-down menu.

The Cluster Console displays the cluster information that is available to a user

with the cluster-admin role.

4. In the left navigation bar, under Home, click Status.


Figure 8. Cluster Status page

5. Return to the OpenShift Cluster Console and, from the left navigation bar, select

Storage > Storage Classes.




Figure 9. OpenShift Storage Classes

6. In the left navigation bar of the Cluster Console, select Networking > Routes to

view the OpenShift service routes.

The following figure shows the service routes.

Figure 10. OpenShift service routes

7. Select Monitoring > Log in with OpenShift to log in to Prometheus.

Prometheus is the open source cluster monitoring stack that is included with

OpenShift Container Platform.




Figure 11. Authorize Access page

8. Click Allow Selected permissions.

Prometheus status information appears, as shown in the following figure.

Figure 12. Prometheus status view



Grafana is an open source package that works with Prometheus for the graphical

display of time series analytics. To configure a visualization of OpenShift

performance, follow the instructions in Grafana Support for Prometheus.

The following figure shows the Web Console view of the Grafana graph of the

OpenShift cluster’s CPU usage.

Figure 13. Grafana for the Web Console

9. Return to the Prometheus page and click Alerts Manager.


https://prometheus.io/docs/visualization/grafana/



Figure 14. Alerts Manager

10. Click the Alerts link beneath Altermanager.

The Alerts page appears, as shown in the following figure. The

DEADMansSwitch test is a default method to validate that the alerts are working.



Figure 15. Default alerts

Chapter 10: Validating OpenShift Operation


Chapter 10 Validating OpenShift Operation


Introduction ....................................................................................................... 48

Deploying a sample application: ruby-ex ........................................................ 48

Deploying a sample application: Lab-Guide-Workshop ................................. 51



Introduction

This chapter describes how to validate that the installed platform is operating correctly.

The validation process involves deploying one or more sample applications to the

OpenShift cluster and running the applications to confirm that key cluster functions are

working.

Deploying a sample application: ruby-ex

This procedure involves downloading a sample website application (ruby-ex), exposing

the service, and validating that you can reach the route (website address).

Follow these steps to deploy the ruby-ex sample application:

1. Log in to OpenShift and run the following command:



2. Create the ruby-ex demo project using the CLI, deploy ruby-ex, and then expose

the website by running the following command:

oc new-project demo


Now using project "demo" on server

"https://ocp.example.com:8443".

You can add applications to this project with the 'new-app'

command. For example, try:

oc new-app centos/ruby-25-

centos7~https://github.com/sclorg/ruby-ex.git

to build a new example application in Ruby.



centos7~https://github.com/sclorg/ruby-ex.git

--> Found Docker image c4310a5 (11 days old) from Docker Hub

for "centos/ruby-25-centos7"

Ruby 2.5

--------

Ruby 2.5 available as container is a base platform for

building and running various Ruby 2.5 applications and

frameworks. Ruby is the interpreted scripting language for

quick and easy object-oriented programming. It has many

features to process text files and to do system management

tasks (as in Perl). It is simple, straight-forward, and

extensible.

Tags: builder, ruby, ruby25, rh-ruby25



* An image stream tag will be created as "ruby-25-

centos7:latest" that will track the source image

* A source build using source code from

https://github.com/sclorg/ruby-ex.git will be created

* The resulting image will be pushed to image stream

tag "ruby-ex:latest"

* Every time "ruby-25-centos7:latest" changes a new

build will be triggered

* This image will be deployed in deployment config

"ruby-ex"

* Port 8080/tcp will be load balanced by service "ruby-

ex"

* Other containers can access this service through the

hostname "ruby-ex"

--> Creating resources ...

imagestream.image.openshift.io "ruby-25-centos7" created

imagestream.image.openshift.io "ruby-ex" created

buildconfig.build.openshift.io "ruby-ex" created

deploymentconfig.apps.openshift.io "ruby-ex" created

service "ruby-ex" created

--> Success

Build scheduled, use 'oc logs -f bc/ruby-ex' to track

its progress.

Application is not exposed. You can expose services to

the outside world by executing one or more of the commands

below:

'oc expose svc/ruby-ex'

Run 'oc status' to view your app.


oc expose svc/ruby-ex

route.route.openshift.io/ruby-ex exposed

The Web Console view of the ruby-ex demo appears, as shown in the following

figure.



Figure 16. Web console view of ruby-ex build

5. To run ruby-ex and the exposed service URL, click the URL.

The ruby-ex website appears, as shown in the following figure.

Figure 17. Ruby application welcome page



Deploying a sample application: Lab-Guide-Workshop

This section describes how to deploy a more complex application and use it to test access

to the resulting website. The application is a sample training workshop environment from

Red Hat called Lab-Guide-Workshop. Lab-Guide-Workshop provides sample applications

to enable users to learn more about OpenShift.

We used the OpenShift Container Platform 3.11 HA with 15 servers, including four

R740xd servers. Each of the servers had 10 NVMe disks that create the OpenShift

Storage Class “glusterfs-nvme,” which is set as the default.

To deploy Lab-Guide-Workshop:

1. Log in to OpenShift from the bastion node by running the following command:



2. To export the variables, run the following commands:

export OCP_ROUTING_SUFFIX=apps.ocp.example.com

export OCP_SUBDOMAIN=dellemc-2030.ocp.example.com

export OCP_SUBDOMAIN_INTERNAL=r5a.local

oc new-project workshop


Now using project "workshop" on server

"https://ocp.example.com:8443". You can add applications to

this project with the 'new-app' command. For example, try:


entos7~https://github.com/sclorg/ruby-ex.git to build a new

example application in Ruby.

3. Deploy the Lab-Guide-Workshop by running the following commands:

oc new-app docker.io/osevg/workshopper:0.2 --name=lab-guide

-e

CONTENT_URL_PREFIX="https://raw.githubusercontent.com/sa-

ne/na-se-openshift-workshop/dell_workshop/labguide" -e

WORKSHOPS_URLS="https://raw.githubusercontent.com/sa-ne/na-

se-openshift-

workshop/dell_workshop/labguide/_ocp_admin_testdrive.yaml"

-e COOLSTORE_PROJECT="user01" -e

OCP_ROUTING_SUFFIX="$OCP_ROUTING_SUFFIX" -e

MASTER_HOSTNAME="master1" -e

MASTER_EXTERNAL_FQDN="master.$OCP_SUBDOMAIN" -e

INFRA_INTERNAL_FQDN="infranode1.$OCP_SUBDOMAIN_INTERNAL" -e

NODE1_INTERNAL_FQDN="node1.$OCP_SUBDOMAIN_INTERNAL" -e

NODE2_INTERNAL_FQDN="node2.$OCP_SUBDOMAIN_INTERNAL"-e

WEB_CONSOLE_URL="https://master.$OCP_SUBDOMAIN_INTERNAL/con

sole" -e SSH_CONSOLE_URL="https://terminal-

workshop.$OCP_ROUTING_SUFFIX" -e

OPENSHIFT_DOCS_BASE="https://docs.openshift.com/container-

platform/3.11" -e KEYNAME="openshift" -e LOG_TO_STDOUT=true

Deploying Lab-

Guide-Workshop



--> Found Docker image d6af21e (17 months old) from

docker.io for "docker.io/osevg/workshopper:0.2" * An image

stream tag will be created as "lab-guide:0.2" that will

track this image * This image will be deployed in

deployment config "lab-guide" * Port 8080/tcp will be load

balanced by service "lab-guide" * Other containers can

access this service through the hostname "lab-guide"

--> Creating resources ...imagestream.image.openshift.io

"lab-guide" created deploymentconfig.apps.openshift.io

"lab-guide" created service "lab-guide" created-->

Success application is not exposed. You can expose services

to the outside world by executing one or more of the

commands below: 'oc expose svc/lab-guide' Run 'oc status' to

view your app.

oc expose svc/lab-guide

route.route.openshift.io/lab-guide exposed


Figure 18. Lab-Guide-Workshop with one lab-guide pod running

4. Click lab-guide, #1.

You can now scale to three pods, as shown in the following figure.



Figure 19. Scaling the lab-guide to run three pods

5. Select Applications > Routes to view the exposed route to the lab-guide service.




Figure 20. Web URL for the exposed lab-guide workshop service

The Lab-Guide-Workshop application is running on sub-domain

apps.ocp.example.com. OpenShift automatically generated lab-guide-

workshop when the route was created.

The wildcard DNS entry *.ocp.example.com permits resolution of all sub-

domains under ocp.example.com and resolves the OpenShift API. In the Ready

Architecture *.ocp.example.com, resolve to 100.82.42.99. This value matches the

values in /etc/ansible/hosts.



The following figure shows the exposed route. The route is ready for users to

access Lab-Guide-Workshop.

Figure 21. Exposed route

The following figure shows an OpenShift Web Console view of Lab-Guide-

Workshop with persistent volume claims on the glusterfs-nvme storage class.



Figure 22. OpenShift Web Console view of Lab-Guide-Workshop

Appendix A: Sample Switch Configuration Files


Appendix A Sample Switch Configuration Files

This appendix presents the following topics:

Introduction ....................................................................................................... 58

Downloading the sample files .......................................................................... 58

Modifying the deployment control file ............................................................. 61

Switch configuration file examples .................................................................. 65



Introduction

This chapter provides sample configuration files, including a kickstart file for the bastion

node, a sample Ansible hosts file, and sample switch configurations for all three of the

Dell Networking switches in the design.

Downloading the sample files

From the GitHub repository, download the following files:

• bastion.ks: https://github.com/dell-esg/openshift-bare-

metal/blob/master/examples/bastion.ks

• hosts.example.com: https://github.com/dell-esg/openshift-bare-

metal/blob/master/examples/hosts.example.com

• Switch configuration files:

▪ S3048: https://github.com/dell-esg/openshift-bare-

metal/blob/master/examples/S3048

▪ S5232F-1: https://github.com/dell-esg/openshift-bare-

metal/blob/master/examples/S5232F-1

▪ S5232F-2: https://github.com/dell-esg/openshift-bare-

metal/blob/master/examples/S5232F-1

bastion.ks is a sample kickstart file for Red Hat Enterprise Linux 7.6 installation:

text

eula --agreed

install

cdrom

lang en_US.UTF-8

keyboard --vckeymap=us --xlayouts='us'

timezone Etc/UTC --isUtc

rootpw OS_PASSWORD

auth --enableshadow --passalgo=sha512

services --enabled="chronyd"

firstboot --enable

%include /tmp/clearpart

zerombr

%include /tmp/bootdrive

volgroup vg00 pv.01

logvol swap --vgname=vg00 --name=swapvol --fstype=swap --size=4000

logvol / --vgname=vg00 --fstype=xfs --size=80000 --name=lv_root

reboot

%pre

bootdrive=`readlink -f /dev/disk/by-id/*BOSS* | head -n 1`

echo clearpart --all --initlabel --drives=$bootdrive >

/tmp/clearpart

Kickstart

installation file


https://github.com/dell-esg/openshift-bare-metal/blob/master/examples/bastion.ks

https://github.com/dell-esg/openshift-bare-metal/blob/master/examples/bastion.ks

https://github.com/dell-esg/openshift-bare-metal/blob/master/examples/hosts.example.com

https://github.com/dell-esg/openshift-bare-metal/blob/master/examples/hosts.example.com

https://github.com/dell-esg/openshift-bare-metal/blob/master/examples/S3048

https://github.com/dell-esg/openshift-bare-metal/blob/master/examples/S3048

https://github.com/dell-esg/openshift-bare-metal/blob/master/examples/S5232F-1






echo part /boot --fstype ext4 --size=500 --ondisk=$bootdrive >

/tmp/bootdrive

echo part /boot/efi --fstype vfat --size=200 --ondisk=$bootdrive

>> /tmp/bootdrive

echo part pv.01 --size=100000 --grow --ondisk=$bootdrive >>

/tmp/bootdrive

mdadm --create -R /dev/md0 --metadata=0.90 --level=1 --raid-

devices=2 /dev/nvme0n1 /dev/nvme1n1 >> /tmp/dockerstorage

sed -i s/"md_component_detection = 1"/"md_component_detection =

0"/ /etc/lvm/lvm.conf >> /tmp/dockerstorage

pvcreate /dev/md0 >> /tmp/dockerstorage

vgcreate docker-vg /dev/md0 >> /tmp/dockerstorage

%end

%packages

@^minimal

@core

chrony

kexec-tools

net-tools

wget

git

tcpdump

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%post --erroronfail

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0

ONBOOT=yes

BOOTPROTO=none

IPADDR=192.168.20.11

NETMASK=255.255.255.0

USERCTL=no

NM_CONTROLLED=yes

PEERDNS=yes

BONDING_OPTS="mode=802.3ad miimon=100 xmit_hash_policy=layer3+4

lacp_rate=1"

EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-bond0.10 DEVICE=bond0.10

IPADDR=192.168.10.10

NETMASK=255.255.255.0

GATEWAY=192.168.10.1

ONBOOT=yes

VLAN=yes

BOOTPROTO=none



USERCTL=no

EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-bond0.421

DEVICE=bond0.421

IPADDR=100.82.42.11

NETMASK=255.255.255.0

GATEWAY=100.82.42.1

DNS1=100.82.42.8

ONBOOT=yes

VLAN=yes

BOOTPROTO=none

USERCTL=no

EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-em1

DEVICE=em1

ONBOOT=yes

MASTER=bond0

BOOTPROTO=none

SLAVE=yes

EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-em2

DEVICE=em2

ONBOOT=yes

MASTER=bond0

BOOTPROTO=none

SLAVE=yes

EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-p2p1

DEVICE=p2p1

ONBOOT=yes

MASTER=bond0

BOOTPROTO=none

SLAVE=yes

EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-p2p2

DEVICE=p2p2

ONBOOT=yes

MASTER=bond0

BOOTPROTO=none

SLAVE=yes

EOF



cat << EOF > /etc/hostname

bastion.example.com

EOF

%end

Modifying the deployment control file

Modify the hosts.example.com file for Ansible and save it to /etc/ansible/hosts on

the bastion node:

[OSEv3:children]

masters

nodes

etcd

lb

local

glusterfs

glusterfs_registry

[OSEv3:vars]

openshift_release=v3.11

openshift_deployment_type=openshift-enterprise

ansible_ssh_user=root

root_password="OS_PASSWORD"

openshift_storage_glusterfs_image=registry.access.redhat.com/rhgs3

/rhgs-server-rhel7:v3.11

openshift_storage_glusterfs_block_image=registry.access.redhat.com

/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11

openshift_storage_glusterfs_heketi_image=registry.access.redhat.co

m/rhgs3/rhgs-volmanager-rhel7:v3.11

openshift_master_cluster_method=native

openshift_master_cluster_hostname=r5a.local

openshift_master_cluster_public_hostname=ocp.example.com

openshift_master_default_subdomain=apps.ocp.example.com

openshift_master_cluster_ip=192.168.20.99

openshift_master_cluster_public_ip=100.82.42.99

openshift_master_portal_net=10.0.0.0/16

openshift_master_identity_providers=[{'name': 'htpasswd_auth',

'login': 'true', 'challenge': 'true', 'kind':

'HTPasswdPasswordIdentityProvider'}]

openshift_master_htpasswd_users={'admin':

'$apr1$i4wJZDwp$dKhqHNvPCzTcXXXa9Gooz0'}

openshift_clock_enabled=true

os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'

# logging

openshift_logging_install_logging=true



openshift_logging_es_pvc_dynamic=true

openshift_logging_es_pvc_size=200Gi

openshift_logging_es_cluster_size=3

openshift_logging_es_pvc_storage_class_name='glusterfs-registry-

block'

openshift_logging_kibana_nodeselector={"node-

role.kubernetes.io/infra": "true"}

openshift_logging_curator_nodeselector={"node-


openshift_logging_es_nodeselector={"node-


# metrics

openshift_metrics_install_metrics=true

openshift_metrics_storage_kind=dynamic

openshift_metrics_storage_volume_size=200Gi

openshift_metrics_cassandra_pvc_storage_class_name='glusterfs-

registry-block'

openshift_metrics_hawkular_nodeselector={"node-


openshift_metrics_cassandra_nodeselector={"node-


openshift_metrics_heapster_nodeselector={"node-


# registry

openshift_master_dynamic_provisioning_enabled=True

openshift_hosted_registry_storage_kind=glusterfs

openshift_hosted_registry_storage_volume_size=1000Gi

openshift_hosted_registry_selector="node-

role.kubernetes.io/infra=true"

# OCS storage cluster for applications

openshift_storage_glusterfs_namespace=app-storage

openshift_storage_glusterfs_storageclass=true

openshift_storage_glusterfs_storageclass_default=false

openshift_storage_glusterfs_block_deploy=false

# OCS storage for OpenShift infrastructure

openshift_storage_glusterfs_registry_namespace=infra-storage

openshift_storage_glusterfs_registry_storageclass=false

openshift_storage_glusterfs_registry_block_deploy=true

openshift_storage_glusterfs_registry_block_host_vol_create=true

openshift_storage_glusterfs_registry_block_host_vol_size=500

openshift_storage_glusterfs_registry_block_storageclass=true

openshift_storage_glusterfs_registry_block_storageclass_default=fa

lse

# Red Hat Subscription information. Use user and password OR

activation key and organization id.



rhel_subscription_user=YourRHaccount

rhel_subscription_pass= YourRHpassword

activationkey=XXXXXXX

org_id=XXXXXXX

# Network settings. Required for OS provisioning.

dns_local=192.168.20.11

dns_upstream=100.82.42.8

external_interface=bond0

external_vlan=421

external_netmask=255.255.255.0

external_gateway=100.82.42.1

internal_interface=bond0

internal_netmask=255.255.255.0

internal_gateway={{ bastion_ip }}

bastion_ip=192.168.20.11

dhcp_first_ip=192.168.20.100

dhcp_last_ip=192.168.20.150

[local]

127.0.0.1

[masters]

master1.r5a.local

master2.r5a.local

master3.r5a.local

[etcd]

master1.r5a.local

master2.r5a.local

master3.r5a.local

[lb]

infra1.r5a.local

infra2.r5a.local

infra3.r5a.local

[glusterfs]

stor1.r5a.local glusterfs_cluster=1 glusterfs_devices='[

"/dev/nvme2n1", "/dev/nvme3n1", "/dev/nvme4n1", "/dev/nvme5n1",


"/dev/nvme10n1", "/dev/nvme11n1" ]'















[glusterfs_registry]

infra1.r5a.local glusterfs_devices='[ "/dev/nvme2n1",

"/dev/nvme3n1" ]'


"/dev/nvme3n1" ]'


"/dev/nvme3n1" ]'

[nodes]

master1.r5a.local openshift_ip=192.168.20.12

idrac_ip=192.168.10.12 serial=DELLSVCTAG

openshift_node_group_name='node-config-master'


idrac_ip=192.168.10.13 serial= DELLSVCTAG



idrac_ip=192.168.10.14 serial= DELLSVCTAG


infra1.r5a.local openshift_ip=192.168.20.15 idrac_ip=192.168.10.15

serial= DELLSVCTAG openshift_node_group_name='node-config-infra'

openshift_public_ip=100.82.42.15







app1.r5a.local openshift_ip=192.168.20.18 idrac_ip=192.168.10.18

serial= DELLSVCTAG openshift_node_group_name='node-config-compute'







stor1.r5a.local openshift_ip=192.168.20.23 idrac_ip=192.168.10.23


interface3='p7p1' interface4='p7p2'












[nodes:vars]

idrac_user=root

idrac_password=iDRAC_Password

oreg_auth_user=YourRHaccount

oreg_auth_password=YourRHpassword

oreg_url=registry.redhat.io/openshift3/ose-${component}:${version}

openshift_examples_modify_imagestreams=true

Switch configuration file examples

S5232F-1 example: running-configuration

This file is run on the first of two ToR switches, which is labeled S5232F-1. This file

contains the following code:

hostname S5232F-1

interface breakout 1/1/1 map 25g-4x

























Top-of-rack

switch

configuration










!

iscsi target port 860


!

spanning-tree rstp priority 24576

aaa authentication login default local

aaa authentication login console local

!

class-map type application class-iscsi

!

policy-map type application policy-iscsi

!

interface vlan1

no shutdown

!

interface vlan10

description iDRAC

no shutdown

ip address 192.168.10.2/24

!

vrrp-group 10

virtual-address 192.168.10.1

!

interface vlan20

description Internal

no shutdown

ip address 192.168.20.2/24

!

vrrp-group 20


!

interface vlan421

description Public

shutdown

!

interface port-channel1

description bastion-node

no shutdown

switchport mode trunk

switchport access vlan 20

switchport trunk allowed vlan 10,421

mtu 9216

vlt-port-channel 1



spanning-tree port type edge

!


description master-nodes

no shutdown


mtu 9216

lacp fallback enable

vlt-port-channel 2


!



no shutdown


mtu 9216


vlt-port-channel 3


!



no shutdown


mtu 9216


vlt-port-channel 4


!


description infra-nodes

no shutdown



switchport trunk allowed vlan 421

mtu 9216


vlt-port-channel 5


!



no shutdown




mtu 9216


vlt-port-channel 6




!



no shutdown




mtu 9216


vlt-port-channel 7


!


description app-nodes

no shutdown


mtu 9216


vlt-port-channel 8


!



no shutdown


mtu 9216


vlt-port-channel 9


!



no shutdown


mtu 9216


vlt-port-channel 10


!



no shutdown


mtu 9216


vlt-port-channel 11


!


description storage-nodes



no shutdown


mtu 9216


vlt-port-channel 12


!



no shutdown


mtu 9216


vlt-port-channel 13


!



no shutdown


mtu 9216


vlt-port-channel 14


!



no shutdown


mtu 9216


vlt-port-channel 15


!


description Uplink-Customer-Site

no shutdown



mtu 9216

vlt-port-channel 30

!


description Uplink-S3048

no shutdown



mtu 9216

vlt-port-channel 34

!



interface ethernet1/1/1:1

no shutdown

channel-group 1 mode active

no switchport

mtu 9216

flowcontrol receive off

!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!




no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown




no switchport

mtu 9216


!


shutdown



!

interface ethernet1/1/5

shutdown



!


shutdown



!


shutdown



!


shutdown



!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216




!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!




no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


shutdown



!


shutdown



!


shutdown



!




shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!




shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


no shutdown


no switchport

mtu 9216


!


no shutdown

no switchport


!


no shutdown

no switchport


!


shutdown



!


no shutdown




no switchport

mtu 9216


!

interface mgmt1/1/1

no shutdown

no ip address dhcp

ip address 100.82.33.81/24

ipv6 address autoconfig

!

management route 0.0.0.0/0 100.82.33.1

!

support-assist

!

vlt-domain 100

backup destination 100.82.33.82

discovery-interface ethernet1/1/31-1/1/32

peer-routing

vlt-mac 00:11:22:33:55:aa

snmp-server contact "Contact Support"

S5232F-2 example: running-configuration

This file is run on the second of two ToR switches, which is labeled S5232F-2. The file

contains the following code:

hostname S5232F-2



































!



spanning-tree rstp priority 24576

aaa authentication login default local

aaa authentication login console local

!

class-map type application class-iscsi

!

policy-map type application policy-iscsi

!

interface vlan1

no shutdown

!

interface vlan10

description iDRAC

no shutdown

ip address 192.168.10.3/24

!

vrrp-group 10

priority 200


!

interface vlan20

description Internal

no shutdown

ip address 192.168.20.3/24

!

vrrp-group 20

priority 200


!

interface vlan421

description Public

shutdown

!


description bastion-node

no shutdown




mtu 9216




vlt-port-channel 1


!



no shutdown


mtu 9216


vlt-port-channel 2


!



no shutdown


mtu 9216


vlt-port-channel 3


!



no shutdown


mtu 9216


vlt-port-channel 4


!



no shutdown




mtu 9216


vlt-port-channel 5


!



no shutdown




mtu 9216




vlt-port-channel 6


!



no shutdown




mtu 9216


vlt-port-channel 7


!



no shutdown


mtu 9216


vlt-port-channel 8


!



no shutdown


mtu 9216


vlt-port-channel 9


!



no shutdown


mtu 9216


vlt-port-channel 10


!



no shutdown


mtu 9216


vlt-port-channel 11


!




no shutdown


mtu 9216


vlt-port-channel 12


!


no shutdown


mtu 9216


vlt-port-channel 13


!


no shutdown


mtu 9216


vlt-port-channel 14


!


no shutdown


mtu 9216


vlt-port-channel 15


!


description Uplink-Customer-Site

no shutdown



mtu 9216

vlt-port-channel 30

!


description Uplink-S3048

no shutdown



mtu 9216

vlt-port-channel 34

!


no shutdown




no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown




no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport



mtu 9216


!


shutdown



!


no shutdown



!


no shutdown



!


no shutdown



!


no shutdown



!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!




no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown




no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


no shutdown


no switchport

mtu 9216


!


shutdown



!


shutdown



!


shutdown



!


shutdown





!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


shutdown





!


shutdown



!


shutdown



!


shutdown



!


shutdown



!


no shutdown


no switchport

mtu 9216


!


no shutdown

no switchport


!


no shutdown

no switchport


!


shutdown



!


no shutdown


no switchport

mtu 9216




!

interface mgmt1/1/1

no shutdown

no ip address dhcp

ip address 100.82.33.82/24

ipv6 address autoconfig

!


!

support-assist

!

vlt-domain 100

backup destination 100.82.33.81

discovery-interface ethernet1/1/31-1/1/32

peer-routing

primary-priority 65535

vlt-mac 00:11:22:33:55:aa

snmp-server contact "Contact Support"

S3048 example: running-configuration

This file is run on the OOB management switch, which is labeled S3048. The file contains

the following code:

!

boot system stack-unit 1 primary system://A

boot system stack-unit 1 secondary system://B

boot system stack-unit 1 default system://A

!

hostname S3048

!

protocol lldp

advertise dot1-tlv port-protocol-vlan-id

advertise dot3-tlv max-frame-size

advertise management-tlv system-description system-name

advertise med

!

redundancy auto-synchronize full

!

username dellemc1 password CHANGEME privilege 15

!

protocol spanning-tree rstp

no disable

!

stack-unit 1 provision S3048-ON

!

interface GigabitEthernet 1/1

no ip address

mtu 9216

switchport

OOB

management

switch

configuration



spanning-tree rstp edge-port

spanning-tree 0 portfast

no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport





no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!




no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address



mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport





no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport



no shutdown

!


no ip address

mtu 9216

switchport





no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216



portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown



!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


description Public-ClientAccess

no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!


description Public-ClientAccess

no ip address

mtu 9216

portmode hybrid

switchport



no shutdown

!

interface TenGigabitEthernet 1/49

no ip address



mtu 9216

portmode hybrid

switchport


no shutdown

!


no ip address

mtu 9216

portmode hybrid

switchport


no shutdown

!


description Up-S5232F

no ip address

mtu 9216

!

port-channel-protocol LACP

port-channel 51 mode active

no shutdown

!


description Up-S5232F

no ip address

mtu 9216

!

port-channel-protocol LACP

port-channel 51 mode active

no shutdown

!

interface ManagementEthernet 1/1

ip address 100.82.33.83/24

no shutdown

!


no shutdown

!


no shutdown

!


no shutdown

!


no shutdown

!


no shutdown



!

interface Port-channel 51

description Up-S5232F-te34

no ip address

mtu 9216

portmode hybrid

switchport

no shutdown

!

interface Vlan 1

!untagged GigabitEthernet 1/33-1/46

!untagged TenGigabitEthernet 1/50

!untagged Port-channel 51

!

interface Vlan 10

description iDRAC

no ip address

tagged Port-channel 51

untagged GigabitEthernet 1/1-1/32

shutdown

!

interface Vlan 421

no ip address

tagged Port-channel 51

untagged GigabitEthernet 1/47-1/48

untagged TenGigabitEthernet 1/49

shutdown

!


!

ip ssh server enable

!

line console 0

line vty 0

line vty 1

line vty 2

line vty 3

line vty 4

line vty 5

line vty 6

line vty 7

line vty 8

line vty 9

!

reload-type

boot-type normal-reload

config-scr-download enable

!

end

Appendix B: All-In-One Installation


Appendix B All-In-One Installation

This appendix presents the following topics:

Introduction ..................................................................................................... 101

Deploying OpenShift All-in-One using a single server ................................. 101

Installing Red Hat Enterprise Linux 7.6 ......................................................... 102

Creating Docker storage ................................................................................. 104

Installing OpenShift 3.11 AIO on a PowerEdge R640 server ........................ 105

Deploying a sample Kubernetes application ................................................ 110



Introduction

This section describes how to create a Red Hat OpenShift single-node developer

deployment. Red Hat refers to this deployment as the All-in-One (AIO). For more

information, see the following Red Hat blog post: OpenShift All-in-One (AIO) for Labs and

Fun.

Deploying OpenShift All-in-One using a single server

Deploying a 3.11 All-in-One bare-metal Kubernetes developer environment using one Dell

EMC PowerEdge R640 server requires:

• RHEL-7.6-20181010.0-Server-x86_64-dvd1.iso

• Red Hat Enterprise Linux subscriptions with OpenShift Enterprise 3.11

• One PowerEdge R640 server with iDRAC Enterprise

• From two to four 10 GB or 25 GB NICs using bond mode=4

• Proper switch configurations supporting LACP port-channel architecture with

tagged public network

• DNS wildcard entries for *.ocp.example.com resolving to your OpenShift bare-metal

server

To start installing Red Hat Enterprise Linux 7.6 on the AIO server, choose one of the

following methods. Both options have the same outcome.

Option 1: Prepare and use an image file with the iDRAC remote console

1. From an existing Red Hat Enterprise Linux 7.6 system, create a USB image by


mkfs.vfat -C ks_usb.img 1024

2. Mount the image by running the following command:

mount -o loop ks_usb.img /mnt

3. Place the customized example.ks file into the image by running the following

command:

cp example.ks /mnt


sync; umount /mnt

5. Copy the ks_usb.img file to the machine that will log in to the server’s iDRAC.

During the installation, you will attach this file to Removable Media under Virtual

Media in the iDRAC remote console.

Prerequisites

Preparing the

deployment

https://blog.openshift.com/openshift-all-in-one-aio-for-labs-and-fun/

https://blog.openshift.com/openshift-all-in-one-aio-for-labs-and-fun/



Option 2: Prepare a USB key for physical boot

1. From an existing Red Hat Enterprise Linux 7.6 system, format a USB key by


mkfs.ext3 /dev/sdb

2. Mount the USB key by running the following command:

mount /dev/sdb /mnt

3. Place the customized example.ks file into the image by running the following

command:

cp example.ks /mnt


sync; umount /mnt

Installing Red Hat Enterprise Linux 7.6

To install Red Hat Enterprise Linux 7.6, follow these steps:

1. Log in to the iDRAC Web interface of the R640 PowerEdge server and launch a

remote console.

2. On the Virtual Media tab, connect the virtual media, and then select Map

CD/DVD.

3. Browse to the location of the Red Hat Enterprise Linux 7.6 installation media and

select Map Device.

4. Use either ks_usb.img file (Option 1) or the USB key (Option 2).

▪ Option 1: In the iDRAC remote console under the Virtual Media tab, select

Map Removable Disk. Browse to the location of the ks_usb.img file and

select Map Device.

▪ Option 2: Insert the USB key into a USB port on the PowerEdge R640 server

before powering on the server.

Note: If only one physical RAID, SAS, or SATA disk is presented to the server, use the

device name sdb for the kickstart location in the Red Hat Enterprise Linux 7 installer.

sda = BOSS RAID1 enumerates as OS drive

sdb = ks_usb_img mounted to the iDRAC remote console’s removable media

Note: NVMe disk are enumerated as nvme0n1, nvme1n1, nvme2n1, and nvme3n1.

5. Select Next Boot and set it to Virtual CD/DVD/ISO.

6. Power on the system.

The server boots to the Red Hat Enterprise Linux 7.6 ISO.

Installing Red

Hat Enterprise

Linux



7. From the Red Hat Enterprise Linux 7.6 installation menu, select Install, and press

the [e] key.

Do not press the [Enter] key.

8. Move the cursor to the end of the line that starts with vmlinuz, and append the

following code:

ks=hd:sdb:/example.ks

The Red Hat Linux installation page appears, as shown in the following figure.

Figure 23. Red Hat Linux installation page

9. Press [CTRL+X] to start the installation.

10. Reboot the server and accept the license.

11. Log in and validate that the networks are functioning as expected.

Use SSH to log in to the PowerEdge R640 server. To register with Red Hat and enable

the proper repositories, run the following commands:

subscription-manager register –username YourRHaccount --password

YourRHpassword --force

subscription-manager attach --pool=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

subscription-manager repos --enable="rhel-7-server-rpms" --

enable="rhel-7-server-extras-rpms" --enable="rhel-7-server-ose-

3.11-rpms" --enable="rhel-7-server-ansible-2.6-rpms"

Registering with

Red Hat



yum -y install wget git net-tools bind-utils iptables-services

bridge-utils bash-completion kexec-tools sos psacct

yum -y install openshift-ansible

yum -y install docker-1.13.1

Creating Docker storage

Create Docker storage as follows:

1. If partitions with previous data exist on the disks, clean the disks by running the

following command, replacing X with the target drive number:

dd if=/dev/zero of=/dev/nvmeXn1 bs=1M count=1

For example: dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=1

2. Create Docker storage by running the following command:

cd /etc/sysconfig/

vi docker-storage-setup

3. Change to remove the overlay and add the following code to the file:

VG=docker-vg


4. Save the edited file.

5. Verify that the changes you made are in the file by running the following

command:

cat /etc/sysconfig/docker-storage-setup

VG=docker-vg


docker-storage-setup

Rounding up size to full physical extent 1.49 GiB

Thin pool volume with chunk size 512.00 KiB can address

at most 126.50 TiB of data. Logical volume "docker-pool"

created. Logical volume docker-vg/docker-pool changed.

6. Enable and start Docker by running the following command:

systemctl enable docker

systemctl start docker

systemctl is-active docker

7. Log in to the Docker registry and install Atomic by running the following command:

docker login -u YourRHaccount -p YourRHpassword

docker login https://registry.redhat.io

yum -y install atomic

yum -y install atomic-openshift-clients



8. Install Red Hat Enterprise Linux updates and reboot the server by running the

following command:

yum clean all

yum makecache

yum -y update

reboot

Installing OpenShift 3.11 AIO on a PowerEdge R640 server

To install OpenShift 3.11 AIO on a PowerEdge R640 server, follow these steps:

1. Copy the modified example.hosts file to /etc/ansible/hosts on the

PowerEdge R640 server by running the following commands:

#bare minimum hostfile ocp 3.11.88 ocp.example.com

[OSEv3:children]

masters

nodes

etcd

[OSEv3:vars]

openshift_release=3.11.88

openshift_deployment_type=openshift-enterprise

openshift_portal_net=172.30.0.0/16

# localhost likely doesn't meet the minimum requirements

openshift_disable_check=disk_availability,memory_availabilit

y

openshift_master_identity_providers=[{'name':

'htpasswd_auth', 'login': 'true', 'challenge': 'true',

'kind': 'HTPasswdPasswordIdentityProvider'}]

openshift_master_htpasswd_users={'admin':

'$apr1$8Ffff2Y3$DXZdrtdQFyPvsow0hc2.71'}

openshift_node_groups=[{'name': 'node-config-all-in-one',

'labels': ['node-role.kubernetes.io/master=true', 'node-

role.kubernetes.io/infra=true', 'node-

role.kubernetes.io/compute=true']}]

openshift_master_api_port=8443

openshift_master_console_port=8443

#Default: openshift_master_cluster_method=native

# Internal cluster name

openshift_master_cluster_hostname=ocp.example.com

# NOTE: Default wildcard domain for applications

openshift_master_default_subdomain=apps.ocp.example.com

[masters]

localhost ansible_connection=local



[etcd]


[nodes]

# openshift_node_group_name should refer to a dictionary

with matching key of name in list openshift_node_groups.


openshift_node_group_name="node-config-all-in-one"

[nodes:vars]

oreg_auth_user=YourRHaccount

oreg_auth_password=YourRHpassword

oreg_url=registry.redhat.io/openshift3/ose-

${component}:${version}

openshift_examples_modify_imagestreams=false

Note: To uninstall the deployment, run ansible-playbook

playbooks/adhoc/uninstall.yml.

2. Change to the openshift-ansible directory and run prerequisites.yml:

cd /usr/share/ansible/openshift-ansible

ansible-playbook playbooks/prerequisites.yml

This command usually finishes in less than 5 minutes with no failures.

3. Change to the openshift-ansible directory and run deploy_cluster.yml:

cd /usr/share/ansible/openshift-ansible

ansible-playbook playbooks/deploy_cluster.yml

This command might run for as long as 30 minutes, depending on your network

speed and hardware.

4. To view the status of the SSH session, run the following command:

tail -f /var/log/messages

watch docker ps

The following ouput appears. Zoom in to view the output.

Figure 24. Cluster deployment script output

New containers are created as the deploy_cluster.yml script runs to

completion. Zoom in to view the output from the script in the following figure.



Figure 25. Output when the deployment script is close to conclusion

When the cluster deployment is complete, the OpenShift Container Platform 3.11.88 is

working and ready for login, as shown in the following output example.

Figure 26. Cluster deployment at completion



To create OpenShift users, follow these steps:

1. Using SSH, log in to the PowerEdge R640 server as root:

cd /root/.kube

oc login -u system:admin

2. Create OpenShift users by running the following commands:



user.user.openshift.io/alice created

oc create user dellemc1


user.user.openshift.io/dellemc1 created

3. Add accounts to cluster-role cluster-admin by running the following commands:

kubectl create clusterrolebinding permissive-binding --

clusterrole=cluster-admin --user=alice --user=kubelet --

user=dellemc1 --group=system:serviceaccounts


clusterrolebinding.rbac.authorization.k8s.io/permissive-

binding created

4. Update passwords for the new accounts by using the htpasswd command:

cd /etc/origin/master

htpasswd htpasswd alice

5. When prompted, enter the password for user alice.

6. Confirm the password.

To validate OpenShift Container Platform deployment:

1. Log in as user alice:

oc login https://ocp.example.com:8443 -u alice -p Password1

Login successful

You have access to the following projects and can

switch between them with 'oc project <projectname>':

* default

kube-public

kube-service-catalog

kube-system

management-infra

openshift

Creating

OpenShift users

Validating

deployment



openshift-ansible-service-broker

openshift-console

openshift-infra

openshift-logging

openshift-monitoring

openshift-node

openshift-sdn

openshift-template-service-broker

openshift-web-console

Using project "default".

2. Create a new OpenShift project:

oc new-project alice

Now using project "alice" on server


3. To list the OpenShift projects, run the following command:

oc get projects

NAME DISPLAY NAME STATUS

default Active

kube-public Active

kube-service-catalog Active

kube-system Active

management-infra Active

openshift Active

openshift-ansible-service-broker Active

openshift-console Active

openshift-infra Active

openshift-logging Active

openshift-monitoring Active

openshift-node Active

openshift-sdn Active

openshift-template-service-broker Active

openshift-web-console Active

4. List all the pods on the OpenShift Container Platform 3.11 AIO by running the

following command:

oc get pods --all-namespaces

NAMESPACE NAME

READY STATUS RESTARTS AGE

default docker-registry-1-gb7kp

1/1 Running 0 19m

default registry-console-1-mqwg7

1/1 Running 0 19m

default router-1-lzrw8

1/1 Running 0 20m

kube-service-catalog apiserver-n89vr

1/1 Running 0 17m

kube-service-catalog controller-manager-t7gjb

1/1 Running 3 17m



kube-system master-api-

ocp.example.com 1/1 Running 0

24m

kube-system master-controllers-


25m

kube-system master-etcd-


24m

openshift-ansible-service-broker asb-1-xb657

1/1 Running 0 16m

openshift-console console-588448f455-ttg2p

1/1 Running 0 18m

openshift-monitoring alertmanager-main-0

3/3 Running 0 17m


3/3 Running 0 17m


3/3 Running 0 17m

openshift-monitoring cluster-monitoring-

operator-5fc9b4994d-8sq6b 1/1 Running 0

19m

openshift-monitoring grafana-75c67ffb7-slfxv

2/2 Running 0 18m

openshift-monitoring kube-state-metrics-

59d85c46fb-s52zd 3/3 Running 0

16m

openshift-monitoring node-exporter-d55m2

2/2 Running 0 16m

openshift-monitoring prometheus-k8s-0

4/4 Running 1 18m

openshift-monitoring prometheus-k8s-1

4/4 Running 1 17m

openshift-monitoring prometheus-operator-

7895dddcdd-9lsdr 1/1 Running 0

19m

openshift-node sync-nwflw

1/1 Running 0 24m

openshift-sdn ovs-t2t8h

1/1 Running 0 24m

openshift-sdn sdn-jzp5v

1/1 Running 0 24m

openshift-template-service-broker apiserver-2m6dr

1/1 Running 0 16m

openshift-web-console webconsole-5877dcf66d-

5ks4z 1/1 Running 0 18m

Deploying a sample Kubernetes application

Deploy a sample Kubernetes application as follows:

1. Run the oc project command.


Using project "alice" on server





Centos7~https://github.com/sclorg/ruby-ex.git

--> Found Docker image c4310a5 (8 days old) from Docker Hub

for "centos/ruby-25-centos7"

Ruby 2.5 -------- Ruby 2.5 available as container is

a base platform for building and running various Ruby 2.5

applications and frameworks. Ruby is the interpreted

scripting language for quick and easy object-oriented

programming. It has many features to process text files and

to do system management tasks (as in Perl). It is simple,

straight-forward, and extensible.

Tags: builder, ruby, ruby25, rh-ruby25

* An image stream tag will be created as "ruby-25-

centos7:latest" that will track the source image

* A source build using source code from

https://github.com/sclorg/ruby-ex.git will be created

* The resulting image will be pushed to image stream

tag "ruby-ex:latest"

* Every time "ruby-25-centos7:latest" changes a new

build will be triggered

* This image will be deployed in deployment config

"ruby-ex"

* Port 8080/tcp will be load balanced by service "ruby-

ex"

* Other containers can access this service through the

hostname "ruby-ex"

--> Creating resources ...

imagestream.image.openshift.io "ruby-25-centos7" created

imagestream.image.openshift.io "ruby-ex" created

buildconfig.build.openshift.io "ruby-ex" created

deploymentconfig.apps.openshift.io "ruby-ex" created

service "ruby-ex" created

--> Success

Build scheduled, use 'oc logs -f bc/ruby-ex' to track

its progress.

Application is not exposed. You can expose services to

the outside world by executing one or more of the commands

below:

'oc expose svc/ruby-ex'

Run 'oc status' to view your app.

2. To view logs of the creation of the ruby-ex web application, run the following

command:

oc logs -f bc/ruby-ex

Cloning "https://github.com/sclorg/ruby-ex.git" ...

Commit: c00ecd7c762590f1d52c316c7d00141a745ede18

(Merge pull request #25 from pvalena/master)

Author: Honza Horak <[email protected]>

Date: Thu Dec 13 15:35:54 2018 +0100

Using centos/ruby-25-

centos7@sha256:7a931fd172cb6a852ca2d7903dd6ac6b65d808e158223

b5276b51016d283fc1b as the s2i builder image

---> Installing application source ...

---> Building your Ruby application from source ...



---> Running 'bundle install --retry 2 --deployment --

without development:test' ...

Warning: the running version of Bundler (1.16.1) is older

than the version that created the lockfile (1.16.4). We

suggest you upgrade to the latest version of Bundler by

running `gem install bundler`

.

Fetching gem metadata from

https://rubygems.org/..............

Using bundler 1.16.1

Fetching puma 3.12.0

Installing puma 3.12.0 with native extensions

Fetching rack 2.0.6

Installing rack 2.0.6

Bundle complete! 2 Gemfile dependencies, 3 gems now

installed.

Gems in the groups development and test were not installed.

Bundled gems are installed into `./bundle`

---> Cleaning up unused ruby gems ...

Running `bundle clean --verbose` with bundler 1.16.1

Warning: the running version of Bundler (1.16.1) is older

than the version that created the lockfile (1.16.4). We

suggest you upgrade to the latest version of Bundler by

running `gem install bundler`.

Frozen, using resolution from the lockfile

Pushing image docker-registry.default.svc:5000/alice/ruby-

ex:latest ...

Pushed 0/10 layers, 1% complete

Pushed 10/10 layers, 100% complete

Push successful

3. Create the OpenShift Container Platform route and expose the website for the

sample application by running the following command:

oc expose svc/ruby-ex

route.route.openshift.io/ruby-ex exposed

4. Log in to the OpenShift Web Console and open the Cluster Console.

5. Select Applications > Routes to view the sample application, as shown in the

following figure.



Figure 27. OpenShift Console: Sample application view

6. Explore the diagnostic pages, carefully reviewing any warnings or alert messages.

If you have questions, contact Red Hat Support.

The following three figures show representative Cluster Console pages for:

▪ Health checks

▪ Nodes

▪ Projects



Figure 28. OpenShift Console health check



Figure 29. Checking nodes

Figure 30. OpenShift Console projects

Documents

Dell EMC Ready Architecture for Red Hat OpenShift ... · Dell EMC Ready Architecture for Red Hat OpenShift Container Platform v3.11 7 Deployment Guide For more information about OpenShift