OpenShift Container Platform 3 - Red Hat Customer Portal · OpenShift Installation and Configuration topics cover the basics of installing and configuring ... Configuring Docker Storage

OpenShift Container Platform 3.4

Installation and Configuration

OpenShift Container Platform 3.4 Installation and Configuration

Last Updated: 2019-04-12

Legal Notice

Copyright © 2019 Red Hat, Inc.

The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA isavailable athttp://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you mustprovide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinitylogo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and othercountries.

Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.

Java ® is a registered trademark of Oracle and/or its affiliates.

XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United Statesand/or other countries.

MySQL ® is a registered trademark of MySQL AB in the United States, the European Union andother countries.

Node.js ® is an official trademark of Joyent. Red Hat Software Collections is not formally related toor endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marksor trademarks/service marks of the OpenStack Foundation, in the United States and other countriesand are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed orsponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Abstract

OpenShift Installation and Configuration topics cover the basics of installing and configuringOpenShift in your environment. Use these topics for the one-time tasks required to get OpenShift upand running.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table of Contents

CHAPTER 1. OVERVIEW

CHAPTER 2. INSTALLING A CLUSTER2.1. PLANNING

2.1.1. Initial Planning2.1.2. Installation Methods2.1.3. Sizing Considerations2.1.4. Environment Scenarios

2.1.4.1. Single Master and Node on One System2.1.4.2. Single Master and Multiple Nodes2.1.4.3. Single Master, Multiple etcd, and Multiple Nodes2.1.4.4. Multiple Masters Using Native HA2.1.4.5. Stand-alone Registry

2.1.5. RPM vs Containerized2.2. PREREQUISITES

2.2.1. System Requirements2.2.1.1. Red Hat Subscriptions2.2.1.2. Minimum Hardware Requirements2.2.1.3. Production Level Hardware Requirements2.2.1.4. Configuring Core Usage2.2.1.5. SELinux2.2.1.6. NTP2.2.1.7. Security Warning

2.2.2. Environment Requirements2.2.2.1. DNS

2.2.2.1.1. Configuring Hosts to Use DNS2.2.2.1.2. Disabling dnsmasq2.2.2.1.3. Configuring a DNS Wildcard

2.2.2.2. Network Access2.2.2.2.1. NetworkManager2.2.2.2.2. Required Ports

2.2.2.3. Persistent Storage2.2.2.4. Cloud Provider Considerations

2.2.2.4.1. Configuring a Security Group2.2.2.4.2. Overriding Detected IP Addresses and Host Names2.2.2.4.3. Post-Installation Configuration for Cloud Providers

2.3. HOST PREPARATION2.3.1. Operating System Requirements2.3.2. Host Registration2.3.3. Installing Base Packages2.3.4. Installing Docker2.3.5. Configuring Docker Storage

2.3.5.1. Reconfiguring Docker Storage2.3.5.2. Managing Container Logs2.3.5.3. Viewing Available Container Logs

2.3.6. Ensuring Host Access2.3.7. Setting Global Proxy Values2.3.8. What’s Next?

2.4. INSTALLING ON CONTAINERIZED HOSTS2.4.1. Overview2.4.2. Install Methods for Containerized Hosts

18

19191919192020202121222222222222232425252525262627282828283131313234343434353637404141414242434343

Table of Contents

1

2.4.3. Required Images2.4.4. Starting and Stopping Containers2.4.5. File Paths2.4.6. Storage Requirements2.4.7. Open vSwitch SDN Initialization

2.5. QUICK INSTALLATION2.5.1. Overview2.5.2. Before You Begin2.5.3. Running an Interactive Installation2.5.4. Defining an Installation Configuration File2.5.5. Running an Unattended Installation2.5.6. Verifying the Installation2.5.7. Uninstalling OpenShift Container Platform2.5.8. What’s Next?

2.6. ADVANCED INSTALLATION2.6.1. Overview2.6.2. Before You Begin2.6.3. Configuring Ansible Inventory Files

Image Version Policy2.6.3.1. Configuring Cluster Variables2.6.3.2. Configuring Deployment Type2.6.3.3. Configuring Host Variables2.6.3.4. Configuring Master API and Console Ports2.6.3.5. Configuring Cluster Pre-install Checks2.6.3.6. Configuring System Containers

2.6.3.6.1. Running Docker as a System Container2.6.3.6.2. Running etcd as a System Container

2.6.3.7. Configuring a Registry Location2.6.3.7.1. Configuring Registry Storage

Option A: NFS Host GroupOption B: External NFS HostOption C: OpenStack PlatformOption D: AWS or Another S3 Storage Solution

2.6.3.8. Configuring Global Proxy Options2.6.3.9. Configuring Schedulability on Masters2.6.3.10. Configuring Node Host Labels

2.6.3.10.1. Configuring Dedicated Infrastructure Nodes2.6.3.11. Configuring Session Options2.6.3.12. Configuring Custom Certificates2.6.3.13. Configuring Cluster Metrics

2.6.3.13.1. Configuring Metrics StorageOption A: NFS Host GroupOption B: External NFS HostOption C: Dynamic

2.6.3.14. Configuring Cluster Logging2.6.3.14.1. Configuring Logging Storage

Option A: NFS Host GroupOption B: External NFS HostOption C: Dynamic

2.6.4. Example Inventory Files2.6.4.1. Single Master Examples

Single Master and Multiple NodesSingle Master, Multiple etcd, and Multiple Nodes

4444444545454546464748494950505051515151555557575960616262626262636365656566676868686969697070707070707172


2

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.6.4.2. Multiple Masters ExamplesMultiple Masters with Multiple etcdMultiple Masters with Master and etcd on the Same Host

2.6.5. Running the Advanced Installation2.6.6. Verifying the Installation

Verifying Multiple etcd HostsVerifying Multiple Masters Using HAProxy

2.6.7. Optionally Securing Builds2.6.8. Uninstalling OpenShift Container Platform

2.6.8.1. Uninstalling Nodes2.6.9. Known Issues2.6.10. What’s Next?

2.7. DISCONNECTED INSTALLATION2.7.1. Overview2.7.2. Prerequisites2.7.3. Required Software and Components

2.7.3.1. Syncing Repositories2.7.3.2. Syncing Images2.7.3.3. Preparing Images for Export

2.7.4. Repository Server2.7.4.1. Placing the Software

2.7.5. OpenShift Container Platform Systems2.7.5.1. Building Your Hosts2.7.5.2. Connecting the Repositories2.7.5.3. Host Preparation

2.7.6. Installing OpenShift Container Platform2.7.6.1. Importing OpenShift Container Platform Containerized Components2.7.6.2. Running the OpenShift Container Platform Installer2.7.6.3. Creating the Internal Docker Registry

2.7.7. Post-Installation Changes2.7.7.1. Re-tagging S2I Builder Images2.7.7.2. Configuring a Registry Location2.7.7.3. Creating an Administrative User2.7.7.4. Modifying the Security Policies2.7.7.5. Editing the Image Stream Definitions2.7.7.6. Loading the Container Images

2.7.8. Installing a Router2.8. INSTALLING A STAND-ALONE DEPLOYMENT OF OPENSHIFT CONTAINER REGISTRY

2.8.1. About OpenShift Container Registry2.8.2. Minimum Hardware Requirements2.8.3. Supported System Topologies2.8.4. Host Preparation2.8.5. Installation Methods

2.8.5.1. Quick Installation for Stand-alone OpenShift Container Registry2.8.5.2. Advanced Installation for Stand-alone OpenShift Container Registry

CHAPTER 3. SETTING UP THE REGISTRY3.1. REGISTRY OVERVIEW

3.1.1. About the Registry3.1.2. Integrated or Stand-alone Registries

3.2. DEPLOYING A REGISTRY ON EXISTING CLUSTERS3.2.1. Overview3.2.2. Deploying the Registry

737476787879808080808181828283838384868787878788888888888989898990919192939393939494949495

98989898989898

Table of Contents

3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.2.3. Deploying the Registry as a DaemonSet3.2.4. Registry Compute Resources3.2.5. Storage for the Registry

3.2.5.1. Production Use3.2.5.1.1. Use Amazon S3 as a Storage Back-end

3.2.5.2. Non-Production Use3.2.6. Enabling the Registry Console

3.2.6.1. Deploying the Registry Console3.2.6.2. Securing the Registry Console3.2.6.3. Troubleshooting the Registry Console

3.2.6.3.1. Debug Mode3.2.6.3.2. Display SSL Certificate Path

3.3. ACCESSING THE REGISTRY3.3.1. Viewing Logs3.3.2. File Storage3.3.3. Accessing the Registry Directly

3.3.3.1. User Prerequisites3.3.3.2. Logging in to the Registry3.3.3.3. Pushing and Pulling Images

3.4. SECURING AND EXPOSING THE REGISTRY3.4.1. Securing the Registry3.4.2. Exposing the Registry

3.4.2.1. Exposing a Secure Registry3.4.2.2. Exposing a Non-Secure Registry

3.5. EXTENDED REGISTRY CONFIGURATION3.5.1. Maintaining the Registry IP Address3.5.2. Whitelisting Docker Registries3.5.3. Overriding the Registry Configuration3.5.4. Registry Configuration Reference

3.5.4.1. Log3.5.4.2. Hooks3.5.4.3. Storage3.5.4.4. Auth3.5.4.5. Middleware

3.5.4.5.1. CloudFront Middleware3.5.4.5.2. Overriding Middleware Configuration Options3.5.4.5.3. Image Pullthrough3.5.4.5.4. Manifest Schema v2 Support

3.5.4.6. Reporting3.5.4.7. HTTP3.5.4.8. Notifications3.5.4.9. Redis3.5.4.10. Health3.5.4.11. Proxy

3.6. KNOWN ISSUES3.6.1. Overview3.6.2. Image Push Errors with Scaled Registry Using Shared NFS Volume3.6.3. Pull of Internally Managed Image Fails with "not found" Error3.6.4. Image Push Fails with "500 Internal Server Error" on S3 Storage3.6.5. Image Pruning Fails

CHAPTER 4. SETTING UP A ROUTER4.1. ROUTER OVERVIEW

999999

100100101101102102104104104104105105107107108108109109112114114116116117117119119120120121121122123123124124124124125125125125125125126126127

128128


4

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1.1. About Routers4.1.2. Router Service Account

4.2. USING THE DEFAULT HAPROXY ROUTER4.2.1. Overview4.2.2. Creating a Router4.2.3. Other Basic Router Commands4.2.4. Filtering Routes to Specific Routers4.2.5. Highly-Available Routers4.2.6. Customizing the Router Service Ports4.2.7. Working With Multiple Routers4.2.8. Adding a Node Selector to a Deployment Configuration4.2.9. Using Router Shards

4.2.9.1. Creating Router Shards4.2.9.2. Modifying Router Shards4.2.9.3. Using Namespace Router Shards

4.2.10. Customizing the Default Routing Subdomain4.2.11. Forcing Route Host Names to a Custom Routing Subdomain4.2.12. Using Wildcard Certificates4.2.13. Manually Redeploy Certificates4.2.14. Using Secured Routes4.2.15. Using Wildcard Routes (for a Subdomain)4.2.16. Using the Container Network Stack4.2.17. Exposing Router Metrics4.2.18. Preventing Connection Failures During Restarts4.2.19. ARP Cache Tuning for Large-scale Clusters4.2.20. Protecting Against DDoS Attacks

4.3. DEPLOYING A CUSTOMIZED HAPROXY ROUTER4.3.1. Overview4.3.2. Obtaining the Router Configuration Template4.3.3. Modifying the Router Configuration Template

4.3.3.1. Background4.3.3.2. Go Template Actions4.3.3.3. Router Provided Information4.3.3.4. Annotations4.3.3.5. Environment Variables4.3.3.6. Example Usage

4.3.4. Using a ConfigMap to Replace the Router Configuration Template4.3.5. Using Stick Tables4.3.6. Rebuilding Your Router

4.4. CONFIGURING THE HAPROXY ROUTER TO USE THE PROXY PROTOCOL4.4.1. Overview4.4.2. Why Use the PROXY Protocol?4.4.3. Using the PROXY Protocol

4.5. USING THE F5 ROUTER PLUG-IN4.5.1. Overview4.5.2. Prerequisites and Supportability

4.5.2.1. Configuring the Virtual Servers4.5.3. Deploying the F5 Router4.5.4. F5 Router Partition Paths4.5.5. Setting Up F5 Native Integration

CHAPTER 5. UPGRADING A CLUSTER5.1. OVERVIEW

128128128128129129130130131131131132134136137138138138139140141146147148149149151151151152152152153157158158160161162163163163164167168168169169171171

174174

Table of Contents

5

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.1.1. In-place or Blue-Green Upgrades5.2. PERFORMING AUTOMATED IN-PLACE CLUSTER UPGRADES

5.2.1. Overview5.2.2. Preparing for an Automated Upgrade5.2.3. Using the Installer to Upgrade5.2.4. Running Upgrade Playbooks Directly

5.2.4.1. Upgrading the Control Plane and Nodes in Separate Phases5.2.4.2. Customizing Node Upgrades5.2.4.3. Upgrading to the Latest OpenShift Container Platform 3.4 Release

5.2.5. Upgrading the EFK Logging Stack5.2.6. Upgrading Cluster Metrics5.2.7. Verifying the Upgrade

5.3. PERFORMING MANUAL IN-PLACE CLUSTER UPGRADES5.3.1. Overview5.3.2. Preparing for a Manual Upgrade5.3.3. Upgrading Master Components5.3.4. Updating Policy Definitions5.3.5. Upgrading Nodes5.3.6. Upgrading the Router5.3.7. Upgrading the Registry5.3.8. Updating the Default Image Streams and Templates5.3.9. Importing the Latest Images5.3.10. Upgrading the EFK Logging Stack5.3.11. Upgrading Cluster Metrics5.3.12. Additional Manual Steps Per Release

5.3.12.1. OpenShift Container Platform 3.4.0.405.3.12.2. OpenShift Container Platform 3.4.1.25.3.12.3. OpenShift Container Platform 3.4.1.55.3.12.4. OpenShift Container Platform 3.4.1.75.3.12.5. OpenShift Container Platform 3.4.1.105.3.12.6. OpenShift Container Platform 3.4.1.125.3.12.7. OpenShift Container Platform 3.4.1.165.3.12.8. OpenShift Container Platform 3.4.1.185.3.12.9. OpenShift Container Platform 3.4.1.245.3.12.10. OpenShift Container Platform 3.4.1.335.3.12.11. OpenShift Container Platform 3.4.1.375.3.12.12. OpenShift Container Platform 3.4.1.445.3.12.13. OpenShift Container Platform 3.4.1.44.26

5.3.13. Verifying the Upgrade5.4. BLUE-GREEN DEPLOYMENTS

5.4.1. Overview5.4.2. Preparing for a Blue-Green Upgrade

5.4.2.1. Sharing Software Entitlements5.4.2.2. Labeling Blue Nodes5.4.2.3. Creating and Labeling Green Nodes5.4.2.4. Verifying Green Nodes

5.4.3. Registry and Router Canary Deployments5.4.4. Warming the Green Nodes5.4.5. Evacuating and Decommissioning Blue Nodes

5.5. OPERATING SYSTEM UPDATES AND UPGRADES5.5.1. Updating and Upgrading the Operating System

CHAPTER 6. DOWNGRADING OPENSHIFT

174174174175177177177178179180180180181181181183185187189190190192193195196196196196196196196196197197197197197197197198198198198199200200201201202203203

205


6

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1. OVERVIEW6.2. VERIFYING BACKUPS6.3. SHUTTING DOWN THE CLUSTER6.4. REMOVING RPMS6.5. DOWNGRADING DOCKER6.6. REINSTALLING RPMS6.7. RESTORING ETCD6.8. BRINGING OPENSHIFT CONTAINER PLATFORM SERVICES BACK ONLINE6.9. VERIFYING THE DOWNGRADE

CHAPTER 7. MASTER AND NODE CONFIGURATION7.1. OVERVIEW7.2. MASTER CONFIGURATION FILES

7.2.1. Admission Control Configuration7.2.2. Asset Configuration7.2.3. Authentication and Authorization Configuration7.2.4. Controller Configuration7.2.5. etcd Configuration7.2.6. Grant Configuration7.2.7. Image Configuration7.2.8. Kubernetes Master Configuration7.2.9. Network Configuration7.2.10. OAuth Authentication Configuration7.2.11. Project Configuration7.2.12. Scheduler Configuration7.2.13. Security Allocator Configuration7.2.14. Service Account Configuration7.2.15. Serving Information Configuration7.2.16. Volume Configuration7.2.17. Audit Configuration

7.3. NODE CONFIGURATION FILES7.3.1. Pod and Node Configuration7.3.2. Docker Configuration7.3.3. Parallel Image Pulls with Docker 1.9+

7.4. PASSWORDS AND OTHER SENSITIVE DATA7.5. CREATING NEW CONFIGURATION FILES7.6. LAUNCHING SERVERS USING CONFIGURATION FILES7.7. CONFIGURING LOGGING LEVELS

CHAPTER 8. ADDING HOSTS TO AN EXISTING CLUSTER8.1. OVERVIEW8.2. ADDING HOSTS USING THE QUICK INSTALLER TOOL8.3. ADDING HOSTS USING THE ADVANCED INSTALL

CHAPTER 9. LOADING THE DEFAULT IMAGE STREAMS AND TEMPLATES9.1. OVERVIEW9.2. OFFERINGS BY SUBSCRIPTION TYPE

9.2.1. OpenShift Container Platform Subscription9.2.2. xPaaS Middleware Add-on Subscriptions

9.3. BEFORE YOU BEGIN9.4. PREREQUISITES9.5. CREATING IMAGE STREAMS FOR OPENSHIFT CONTAINER PLATFORM IMAGES9.6. CREATING IMAGE STREAMS FOR XPAAS MIDDLEWARE IMAGES9.7. CREATING DATABASE SERVICE TEMPLATES

205205205206206207208208208

210210210210211212212212213214214215216217217217218219220220222223223224224225226226

230230230231

234234234234235235235236236236

Table of Contents

7

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9.8. CREATING INSTANT APP AND QUICKSTART TEMPLATES9.9. WHAT’S NEXT?

CHAPTER 10. CONFIGURING CUSTOM CERTIFICATES10.1. OVERVIEW10.2. CONFIGURING CUSTOM CERTIFICATES WITH ANSIBLE10.3. CONFIGURING CUSTOM CERTIFICATES10.4. CONFIGURING A CUSTOM CERTIFICATE FOR A LOAD BALANCER

CHAPTER 11. REDEPLOYING CERTIFICATES11.1. OVERVIEW11.2. CHECKING CERTIFICATE EXPIRATIONS

11.2.1. Role Variables11.2.2. Running Certificate Expiration Playbooks

Other Example Playbooks11.2.3. Output Formats

HTML ReportJSON Report

11.3. REDEPLOYING CERTIFICATES11.3.1. Redeploying All Certificates Using the Current OpenShift Container Platform and etcd CA11.3.2. Redeploying a New or Custom OpenShift Container Platform CA11.3.3. Redeploying a New etcd CA11.3.4. Redeploying Master Certificates Only11.3.5. Redeploying etcd Certificates Only11.3.6. Redeploying Node Certificates Only11.3.7. Redeploying Registry or Router Certificates Only

11.3.7.1. Redeploying Registry Certificates Only11.3.7.2. Redeploying Router Certificates Only

11.3.8. Redeploying Custom Registry or Router Certificates11.3.8.1. Redeploying Registry Certificates Manually11.3.8.2. Redeploying Router Certificates Manually

CHAPTER 12. CONFIGURING AUTHENTICATION AND USER AGENT12.1. OVERVIEW12.2. IDENTITY PROVIDER PARAMETERS12.3. CONFIGURING IDENTITY PROVIDERS

12.3.1. Configuring identity providers with Ansible12.3.2. Configuring identity providers in the master configuration file

12.3.2.1. Manually provisioning a user when using the lookup mapping method12.3.3. Allow all12.3.4. Deny all12.3.5. HTPasswd12.3.6. Keystone12.3.7. LDAP authentication12.3.8. Basic authentication (remote)12.3.9. Request header12.3.10. GitHub12.3.11. GitLab12.3.12. Google12.3.13. OpenID connect

12.4. TOKEN OPTIONS12.5. GRANT OPTIONS12.6. SESSION OPTIONS12.7. PREVENTING CLI VERSION MISMATCH WITH USER AGENT

237238

239239239239240

242242242242243244244244245245246246247247248248248248249249249250

253253253254255255256257258258260261264265272273274275278279279280


8

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CHAPTER 13. SYNCING GROUPS WITH LDAP13.1. OVERVIEW13.2. CONFIGURING LDAP SYNC

13.2.1. LDAP Client Configuration13.2.2. LDAP Query Definition13.2.3. User-Defined Name Mapping

13.3. RUNNING LDAP SYNC13.4. RUNNING A GROUP PRUNING JOB13.5. SYNC EXAMPLES

13.5.1. RFC 230713.5.1.1. RFC2307 with User-Defined Name Mappings

13.5.2. RFC 2307 with User-Defined Error Tolerances13.5.3. Active Directory13.5.4. Augmented Active Directory

13.6. NESTED MEMBERSHIP SYNC EXAMPLE13.7. LDAP SYNC CONFIGURATION SPECIFICATION

13.7.1. v1.LDAPSyncConfig13.7.2. v1.StringSource13.7.3. v1.LDAPQuery13.7.4. v1.RFC2307Config13.7.5. v1.ActiveDirectoryConfig13.7.6. v1.AugmentedActiveDirectoryConfig

CHAPTER 14. CONFIGURING LDAP FAILOVER14.1. PREREQUISITES FOR CONFIGURING BASIC REMOTE AUTHENTICATION14.2. GENERATING AND SHARING CERTIFICATES WITH THE REMOTE BASIC AUTHENTICATION SERVER

14.3. CONFIGURING SSSD FOR LDAP FAILOVER14.4. CONFIGURING APACHE TO USE SSSD14.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM TO USE SSSD AS THE BASIC REMOTEAUTHENTICATION SERVER

CHAPTER 15. CONFIGURING THE SDN15.1. OVERVIEW15.2. AVAILABLE SDN PROVIDERS15.3. CONFIGURING THE POD NETWORK WITH ANSIBLE15.4. CONFIGURING THE POD NETWORK ON MASTERS15.5. CONFIGURING THE POD NETWORK ON NODES15.6. MIGRATING BETWEEN SDN PLUG-INS15.7. EXTERNAL ACCESS TO THE CLUSTER NETWORK15.8. USING FLANNEL

CHAPTER 16. CONFIGURING NUAGE SDN16.1. NUAGE SDN AND OPENSHIFT CONTAINER PLATFORM16.2. DEVELOPER WORKFLOW16.3. OPERATIONS WORKFLOW16.4. INSTALLATION

16.4.1. Installation for a Single Master16.4.2. Installation for Multiple Masters (HA)

CHAPTER 17. CONFIGURING FOR AWS17.1. OVERVIEW17.2. CONFIGURING AWS VARIABLES17.3. CONFIGURING OPENSHIFT CONTAINER PLATFORM MASTERS FOR AWS

282282282282283284284285285286288290292294297300300302303304305306

307307

307308310

313

315315315315316317317317318

321321321321321321322

324324324324

Table of Contents

9

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17.3.1. Configuring OpenShift Container Platform for AWS with Ansible17.3.2. Manually Configuring OpenShift Container Platform Masters for AWS17.3.3. Manually Configuring OpenShift Container Platform Nodes for AWS

17.4. SETTING KEY VALUE ACCESS PAIRS17.5. APPLYING CONFIGURATION CHANGES

CHAPTER 18. CONFIGURING FOR OPENSTACK18.1. OVERVIEW18.2. CONFIGURING OPENSTACK VARIABLES18.3. CONFIGURING OPENSHIFT CONTAINER PLATFORM MASTERS FOR OPENSTACK

18.3.1. Configuring OpenShift Container Platform for OpenStack with Ansible18.3.2. Manually Configuring OpenShift Container Platform Masters for OpenStack18.3.3. Manually Configuring OpenShift Container Platform Nodes for OpenStack

18.4. APPLYING CONFIGURATION CHANGES

CHAPTER 19. CONFIGURING FOR GCE19.1. OVERVIEW19.2. CONFIGURING MASTERS

19.2.1. Configuring OpenShift Container Platform Masters for GCE with Ansible19.2.2. Manually Configuring OpenShift Container Platform Masters for GCE

19.3. CONFIGURING NODES19.4. CONFIGURING MULTIZONE SUPPORT IN A GCE DEPLOYMENT19.5. APPLYING CONFIGURATION CHANGES

CHAPTER 20. CONFIGURING FOR AZURE20.1. OVERVIEW20.2. THE AZURE CONFIGURATION FILE20.3. CONFIGURING MASTERS20.4. CONFIGURING NODES20.5. APPLYING CONFIGURATION CHANGES

CHAPTER 21. CONFIGURING PERSISTENT STORAGE21.1. OVERVIEW21.2. PERSISTENT STORAGE USING NFS

21.2.1. Overview21.2.2. Provisioning21.2.3. Enforcing Disk Quotas21.2.4. NFS Volume Security

21.2.4.1. Group IDs21.2.4.2. User IDs21.2.4.3. SELinux21.2.4.4. Export Settings

21.2.5. Reclaiming Resources21.2.6. Automation21.2.7. Additional Configuration and Troubleshooting

21.3. PERSISTENT STORAGE USING GLUSTERFS21.3.1. Overview

21.3.1.1. Containerized Red Hat Gluster Storage21.3.1.2. Dedicated Storage Cluster

21.3.2. Support Requirements21.3.2.1. Supported Operating Systems21.3.2.2. Environment Requirements

21.3.3. Provisioning21.3.3.1. Creating Gluster Endpoints

324325325326326

327327327327327328329329

331331331331331332332333

334334334334335335

337337337337337339339340341342342343344344344344345345346346347347348


10

21.3.3.2. Creating the Persistent Volume21.3.3.3. Creating the Persistent Volume Claim

21.3.4. Gluster Volume Security21.3.4.1. Group IDs21.3.4.2. User IDs21.3.4.3. SELinux

21.4. PERSISTENT STORAGE USING OPENSTACK CINDER21.4.1. Overview21.4.2. Provisioning

21.4.2.1. Creating the Persistent Volume21.4.2.2. Volume Format

21.5. PERSISTENT STORAGE USING CEPH RADOS BLOCK DEVICE (RBD)21.5.1. Overview21.5.2. Provisioning

21.5.2.1. Creating the Ceph Secret21.5.2.2. Creating the Persistent Volume

21.5.3. Ceph Volume Security21.6. PERSISTENT STORAGE USING AWS ELASTIC BLOCK STORE

21.6.1. Overview21.6.2. Provisioning


21.7. PERSISTENT STORAGE USING GCE PERSISTENT DISK21.7.1. Overview21.7.2. Provisioning

21.7.2.1. Creating the Persistent Volume21.7.2.2. Volume Format21.7.2.3. Multi-zone Configuration

21.8. PERSISTENT STORAGE USING ISCSI21.8.1. Overview21.8.2. Provisioning

21.8.2.1. Enforcing Disk Quotas21.8.2.2. iSCSI Volume Security

21.9. PERSISTENT STORAGE USING FIBRE CHANNEL21.9.1. Overview21.9.2. Provisioning

21.9.2.1. Enforcing Disk Quotas21.9.2.2. Fibre Channel Volume Security

21.10. PERSISTENT STORAGE USING AZURE DISK21.10.1. Overview21.10.2. Prerequisites21.10.3. Provisioning


21.11. PERSISTENT STORAGE USING AZURE FILE21.11.1. Overview21.11.2. Before you begin21.11.3. Creating the Persistent Volume21.11.4. Creating the Azure Storage Account Secret

21.12. DYNAMIC PROVISIONING AND CREATING STORAGE CLASSES21.12.1. Overview21.12.2. Available Dynamically Provisioned Plug-ins21.12.3. Defining a StorageClass

349350351351352353353353354354355355355356356357358359359360360361361361361362363363364364364365365365365365366366366366366367367368368368369369369371371371372

Table of Contents

11

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21.12.3.1. Basic StorageClass Object Definition21.12.3.2. StorageClass Annotations21.12.3.3. OpenStack Cinder Object Definition21.12.3.4. AWS ElasticBlockStore (EBS) Object Definition21.12.3.5. GCE PersistentDisk (gcePD) Object Definition21.12.3.6. GlusterFS Object Definition21.12.3.7. Ceph RBD Object Definition21.12.3.8. Trident Object Definition21.12.3.9. VMware vSphere Object Definition

21.12.4. Changing the Default StorageClass21.12.5. Additional Information and Examples

21.13. VOLUME SECURITY21.13.1. Overview21.13.2. SCCs, Defaults, and Allowed Ranges21.13.3. Supplemental Groups21.13.4. fsGroup21.13.5. User IDs21.13.6. SELinux Options

21.14. SELECTOR-LABEL VOLUME BINDING21.14.1. Overview21.14.2. Motivation21.14.3. Deployment

21.14.3.1. Prerequisites21.14.3.2. Define the Persistent Volume and Claim21.14.3.3. Deploy the Persistent Volume and Claim

21.15. ENABLING CONTROLLER-MANAGED ATTACHMENT AND DETACHMENT21.15.1. Overview21.15.2. Determining What Is Managing Attachment and Detachment21.15.3. Configuring Nodes to Enable Controller-managed Attachment and Detachment

CHAPTER 22. PERSISTENT STORAGE EXAMPLES22.1. OVERVIEW22.2. SHARING AN NFS MOUNT ACROSS TWO PERSISTENT VOLUME CLAIMS

22.2.1. Overview22.2.2. Creating the Persistent Volume22.2.3. Creating the Persistent Volume Claim22.2.4. Ensuring NFS Volume Access22.2.5. Creating the Pod22.2.6. Creating an Additional Pod to Reference the Same PVC

22.3. COMPLETE EXAMPLE USING CEPH RBD22.3.1. Overview22.3.2. Installing the ceph-common Package22.3.3. Creating the Ceph Secret22.3.4. Creating the Persistent Volume22.3.5. Creating the Persistent Volume Claim22.3.6. Creating the Pod22.3.7. Defining Group and Owner IDs (Optional)

22.4. COMPLETE EXAMPLE USING GLUSTERFS22.4.1. Overview22.4.2. Installing the glusterfs-fuse Package22.4.3. Creating the Gluster Endpoints and Gluster Service for Persistence22.4.4. Creating the Persistent Volume22.4.5. Creating the Persistent Volume Claim

372373373373374374376376377377378378378379382385387389391391391391391391392393393393394

395395395395395396397398402404404404404405406407408408408409409410411


12

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22.4.6. Defining GlusterFS Volume Access22.4.7. Creating the Pod using NGINX Web Server image

22.5. COMPLETE EXAMPLE OF DYNAMIC PROVISIONING USING GLUSTERFS22.5.1. Overview22.5.2. Verify the Environment and Gather Needed Information22.5.3. Create a Storage Class for Your GlusterFS Dynamic Provisioner22.5.4. Create a PVC to Request Storage for Your Application22.5.5. Create a NGINX Pod That Uses the PVC

22.6. MOUNTING VOLUMES ON PRIVILEGED PODS22.6.1. Overview22.6.2. Prerequisites22.6.3. Creating the Persistent Volume22.6.4. Creating a Regular User22.6.5. Creating the Persistent Volume Claim22.6.6. Verifying the Setup

22.6.6.1. Checking the Pod SCC22.6.6.2. Verifying the Mount

22.7. BACKING DOCKER REGISTRY WITH GLUSTERFS STORAGE22.7.1. Overview22.7.2. Prerequisites22.7.3. Create the Gluster Persistent Volume22.7.4. Attach the PVC to the Docker Registry22.7.5. Known Issues

22.7.5.1. Pod Cannot Resolve the Volume Host22.8. BINDING PERSISTENT VOLUMES BY LABELS

22.8.1. Overview22.8.1.1. Assumptions

22.8.2. Defining Specifications22.8.2.1. Persistent Volume with Labels22.8.2.2. Persistent Volume Claim with Selectors22.8.2.3. Volume Endpoints22.8.2.4. Deploy the PV, PVC, and Endpoints

22.9. USING STORAGE CLASSES FOR DYNAMIC PROVISIONING22.9.1. Overview22.9.2. Scenario 1: Basic Dynamic Provisioning with Two Types of StorageClasses22.9.3. Scenario 2: How to enable Default StorageClass behavior for a Cluster

22.10. USING STORAGE CLASSES FOR EXISTING LEGACY STORAGE22.10.1. Overview

22.10.1.1. Scenario 1: Link StorageClass to existing Persistent Volume with Legacy Data

CHAPTER 23. WORKING WITH HTTP PROXIES23.1. OVERVIEW23.2. CONFIGURING NO_PROXY23.3. CONFIGURING HOSTS FOR PROXIES23.4. CONFIGURING HOSTS FOR PROXIES USING ANSIBLE23.5. PROXYING DOCKER PULL23.6. USING MAVEN BEHIND A PROXY23.7. CONFIGURING S2I BUILDS FOR PROXIES23.8. CONFIGURING DEFAULT TEMPLATES FOR PROXIES23.9. SETTING PROXY ENVIRONMENT VARIABLES IN PODS23.10. GIT REPOSITORY ACCESS

CHAPTER 24. CONFIGURING GLOBAL BUILD DEFAULTS AND OVERRIDES

412413417417418419419420422422422423423424425425425425425425426426427427428428428428428429430430431431431433438438438

441441441442442443444444444444445

446

Table of Contents

13

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24.1. OVERVIEW24.2. SETTING GLOBAL BUILD DEFAULTS

24.2.1. Configuring Global Build Defaults with Ansible24.2.2. Manually Setting Global Build Defaults

24.3. SETTING GLOBAL BUILD OVERRIDES

CHAPTER 25. CONFIGURING PIPELINE EXECUTION25.1. OVERVIEW

CHAPTER 26. CONFIGURING ROUTING26.1. OVERVIEW26.2. CONFIGURING ROUTE TIMEOUTS26.3. CONFIGURING NATIVE CONTAINER ROUTING

26.3.1. Network Overview

CHAPTER 27. ROUTING FROM EDGE LOAD BALANCERS27.1. OVERVIEW27.2. INCLUDING THE LOAD BALANCER IN THE SDN27.3. ESTABLISHING A TUNNEL USING A RAMP NODE

27.3.1. Configuring a Highly-Available Ramp Node

CHAPTER 28. AGGREGATING CONTAINER LOGS28.1. OVERVIEW28.2. PRE-DEPLOYMENT CONFIGURATION28.3. SPECIFYING LOGGING ANSIBLE VARIABLES28.4. SPECIFYING DEPLOYER PARAMETERS28.5. DEPLOYING THE EFK STACK28.6. UNDERSTANDING AND ADJUSTING THE DEPLOYMENT

28.6.1. Ops Cluster28.6.2. Elasticsearch28.6.3. Fluentd28.6.4. Kibana28.6.5. Curator

28.6.5.1. Creating the Curator Configuration28.7. CLEANUP28.8. UPGRADING28.9. TROUBLESHOOTING KIBANA28.10. SENDING LOGS TO AN EXTERNAL ELASTICSEARCH INSTANCE28.11. PERFORMING ADMINISTRATIVE ELASTICSEARCH OPERATIONS28.12. UPDATING FLUENTD’S LOG SOURCE AFTER A DOCKER LOG DRIVER UPDATE

CHAPTER 29. AGGREGATE LOGGING SIZING GUIDELINES29.1. OVERVIEW29.2. INSTALLATION

29.2.1. Large Clusters29.3. SYSTEMD-JOURNALD AND RSYSLOG29.4. SCALING UP EFK LOGGING29.5. STORAGE CONSIDERATIONS

CHAPTER 30. ENABLING CLUSTER METRICS30.1. OVERVIEW30.2. BEFORE YOU BEGIN30.3. SERVICE ACCOUNTS

30.3.1. Metrics Deployer Service Account30.3.2. Heapster Service Account

446446446447449

450450

452452452452452

455455455455458

459459459460461463464464465469474475476477477477479480481

482482482484485486486

488488488488489489


14

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30.4. METRICS DATA STORAGE30.4.1. Persistent Storage30.4.2. Capacity Planning for Cluster Metrics30.4.3. Non-Persistent Storage

30.5. METRICS DEPLOYER30.5.1. Using Secrets

30.5.1.1. Providing Your Own Certificates30.5.1.2. Using the Router’s Default Certificate30.5.1.3. Deployer Secret Options

30.5.2. Modifying the Deployer Template30.5.2.1. Deployer Template Parameters

30.6. DEPLOYING THE METRIC COMPONENTS30.6.1. Metrics Deployer Validations

30.7. SETTING THE METRICS PUBLIC URL30.8. ACCESSING HAWKULAR METRICS DIRECTLY

30.8.1. OpenShift Container Platform Projects and Hawkular Tenants30.8.2. Authorization

30.9. SCALING OPENSHIFT CONTAINER PLATFORM METRICS PODS30.9.1. Prerequisites30.9.2. Scaling the Cassandra Components

30.10. CLEANUP

CHAPTER 31. CUSTOMIZING THE WEB CONSOLE31.1. OVERVIEW31.2. LOADING EXTENSION SCRIPTS AND STYLESHEETS

31.2.1. Setting Extension Properties31.2.2. Customizing the Logo31.2.3. Changing Links to Documentation31.2.4. Adding or Changing Links to Download the CLI31.2.5. Customizing the About Page31.2.6. Configuring Navigation Menus

31.2.6.1. Top Navigation Dropdown Menus31.2.6.2. Project Left Navigation

31.2.7. Configuring Catalog Categories31.3. ENABLING WILDCARD ROUTES

31.3.1. Enabling Features in Technology Preview31.4. SERVING STATIC FILES

31.4.1. Enabling HTML5 Mode31.5. CUSTOMIZING THE LOGIN PAGE

31.5.1. Example Usage31.6. CUSTOMIZING THE OAUTH ERROR PAGE31.7. CHANGING THE LOGOUT URL31.8. CONFIGURING WEB CONSOLE CUSTOMIZATIONS WITH ANSIBLE

CHAPTER 32. REVISION HISTORY: INSTALLATION AND CONFIGURATION32.1. WED MAR 07 201832.2. TUE NOV 21 201732.3. FRI NOV 03 201732.4. MON OCT 16 201732.5. WED OCT 11 201732.6. FRI SEP 22 201732.7. MON SEP 18 201732.8. WED SEP 06 2017

489489490491492492492492493493494496497497498498498499499499500

501501501502502503503504504504505507509509510510511511511512512

514514514514514514515515516

Table of Contents

15

32.9. MON AUG 28 201732.10. FRI AUG 25 201732.11. TUE AUG 15 201732.12. TUE AUG 08 201732.13. FRI JUL 28 201732.14. THU JUL 27 201732.15. WED JUL 12 201732.16. THU JUL 06 201732.17. TUE JUN 13 201732.18. WED MAY 31 201732.19. THU MAY 25 201732.20. MON MAY 15 201732.21. MON MAY 08 201732.22. TUE APR 25 201732.23. WED APR 12 201732.24. FRI APR 07 201732.25. MON APR 03 201732.26. MON MAR 27 201732.27. WED MAR 22 201732.28. MON MAR 20 201732.29. TUE MAR 14 201732.30. MON MAR 06 201732.31. THU FEB 16 201732.32. THU FEB 09 201732.33. MON FEB 06 201732.34. MON JAN 30 201732.35. WED JAN 25 201732.36. WED JAN 18 2017

516516516516517517517517517518518518519519519520520521521522522523523524525526526527


16

Table of Contents

17

CHAPTER 1. OVERVIEWOpenShift Container Platform Installation and Configuration topics cover the basics of installing andconfiguring OpenShift Container Platform in your environment. Configuration, management, and loggingare also covered. Use these topics for the one-time tasks required to quickly set up your OpenShiftContainer Platform environment and configure it based on your organizational needs.

For day to day cluster administration tasks, see Cluster Administration.


18

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/cluster_administration/#admin-guide-index

CHAPTER 2. INSTALLING A CLUSTER

2.1. PLANNING

2.1.1. Initial Planning

For production environments, several factors influence installation. Consider the following questions asyou read through the documentation:

Which installation method do you want to use? The Installation Methods section provides someinformation about the quick and advanced installation methods.

How many hosts do you require in the cluster? The Environment Scenarios section providesmultiple examples of Single Master and Multiple Master configurations.

How many pods are required in your cluster? The Sizing Considerations section provides limitsfor nodes and pods so you can calculate how large your environment needs to be.

Is high availability required? High availability is recommended for fault tolerance. In thissituation, you might aim to use the Multiple Masters Using Native HA example as a basis foryour environment.

Which installation type do you want to use: RPM or containerized? Both installations provide aworking OpenShift Container Platform environment, but you might have a preference for aparticular method of installing, managing, and updating your services.

Is my installation supported if integrating with other technologies? See the OpenShift ContainerPlatform Tested Integrations for a list of tested integrations.

2.1.2. Installation Methods

Both the quick and advanced installation methods are supported for development and productionenvironments. If you want to quickly get OpenShift Container Platform up and running to try out for thefirst time, use the quick installer and let the interactive CLI guide you through the configuration optionsrelevant to your environment.

For the most control over your cluster’s configuration, you can use the advanced installation method.This method is particularly suited if you are already familiar with Ansible. However, following along withthe OpenShift Container Platform documentation should equip you with enough information to reliablydeploy your cluster and continue to manage its configuration post-deployment using the provided Ansibleplaybooks directly.

If you install initially using the quick installer, you can always further tweak your cluster’s configurationand adjust the number of hosts in the cluster using the same installer tool. If you wanted to later switch tousing the advanced method, you can create an inventory file for your configuration and carry on thatway.

2.1.3. Sizing Considerations

Determine how many nodes and pods you require for your OpenShift Container Platform cluster. Clusterscalability correlates to the number of pods in a cluster environment. That number influences the othernumbers in your setup.

The following table provides the maximum sizing limits for nodes and pods:


19

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/cluster_administration/#admin-guide-high-availabilityhttps://access.redhat.com/articles/2176281

Type Maximum

Maximum nodes per cluster 1000

Maximum pods per cluster 120,000

Maximum pods per node 250

Maximum pods per core 10

IMPORTANT

Oversubscribing the physical resources on a node affects resource guarantees theKubernetes scheduler makes during pod placement. Learn what measures you can taketo avoid memory swapping.

Determine how many pods are expected to fit per node:

Maximum Pods per Cluster / Expected Pods per Node = Total Number of Nodes

Example Scenario

If you want to scope your cluster for 2200 pods per cluster, you would need at least 9 nodes, assumingthat there are 250 maximum pods per node:

2200 / 250 = 8.8

If you increase the number of nodes to 20, then the pod distribution changes to 110 pods per node:

2200 / 20 = 110

2.1.4. Environment Scenarios

This section outlines different examples of scenarios for your OpenShift Container Platform environment.Use these scenarios as a basis for planning your own OpenShift Container Platform cluster, based onyour sizing needs.

NOTE

Moving from a single master cluster to multiple masters after installation is not supported.

2.1.4.1. Single Master and Node on One System

OpenShift Container Platform can be installed on a single system for a development environment only.An all-in-one environment is not considered a production environment.

2.1.4.2. Single Master and Multiple Nodes

The following table describes an example environment for a single master (with embedded etcd) andtwo nodes:


20

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/cluster_administration/#disabling-swap-memoryhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/developer_guide/#dev-guide-promoting-application-dehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#node

Host Name Infrastructure Component to Install

master.example.com Master and node

node1.example.com Node

node2.example.com

2.1.4.3. Single Master, Multiple etcd, and Multiple Nodes

The following table describes an example environment for a single master, three etcd hosts, and twonodes:


master.example.com Master and node

etcd1.example.com etcd

etcd2.example.com

etcd3.example.com


node2.example.com

NOTE

When specifying multiple etcd hosts, external etcd is installed and configured. Clusteringof OpenShift Container Platform’s embedded etcd is not supported.

2.1.4.4. Multiple Masters Using Native HA

The following describes an example environment for three masters, one HAProxy load balancer, threeetcd hosts, and two nodes using the native HA method:


master1.example.com Master (clustered using native HA) and node

master2.example.com

master3.example.com

lb.example.com HAProxy to load balance API master endpoints


21

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#node

etcd1.example.com etcd

etcd2.example.com

etcd3.example.com


node2.example.com


NOTE

When specifying multiple etcd hosts, external etcd is installed and configured. Clusteringof OpenShift Container Platform’s embedded etcd is not supported.

2.1.4.5. Stand-alone Registry

You can also install OpenShift Container Platform to act as a stand-alone registry using the OpenShiftContainer Platform’s integrated registry. See Installing a Stand-alone Registry for details on thisscenario.

2.1.5. RPM vs Containerized

An RPM installation installs all services through package management and configures services to runwithin the same user space, while a containerized installation installs services using container imagesand runs separate services in individual containers.

See the Installing on Containerized Hosts topic for more details on configuring your installation to usecontainerized services.

2.2. PREREQUISITES

2.2.1. System Requirements

The following sections identify the hardware specifications and system-level requirements of all hostswithin your OpenShift Container Platform environment.

2.2.1.1. Red Hat Subscriptions

You must have an active OpenShift Container Platform subscription on your Red Hat account toproceed. If you do not, contact your sales representative for more information.

IMPORTANT

OpenShift Container Platform 3.4 requires Docker 1.12.

2.2.1.2. Minimum Hardware Requirements


22

The system requirements vary per host type:

MastersPhysical or virtual system, or an instance running on a public or private IaaS.

Base OS: RHEL 7.3 or later with "Minimal" installation option, or RHEL Atomic Host7.3.2 or later. RHEL 7.2 is also supported using Docker 1.12 and its dependencies.

2 vCPU.

Minimum 16 GB RAM.

Minimum 40 GB hard disk space for the file system containing /var/.

NodesPhysical or virtual system, or an instance running on a public or private IaaS.

Base OS: RHEL 7.3 or later with "Minimal" installation option, or RHEL Atomic Host7.3.2 or later. RHEL 7.2 is also supported using Docker 1.12 and its dependencies.

NetworkManager 1.0 or later.

1 vCPU.

Minimum 8 GB RAM.

Minimum 15 GB hard disk space for the file system containing /var/.

An additional minimum 15 GB unallocated space to be used for Docker’s storage backend; see Configuring Docker Storage.

SeparateetcdNodes

Minimum 20 GB hard disk space for etcd data.

Consult Hardware Recommendations to properly size your etcd nodes.

Currently, OpenShift Container Platform stores image, build, and deployment metadatain etcd. You must periodically prune old resources. If you are planning to leverage alarge number of images/builds/deployments, place etcd on machines with largeamounts of memory and fast SSD drives.

IMPORTANT

OpenShift Container Platform only supports servers with the x86_64 architecture.

NOTE

Meeting the /var/ file system sizing requirements in RHEL Atomic Host requires makingchanges to the default configuration. See Managing Storage in Red Hat Enterprise LinuxAtomic Host for instructions on configuring this during or after installation.

2.2.1.3. Production Level Hardware Requirements

Test or sample environments function with the minimum requirements. For production environments, thefollowing recommendations apply:


23

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#nodehttps://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendationshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/cluster_administration/#admin-guide-pruning-resourceshttps://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#managing_storage_in_red_hat_enterprise_linux_atomic_host

Master Hosts

In a highly available OpenShift Container Platform cluster with a separate etcd cluster, a master hostshould have, in addition to the minimum requirements in the table above, 1 CPU core and 1.5 GB ofmemory for each 1000 pods. Therefore, the recommended size of a master host in an OpenShiftContainer Platform cluster of 2000 pods would be the minimum requirements of 2 CPU cores and 16GB of RAM, plus 2 CPU cores and 3 GB of RAM, totaling 4 CPU cores and 19 GB of RAM.

When planning an environment with multiple masters, a minimum of three etcd hosts and a load-balancer between the master hosts are required.

The OpenShift Container Platform master caches deserialized versions of resources aggressively toease CPU load. However, in smaller clusters of less than 1000 pods, this cache can waste a lot ofmemory for negligible CPU load reduction. The default cache size is 50,000 entries, which, depending onthe size of your resources, can grow to occupy 1 to 2 GB of memory. This cache size can be reducedusing the following setting the in /etc/origin/master/master-config.yaml:

kubernetesMasterConfig: apiServerArguments: deserialization-cache-size: - "1000"

Node Hosts

The size of a node host depends on the expected size of its workload. As an OpenShift ContainerPlatform cluster administrator, you will need to calculate the expected workload, then add about 10percent for overhead. For production environments, allocate enough resources so that a node hostfailure does not affect your maximum capacity.

Use the above with the following table to plan the maximum loads for nodes and pods:

Host Sizing Recommendation

Maximum nodes per cluster 1000

Maximum pods per cluster 120000

Maximum pods per nodes 250

Maximum pods per core 10

IMPORTANT

Oversubscribing the physical resources on a node affects resource guarantees theKubernetes scheduler makes during pod placement. Learn what measures you can taketo avoid memory swapping.

2.2.1.4. Configuring Core Usage

By default, OpenShift Container Platform masters and nodes use all available cores in the system theyrun on. You can choose the number of cores you want OpenShift Container Platform to use by settingthe GOMAXPROCS environment variable.


24

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/cluster_administration/#disabling-swap-memoryhttps://golang.org/pkg/runtime/

For example, run the following before starting the server to make OpenShift Container Platform only runon one core:

# export GOMAXPROCS=1

2.2.1.5. SELinux

Security-Enhanced Linux (SELinux) must be enabled on all of the servers before installing OpenShiftContainer Platform or the installer will fail. Also, configure SELINUXTYPE=targeted in the/etc/selinux/config file:

# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=enforcing# SELINUXTYPE= can take one of these three values:# targeted - Targeted processes are protected,# minimum - Modification of targeted policy. Only selected processes are protected.# mls - Multi Level Security protection.SELINUXTYPE=targeted

2.2.1.6. NTP

You must enable Network Time Protocol (NTP) to prevent masters and nodes in the cluster from goingout of sync. Set openshift_clock_enabled to true in the Ansible playbook to enable NTP onmasters and nodes in the cluster during Ansible installation.

# openshift_clock_enabled=true

2.2.1.7. Security Warning

OpenShift Container Platform runs containers on your hosts, and in some cases, such as buildoperations and the registry service, it does so using privileged containers. Furthermore, those containersaccess your host’s Docker daemon and perform docker build and docker push operations. Assuch, you should be aware of the inherent security risks associated with performing docker runoperations on arbitrary images as they effectively have root access.

For more information, see these articles:

http://opensource.com/business/14/7/docker-security-selinux

https://docs.docker.com/engine/security/security/

To address these risks, OpenShift Container Platform uses security context constraints that control theactions that pods can perform and what it has the ability to access.

2.2.2. Environment Requirements


25

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#containershttp://opensource.com/business/14/7/docker-security-selinuxhttps://docs.docker.com/engine/security/security/https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#security-context-constraints

The following section defines the requirements of the environment containing your OpenShift ContainerPlatform configuration. This includes networking considerations and access to external services, such asGit repository access, storage, and cloud infrastructure providers.

2.2.2.1. DNS

OpenShift Container Platform requires a fully functional DNS server in the environment. This is ideally aseparate host running DNS software and can provide name resolution to hosts and containers runningon the platform.

IMPORTANT

Adding entries into the /etc/hosts file on each host is not enough. This file is not copiedinto containers running on the platform.

Key components of OpenShift Container Platform run themselves inside of containers and use thefollowing process for name resolution:

1. By default, containers receive their DNS configuration file (/etc/resolv.conf) from their host.

2. OpenShift Container Platform then inserts one DNS value into the pods (above the node’snameserver values). That value is defined in the /etc/origin/node/node-config.yaml file by the dnsIP parameter, which by default is set to the address of the host node because the host isusing dnsmasq.

3. If the dnsIP parameter is omitted from the node-config.yaml file, then the value defaults to thekubernetes service IP, which is the first nameserver in the pod’s /etc/resolv.conf file.

As of OpenShift Container Platform 3.2, dnsmasq is automatically configured on all masters and nodes.The pods use the nodes as their DNS, and the nodes forward the requests. By default, dnsmasq isconfigured on the nodes to listen on port 53, therefore the nodes cannot run any other type of DNSapplication.

NOTE

NetworkManager is required on the nodes in order to populate dnsmasq with the DNSIP addresses.

The following is an example set of DNS records for the Single Master and Multiple Nodes scenario:

master A 10.64.33.100node1 A 10.64.33.101node2 A 10.64.33.102

If you do not have a properly functioning DNS environment, you could experience failure with:

Product installation via the reference Ansible-based scripts

Deployment of the infrastructure containers (registry, routers)

Access to the OpenShift Container Platform web console, because it is not accessible via IPaddress alone

2.2.2.1.1. Configuring Hosts to Use DNS


26

Make sure each host in your environment is configured to resolve hostnames from your DNS server. Theconfiguration for hosts' DNS resolution depend on whether DHCP is enabled. If DHCP is:

Disabled, then configure your network interface to be static, and add DNS nameservers toNetworkManager.

Enabled, then the NetworkManager dispatch script automatically configures DNS based on theDHCP configuration. Optionally, you can add a value to dnsIP in the node-config.yaml file toprepend the pod’s resolv.conf file. The second nameserver is then defined by the host’s firstnameserver. By default, this will be the IP address of the node host.

NOTE

For most configurations, do not set the openshift_dns_ip option during theadvanced installation of OpenShift Container Platform (using Ansible), becausethis option overrides the default IP address set by dnsIP.

Instead, allow the installer to configure each node to use dnsmasq and forwardrequests to SkyDNS or the external DNS provider. If you do set the openshift_dns_ip option, then it should be set either with a DNS IP thatqueries SkyDNS first, or to the SkyDNS service or endpoint IP (the Kubernetesservice IP).

To verify that hosts can be resolved by your DNS server:

1. Check the contents of /etc/resolv.conf:

$ cat /etc/resolv.conf# Generated by NetworkManagersearch example.comnameserver 10.64.33.1# nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh

In this example, 10.64.33.1 is the address of our DNS server.

2. Test that the DNS servers listed in /etc/resolv.conf are able to resolve host names to the IPaddresses of all masters and nodes in your OpenShift Container Platform environment:

$ dig @ +short

For example:

$ dig master.example.com @10.64.33.1 +short10.64.33.100$ dig node1.example.com @10.64.33.1 +short10.64.33.101

2.2.2.1.2. Disabling dnsmasq

If you want to disable dnsmasq (for example, if your /etc/resolv.conf is managed by a configurationtool other than NetworkManager), then set openshift_use_dnsmasq to false in the Ansible playbook.


27

However, certain containers do not properly move to the next nameserver when the first issuesSERVFAIL. Red Hat Enterprise Linux (RHEL)-based containers do not suffer from this, but certainversions of uclibc and musl do.

2.2.2.1.3. Configuring a DNS Wildcard

Optionally, configure a wildcard for the router to use, so that you do not need to update your DNSconfiguration when new routes are added.

A wildcard for a DNS zone must ultimately resolve to the IP address of the OpenShift Container Platformrouter.

For example, create a wildcard DNS entry for cloudapps that has a low time-to-live value (TTL) andpoints to the public IP address of the host where the router will be deployed:

*.cloudapps.example.com. 300 IN A 192.168.133.2

In almost all cases, when referencing VMs you must use host names, and the host names that you usemust match the output of the hostname -f command on each node.

WARNING

In your /etc/resolv.conf file on each node host, ensure that the DNS server that hasthe wildcard entry is not listed as a nameserver or that the wildcard domain is notlisted in the search list. Otherwise, containers managed by OpenShift ContainerPlatform may fail to resolve host names properly.

2.2.2.2. Network Access

A shared network must exist between the master and node hosts. If you plan to configure multiplemasters for high-availability using the advanced installation method, you must also select an IP to beconfigured as your virtual IP (VIP) during the installation process. The IP that you select must be routablebetween all of your nodes, and if you configure using a FQDN it should resolve on all nodes.

2.2.2.2.1. NetworkManager

NetworkManager, a program for providing detection and configuration for systems to automaticallyconnect to the network, is required.

2.2.2.2.2. Required Ports

The OpenShift Container Platform installation automatically creates a set of internal firewall rules oneach host using iptables. However, if your network configuration uses an external firewall, such as ahardware-based firewall, you must ensure infrastructure components can communicate with each otherthrough specific ports that act as communication endpoints for certain processes or services.

Ensure the following ports required by OpenShift Container Platform are open on your network andconfigured to allow access between hosts. Some ports are optional depending on your configuration andusage.


28

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#routershttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#high-availability-mastershttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#master-components

Table 2.1. Node to Node

4789 UDP Required for SDN communication between pods on separate hosts.

Table 2.2. Nodes to Master

53 or 8053 TCP/UDP

Required for DNS resolution of cluster services (SkyDNS). Installations prior to3.2 or environments upgraded to 3.2 use port 53. New installations will use8053 by default so that dnsmasq may be configured.


443 or 8443 TCP Required for node hosts to communicate to the master API, for the node hoststo post back status, to receive tasks, and so on.

Table 2.3. Master to Node


10250 TCP The master proxies to node hosts via the Kubelet for oc commands.

NOTE

In the following table, (L) indicates the marked port is also used in loopback mode,enabling the master to communicate with itself.

In a single-master cluster:

Ports marked with (L) must be open.

Ports not marked with (L) need not be open.

In a multiple-master cluster, all the listed ports must be open.

Table 2.4. Master to Master

53 (L) or 8053(L)

TCP/UDP

Required for DNS resolution of cluster services (SkyDNS). Installations prior to3.2 or environments upgraded to 3.2 use port 53. New installations will use8053 by default so that dnsmasq may be configured.

2049 (L) TCP/UDP

Required when provisioning an NFS host as part of the installer.

2379 TCP Used for standalone etcd (clustered) to accept changes in state.

2380 TCP etcd requires this port be open between masters for leader election and peeringconnections when using standalone etcd (clustered).

4001 (L) TCP Used for embedded etcd (non-clustered) to accept changes in state.


29

4789 (L) UDP Required for SDN communication between pods on separate hosts.

Table 2.5. External to Load Balancer

9000 TCP If you choose the native HA method, optional to allow access to theHAProxy statistics page.

Table 2.6. External to Master

443 or 8443 TCP Required for node hosts to communicate to the master API, for node hosts topost back status, to receive tasks, and so on.

Table 2.7. IaaS Deployments

22 TCP Required for SSH by the installer or system administrator.

53 or 8053 TCP/UDP

Required for DNS resolution of cluster services (SkyDNS). Installations prior to3.2 or environments upgraded to 3.2 use port 53. New installations will use8053 by default so that dnsmasq may be configured. Only required to beinternally open on master hosts.

80 or 443 TCP For HTTP/HTTPS use for the router. Required to be externally open on nodehosts, especially on nodes running the router.

1936 TCP For router statistics use. Required to be open when running the template routerto access statistics, and can be open externally or internally to connectionsdepending on if you want the statistics to be expressed publicly.

4001 TCP For embedded etcd (non-clustered) use. Only required to be internally open onthe master host. 4001 is for server-client connections.

2379 and 2380 TCP For standalone etcd use. Only required to be internally open on the masterhost. 2379 is for server-client connections. 2380 is for server-serverconnections, and is only required if you have clustered etcd.

4789 UDP For VxLAN use (OpenShift SDN). Required only internally on node hosts.

8443 TCP For use by the OpenShift Container Platform web console, shared with the APIserver.

10250 TCP For use by the Kubelet. Required to be externally open on nodes.

Notes

In the above examples, port 4789 is used for User Datagram Protocol (UDP).

When deployments are using the SDN, the pod network is accessed via a service proxy, unlessit is accessing the registry from the same node the registry is deployed on.


30

OpenShift Container Platform internal DNS cannot be received over SDN. Depending on thedetected values of openshift_facts, or if the openshift_ip and openshift_public_ipvalues are overridden, it will be the computed value of openshift_ip. For non-clouddeployments, this will default to the IP address associated with the default route on the masterhost. For cloud deployments, it will default to the IP address associated with the first internalinterface as defined by the cloud metadata.

The master host uses port 10250 to reach the nodes and does not go over SDN. It depends onthe target host of the deployment and uses the computed values of openshift_hostname and openshift_public_hostname.

Table 2.8. Aggregated Logging

9200 TCP For Elasticsearch API use. Required to be internally open on any infrastructurenodes so Kibana is able to retrieve logs for display. It can be externally openedfor direct access to Elasticsearch by means of a route. The route can becreated using oc expose.

9300 TCP For Elasticsearch inter-cluster use. Required to be internally open on anyinfrastructure node so the members of the Elasticsearch cluster maycommunicate with each other.

2.2.2.3. Persistent Storage

The Kubernetes persistent volume framework allows you to provision an OpenShift Container Platformcluster with persistent storage using networked storage available in your environment. This can be doneafter completing the initial OpenShift Container Platform installation depending on your applicationneeds, giving users a way to request those resources without having any knowledge of the underlyinginfrastructure.

The Installation and Configuration Guide provides instructions for cluster administrators on provisioningan OpenShift Container Platform cluster with persistent storage using NFS, GlusterFS, Ceph RBD,OpenStack Cinder, AWS Elastic Block Store (EBS), GCE Persistent Disks, and iSCSI.

2.2.2.4. Cloud Provider Considerations

There are certain aspects to take into consideration if installing OpenShift Container Platform on a cloudprovider.

2.2.2.4.1. Configuring a Security Group

When installing on AWS or OpenStack, ensure that you set up the appropriate security groups. Theseare some ports that you should have in your security groups, without which the installation will fail. Youmay need more depending on the cluster configuration you want to install. For more information and toadjust your security groups accordingly, see Required Ports for more information.

All OpenShift ContainerPlatform Hosts tcp/22 from host running the installer/Ansible


31

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html-single/architecture/#architecture-additional-concepts-storage

etcd Security Grouptcp/2379 from masters

tcp/2380 from etcd hosts

Master Security Grouptcp/8443 from 0.0.0.0/0

tcp/53 from all OpenShift Container Platform hosts forenvironments installed prior to or upgraded to 3.2

udp/53 from all OpenShift Container Platform hosts forenvironments installed prior to or upgraded to 3.2

tcp/8053 from all OpenShift Container Platform hosts for newenvironments installed with 3.2

udp/8053 from all OpenShift Container Platform hosts for newenvironments installed with 3.2

Node Security Grouptcp/10250 from masters

udp/4789 from nodes

Infrastructure Nodes (ones thatcan host the OpenShift ContainerPlatform router)

tcp/443 from 0.0.0.0/0

tcp/80 from 0.0.0.0/0

If configuring ELBs for load balancing the masters and/or routers, you also need to configure Ingress andEgress security groups for the ELBs appropriately.

2.2.2.4.2. Overriding Detected IP Addresses and Host Names

Some deployments require that the user override the detected host names and IP addresses for thehosts. To see the default values, run the openshift_facts playbook:

# ansible-playbook playbooks/byo/openshift_facts.yml

Now, verify the detected common settings. If they are not what you expect them to be, you can overridethem.

The Advanced Installation topic discusses the available Ansible variables in greater detail.

Variable Usage

hostnameShould resolve to the internal IP from the instances themselves.

openshift_hostname overrides.


32

ipShould be the internal IP of the instance.

openshift_ip will overrides.

public_hostnameShould resolve to the external IP from hosts outside of thecloud.

Provider openshift_public_hostname overrides.

public_ipShould be the externally accessible IP associated with theinstance.

openshift_public_ip overrides.

use_openshift_sdnShould be true unless the cloud is GCE.

openshift_use_openshift_sdn overrides.

Variable Usage

WARNING

If openshift_hostname is set to a value other than the metadata-provided private-dns-name value, the native cloud integration for those providers will nolonger work.

In AWS, situations that require overriding the variables include:

Variable Usage

hostname The user is installing in a VPC that is not configured for both DNS hostnames and DNS resolution.

ip Possibly if they have multiple network interfaces configured and theywant to use one other than the default. You must first set openshift_set_node_ip to True. Otherwise, the SDN wouldattempt to use the hostname setting or try to resolve the host name forthe IP.


33

public_hostnameA master instance where the VPC subnet is not configured for Auto-assign Public IP. For external access to thismaster, you need to have an ELB or other load balancerconfigured that would provide the external access needed, oryou need to connect over a VPN connection to the internalname of the host.

A master instance where metadata is disabled.

This value is not actually used by the nodes.

public_ipA master instance where the VPC subnet is not configured for Auto-assign Public IP.

A master instance where metadata is disabled.

This value is not actually used by the nodes.

Variable Usage

If setting openshift_hostname to something other than the metadata-provided private-dns-namevalue, the native cloud integration for those providers will no longer work.

For EC2 hosts in particular, they must be deployed in a VPC that has both DNS host names and DNS resolution enabled, and openshift_hostname should not be overridden.

2.2.2.4.3. Post-Installation Configuration for Cloud Providers

Following the installation process, you can configure OpenShift Container Platform for AWS, OpenStack,or GCE.

2.3. HOST PREPARATION

2.3.1. Operating System Requirements

A base installation of RHEL 7.3 or later or RHEL Atomic Host 7.3.2 or later is required for master andnode hosts. RHEL 7.2 is also supported using Docker 1.12 and its dependencies. See the followingdocumentation for the respective installation instructions, if required:

Red Hat Enterprise Linux 7 Installation Guide

Red Hat Enterprise Linux Atomic Host 7 Installation and Configuration Guide

2.3.2. Host Registration

Each host must be registered using Red Hat Subscription Manager (RHSM) and have an activeOpenShift Container Platform subscription attached to access the required packages.

1. On each host, register with

Documents

OpenShift Container Platform 3 - Red Hat Customer Portal · OpenShift Installation and Configuration topics cover the basics of installing and configuring ... Configuring Docker Storage