Global TowerMigrating complex multi-cloud Infrastructure as Code to a Global Tower Platform Tony Kay tok@redhat.comGlobal Lead AutomationRed Hat GPTE DevOps and Automation

Share your automation story

1. How did you get started with Ansible?

2. How long have you been using it?

3. What's your favorite thing to do when you Ansible?

What we do…Build and Destroy > 1,000 Cloud Instances a day

● Deploy● Idle● Wake● Destroy

What we deploy - “Configs “

● Infrastructure Definitions e.g.○ OpenShift Cluster○ Three Tier App○ Ansible Tower Cluster

● Cloud Agnostic○ But mainly AWS

● Dynamic○ Vars determine everything

https://github.com/redhat-cop/agnosticd/tree/development/ansible/configs

APAC

EMEA

agnosticd

OpenTLC

Users

Users

Users RHPDS

Americas

admin-dev

admin

How we do it today

Direct Configuration

APAC

EMEA

agnosticd

OpenTLC

Users

Users

Users RHPDS

Americas

admin-dev

admin● Single Point(s) of Failure (No Backup)

● Hard to scale● Brittle workflow

○ Deployer scripts○ At jobs (suspend, destroy)

● Events require 1 off infrastructure● Code and Config stored together● Administrative bottlenecks

How we do it today

Direct Configuration

Conway’s Law?

● Team has unusual depth in OpenShift○ 7 Advanced OpenShift Instructors

● 4.5 Advanced Ansible Instructors● Full time Ansible Developer with Strong OpenShifts● Red Hat Certified Architects up to level 7● Experienced Ops Team (CloudForms, Satelite, OpenStack)● Jenkins and CI/CD skills (but….)● Virtually no infrastructure maintenance work

○ No patching○ No upgrading○ No rollback

SLIDE

Project Babylon

Users

Users

Users

Service CatalogManager

AgnosticV Operator

PoolBoy Operator

Anarchy Operator

AgnosticV

AgnosticD

Dark Tower

EMEA APAC

Americas

Isolated Nodes

AND A LOT OF YAML….

I SEE A SMALL ROOM

I SEE YOU... ...ALONE….

JEEZ THAT’S A LOT OF YAML….

AgnosticV

AgnosticV - The Configuration

<ORG>/<CONFIG>/<stage>.yml

<ORG>/<CONFIG>/common.yml

common.yml$ tree ansiblefest ├── common.yaml └── gpte ├── THREE_TIER_APP │ ├── common.yaml │ ├── dev.yaml │ └── prod.yml └── account.yaml

<ORG>/account.yml<ORG><CONFIG>

<STAGES>

http://bit.ly/babylon-dark-tower

----job_vars: guid: baby-three-01 __meta__: deployer: scm_type: git scm_url: https://github.com/redhat-cop/agnosticd.git scm_ref: three-tier-app-prod-1.14 type: agnosticd # default agnosticd entry_point: ansible/main.yml

callback: url: "" token: ""

tower: organization: gpte run_group: emea # e.g.region hint, stage hint (dev|prod) action: deploy

env_type: three-tier-app

...

StateOperators

AgnosticVOperator

AgnosticDRepo

Stateful Config to Deployment

AgnosticVRepo

The Operators

What is an Operator?An Operator is a method of packaging, deploying and managing a Kubernetes-native application. A Kubernetes-native application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling.

An Operator is essentially a custom controller.

AgnosticVCreates Catalog Entries and combines Configuration

Parameters into a deployable config

AnarchyRequests and manages the life cycle of a deployed config

PoolBoyManages, and match makes, pools of pre-deployed configs

AgnosticDThe Code

AgnosticDThe Code

● ~32 “configs”○ Stand alone instances○ OpenShift Clusters○ Mixed Linux and Windows○ Tower Cluster○ Multi-region infrastructure○ ...

● Extensively tagged● Can deploy bare infrastructure● Full end to end deploys● Pre-load workloads

○ E.G. Preconfigured Towers

AgnosticVGit Repo: Configuration

AgnosticDGit Repo: Code

Deployment

AgnosticV Operator

PoolBoy Operator

Anarchy Operator

Dark Tower

Dark Tower

Dark Tower

EMEA APAC

Americas

Isolated Nodes

● Tower Cluster○ 3.5○ Typically 3 nodes○ Currently they deploy to us-east-1

● Can “self replicate” - deploy itself● Supports Isolated Nodes● Supports multi-region peered VPCs

Dynamic Self Configuration

● Single job template job-runner● Single project● Empty per region inventories● Everything created dynamically

● Doesn’t require backup

http://bit.ly/babylon-dark-tower

----- name: Prepare any necessary Tower components then launch job hosts: localhost connection: local gather_facts: false become: false

roles:

- babylon-security-checks # IMPORTANT - ALWAYS first role - babylon-api-checks # Validation and additional logging - babylon-setup-for-new-job # Select execution instance groups etc - git-tag-lookup # Optionally select optimal stage git tag - create-job-manifest # Create a record of job prior to running - confirm-tower-organization # Ensure correct organization exists - confirm-tower-project # Confirm correct repo AND tag/reference - confirm-tower-job-template # Create 1 time job template - launch-tower-job # Launch job with callbacks - post-job-housekeeping # Cleanup 1 time objects plus misc cleanup...

Next Steps

● Global Production Deploy○ Post RHTE

● Per config virtualenvs● Light use of workflows● Automated Testing● CY 2020 - Tower on OpenShift