ANSIBLE MODULES: 3 4 MAINTAINING OVER 40 · FOREMAN + ANSIBLE = a Foreman has an API Everyone loves...

MAINTAINING OVER 40MAINTAINING OVER 40ANSIBLE MODULES: ANSIBLE MODULES: 33 4 4

YEARS LATERYEARS LATER

1 / 38

$ WHOAMI$ WHOAMI

Evgeni Golov

Senior Software Engineer at Red Hat

ex-Consultant at Red Hat

Debian Developer

♥ FOSS ♥

♥ automation ♥

2 / 38

FOREMAN + ANSIBLE = FOREMAN + ANSIBLE = ♥♥Foreman has an APIEveryone loves writing YAML instead ofclicking in a GUISo we wrote modules, rewrote them again,refactored them and stu�ed them into acollectionThis is the story of our journey

3 / 38

PRELUDE:PRELUDE:MOTIVATION / WTFMOTIVATION / WTF

4 / 38

WHAT'S FOREMAN?WHAT'S FOREMAN?lifecycle management tool for physical andvirtual serverspower management, provisioning,con�gurationBare-Metal, VMware, RHV, OpenStack, GCE,Azure, etchuge plugin ecosystem (Katello, Monitoring,Ansible, …)

5 / 38

WHAT'S ANSIBLE?WHAT'S ANSIBLE?"radically simple IT automation engine"huge number of modules for varioususecaseswriting own modules is very easyintegrates well with REST APIs

6 / 38

WHY AUTOMATINGWHY AUTOMATINGFOREMAN WITH ANSIBLE?FOREMAN WITH ANSIBLE?

We have daily tasks in our environmentWebUI and hammer don't scale wellUsing Ansible as a declarative API client

7 / 38

CHAPTER 1:CHAPTER 1:ANSIBLE/ANSIBLEANSIBLE/ANSIBLE

8 / 38

FOREMANFOREMAN AND AND KATELLOKATELLO

MODULES INMODULES INANSIBLE/ANSIBLEANSIBLE/ANSIBLE

Ansible upstream since 2.3 (2016)one module to rule them all, thuscumbersome to useuses the (Satellite speci�c) nailgun librarymostly Katello oriented

9 / 38

FOREMANFOREMAN AND AND KATELLOKATELLO

MODULES INMODULES INANSIBLE/ANSIBLEANSIBLE/ANSIBLE

Turns out one maintainer for code inansible/ansible is not enoughDidn't have tests until 2018Deprecated since 2.8To be removed in 2.12

10 / 38

CHAPTER 2: A NEWCHAPTER 2: A NEWREPOSITORYREPOSITORY

11 / 38

FOREMAN-ANSIBLE-FOREMAN-ANSIBLE-

MODULES.GITMODULES.GIT

Started in June 2017A new repository under @theforemanorganizationGoal: central place for collaboration aroundAnsible modules for ForemanFirst step: split foreman and katello into"one module per entity"

started with 6 modulesCentralized module_utils: July 2017

12 / 38

CHAPTER 3: TESTSCHAPTER 3: TESTS

13 / 38

CHAPTER 3.1: TESTCHAPTER 3.1: TESTPLAYBOOKSPLAYBOOKS

First set of tests added in November 2017Playbooks that would use the modulesagainst a live serverGood start, but expensive test executionDoesn't play well with Travis CI and friends

14 / 38

CHAPTER 3.2: VCR BASEDCHAPTER 3.2: VCR BASEDTESTSTESTS

VCR (vcrpy) is a great way to record andreplay HTTP requests/responsesAllows recording "good" API interactions andreplay them on TravisAdded January 2018Ensured modules work on Python 2.7 + 3.5First PlaybookCLI, now ansible-runnerFull coverage: August 2019

15 / 38

CHAPTER 3.3: CHECK MODECHAPTER 3.3: CHECK MODETESTSTESTS

All our modules support check modeWe re-run the VCR based tests with --check

16 / 38

CHAPTER 3.4: SANITY TESTSCHAPTER 3.4: SANITY TESTSAnsible provides ansible-test for in-treemodulesSince Ansible 2.9 it can also handleCollectionsWe run ansible-test sanity --venvplugins/ across all supported Pythons

17 / 38

CHAPTER 3.6: EXPECTEDCHAPTER 3.6: EXPECTEDCHANGE TESTSCHANGE TESTS

Our test playbooks execute every task twiceThe �rst execution is expected to havechanged=True

The second changed=FalseThis ensures the modules are idempotent

18 / 38

CHAPTER 3.7: DIFF MODECHAPTER 3.7: DIFF MODETESTSTESTS

Our modules return before/after diff datato AnsibleWe access that data in our test playbooksand analyze the content

19 / 38

CHAPTER 4:CHAPTER 4:DOCUMENTATIONDOCUMENTATION

20 / 38

CHAPTER 4.1: BUILDINGCHAPTER 4.1: BUILDINGDOCUMENTATIONDOCUMENTATION

All modules have DOCUMENTATIONpopulatedWe use build-ansible.py document-plugins with a customized templateAnsible internal, our use of it breakssometimes

Would be great to have o�cial toolingAutomatic builds on Galaxy?

Need to �gure out how to autopublish docs

21 / 38

CHAPTER 4.2:CHAPTER 4.2:DOCUMENTATIONDOCUMENTATION

FRAGMENTSFRAGMENTSAnsible 2.8 introduced documentationfragmentsWe use them heavily to document commonparameters (credentials etc)Fragments for return values would be cool

22 / 38

CHAPTER 5:CHAPTER 5:FOREMANANSIBLEMOFOREMANANSIBLEMO

DULEDULE

23 / 38

CHAPTER 5.1:CHAPTER 5.1:FOREMANANSIBLEMODULEFOREMANANSIBLEMODULE

ForemanAnsibleModule is a sub-class ofAnsibleModule

Simpli�ed de�nition of common parametersin argument_specImport error handlingEntity create/update/delete/comparehelpersbefore/after diff handling

24 / 38

CHAPTER 5.2:CHAPTER 5.2:FOREMANENTITY…,FOREMANENTITY…,

KATELLOANSIBLEMODULE,KATELLOANSIBLEMODULE,……

Further sub-classing usefulForemanEntityAnsibleModule adds a stateparameterKatelloAnsibleModule makes makesorganization required

25 / 38

CHAPTER 6: LIBRARIESCHAPTER 6: LIBRARIES

26 / 38

CHAPTER 6.1: NAILGUNCHAPTER 6.1: NAILGUNWe started with the nailgun libraryOriginally developed by Satellite QETargeted at Satellite environments

no support for non-Satellite pluginsreleased at the same cadence asSatellite

Designed to test the Satellite API

27 / 38

CHAPTER 6.2: APYPIECHAPTER 6.2: APYPIEnailgun was �ne when we targeted SatelliteenvironmentsKatello (and Foreman) were moving quickerDecided to write an own API library

using the published apidoc.json, thusmostly version agnostic

Switching libraries was rather easy due tothe abstraction we've built

And tests, tests will save you!

28 / 38

CHAPTER 7: USE THECHAPTER 7: USE THEFORCEFORCE

29 / 38

CHAPTER 7.1: USE THE FORCECHAPTER 7.1: USE THE FORCEOF THE ARGUMENT_SPECOF THE ARGUMENT_SPECAnsible supports complex (nested)argument_specs

elements='dict',

options=dict(…)

Allows better checking of complexparameters

30 / 38

CHAPTER 7.2: USE THE FORCECHAPTER 7.2: USE THE FORCEOF THE ENTITY_SPECOF THE ENTITY_SPEC

We always had a need to map from Ansibleparam names to Foreman API parametersThis resulted in the introduction of theentity_spec

argument_spec extended withForeman speci�c dataThe plain argument_spec can begenerated from it

31 / 38

CHAPTER 7.3: USE THE FORCECHAPTER 7.3: USE THE FORCEOF THE ENTITY_SPEC PT. 2OF THE ENTITY_SPEC PT. 2

Many modules perform simple CRUDoperations:

take user input�nd matching entitycreate/update/delete based on inputreport

We used to have write code for that, now thisis generated from the entity_spec

32 / 38

CHAPTER 8:CHAPTER 8:COMMUNITYCOMMUNITY

33 / 38

CHAPTER 8: COMMUNITYCHAPTER 8: COMMUNITYOriginally started as "my team needs this"Quickly gained contributions from ATIXToday: 35 contributors, many from Red Hatand ATIXDevelopers, Consultants, Ops, CustomersAdding their usecases and features

34 / 38

CHAPTER 8: COMMUNITYCHAPTER 8: COMMUNITYInitial contribution was hard, duplicatedcode, hard to testIncreased contribution when we moved to amore centralized codebaseHaving a collection and RPMs madeconsumption easierRecording VCR test results is still the biggestblocker

35 / 38

CHAPTER 9: OUTLOOKCHAPTER 9: OUTLOOK

36 / 38

CHAPTER 9: OUTLOOKCHAPTER 9: OUTLOOKforeman_host supporting ALL theparameterso�cial roles to support central work�owsmore modulesdocumentation autopublishing andversioningeasier contributioncollection defaults like

?module defaults

groups

37 / 38

THANKS!THANKS!

evgeni@golov.de

die-welt.net

@zhenech

@zhenech@chaos.social

@evgeni

38 / 38