Embed Size (px)
Citation preview
MAINTAINING OVER 40MAINTAINING OVER 40ANSIBLE MODULES: ANSIBLE MODULES: 33 4 4
YEARS LATERYEARS LATER
1 / 38
$ WHOAMI$ WHOAMI
Evgeni Golov
Senior Software Engineer at Red Hat
ex-Consultant at Red Hat
Debian Developer
♥ FOSS ♥
♥ automation ♥
2 / 38
FOREMAN + ANSIBLE = FOREMAN + ANSIBLE = ♥♥Foreman has an APIEveryone loves writing YAML instead ofclicking in a GUISo we wrote modules, rewrote them again,refactored them and stu�ed them into acollectionThis is the story of our journey
3 / 38
PRELUDE:PRELUDE:MOTIVATION / WTFMOTIVATION / WTF
4 / 38
WHAT'S FOREMAN?WHAT'S FOREMAN?lifecycle management tool for physical andvirtual serverspower management, provisioning,con�gurationBare-Metal, VMware, RHV, OpenStack, GCE,Azure, etchuge plugin ecosystem (Katello, Monitoring,Ansible, …)
5 / 38
WHAT'S ANSIBLE?WHAT'S ANSIBLE?"radically simple IT automation engine"huge number of modules for varioususecaseswriting own modules is very easyintegrates well with REST APIs
6 / 38
WHY AUTOMATINGWHY AUTOMATINGFOREMAN WITH ANSIBLE?FOREMAN WITH ANSIBLE?
We have daily tasks in our environmentWebUI and hammer don't scale wellUsing Ansible as a declarative API client
7 / 38
CHAPTER 1:CHAPTER 1:ANSIBLE/ANSIBLEANSIBLE/ANSIBLE
8 / 38
FOREMANFOREMAN AND AND KATELLOKATELLO
MODULES INMODULES INANSIBLE/ANSIBLEANSIBLE/ANSIBLE
Ansible upstream since 2.3 (2016)one module to rule them all, thuscumbersome to useuses the (Satellite speci�c) nailgun librarymostly Katello oriented
9 / 38
FOREMANFOREMAN AND AND KATELLOKATELLO
MODULES INMODULES INANSIBLE/ANSIBLEANSIBLE/ANSIBLE
Turns out one maintainer for code inansible/ansible is not enoughDidn't have tests until 2018Deprecated since 2.8To be removed in 2.12
10 / 38
CHAPTER 2: A NEWCHAPTER 2: A NEWREPOSITORYREPOSITORY
11 / 38
FOREMAN-ANSIBLE-FOREMAN-ANSIBLE-
MODULES.GITMODULES.GIT
Started in June 2017A new repository under @theforemanorganizationGoal: central place for collaboration aroundAnsible modules for ForemanFirst step: split foreman and katello into"one module per entity"
started with 6 modulesCentralized module_utils: July 2017
12 / 38
CHAPTER 3: TESTSCHAPTER 3: TESTS
13 / 38
CHAPTER 3.1: TESTCHAPTER 3.1: TESTPLAYBOOKSPLAYBOOKS
First set of tests added in November 2017Playbooks that would use the modulesagainst a live serverGood start, but expensive test executionDoesn't play well with Travis CI and friends
14 / 38
CHAPTER 3.2: VCR BASEDCHAPTER 3.2: VCR BASEDTESTSTESTS
VCR (vcrpy) is a great way to record andreplay HTTP requests/responsesAllows recording "good" API interactions andreplay them on TravisAdded January 2018Ensured modules work on Python 2.7 + 3.5First PlaybookCLI, now ansible-runnerFull coverage: August 2019
15 / 38
CHAPTER 3.3: CHECK MODECHAPTER 3.3: CHECK MODETESTSTESTS
All our modules support check modeWe re-run the VCR based tests with --check
16 / 38
CHAPTER 3.4: SANITY TESTSCHAPTER 3.4: SANITY TESTSAnsible provides ansible-test for in-treemodulesSince Ansible 2.9 it can also handleCollectionsWe run ansible-test sanity --venvplugins/ across all supported Pythons
17 / 38
CHAPTER 3.6: EXPECTEDCHAPTER 3.6: EXPECTEDCHANGE TESTSCHANGE TESTS
Our test playbooks execute every task twiceThe �rst execution is expected to havechanged=True
The second changed=FalseThis ensures the modules are idempotent
18 / 38
CHAPTER 3.7: DIFF MODECHAPTER 3.7: DIFF MODETESTSTESTS
Our modules return before/after diff datato AnsibleWe access that data in our test playbooksand analyze the content
19 / 38
CHAPTER 4:CHAPTER 4:DOCUMENTATIONDOCUMENTATION
20 / 38
CHAPTER 4.1: BUILDINGCHAPTER 4.1: BUILDINGDOCUMENTATIONDOCUMENTATION
All modules have DOCUMENTATIONpopulatedWe use build-ansible.py document-plugins with a customized templateAnsible internal, our use of it breakssometimes
Would be great to have o�cial toolingAutomatic builds on Galaxy?
Need to �gure out how to autopublish docs
21 / 38
CHAPTER 4.2:CHAPTER 4.2:DOCUMENTATIONDOCUMENTATION
FRAGMENTSFRAGMENTSAnsible 2.8 introduced documentationfragmentsWe use them heavily to document commonparameters (credentials etc)Fragments for return values would be cool
22 / 38
CHAPTER 5:CHAPTER 5:FOREMANANSIBLEMOFOREMANANSIBLEMO
DULEDULE
23 / 38
CHAPTER 5.1:CHAPTER 5.1:FOREMANANSIBLEMODULEFOREMANANSIBLEMODULE
ForemanAnsibleModule is a sub-class ofAnsibleModule
Simpli�ed de�nition of common parametersin argument_specImport error handlingEntity create/update/delete/comparehelpersbefore/after diff handling
24 / 38
CHAPTER 5.2:CHAPTER 5.2:FOREMANENTITY…,FOREMANENTITY…,
KATELLOANSIBLEMODULE,KATELLOANSIBLEMODULE,……
Further sub-classing usefulForemanEntityAnsibleModule adds a stateparameterKatelloAnsibleModule makes makesorganization required
25 / 38
CHAPTER 6: LIBRARIESCHAPTER 6: LIBRARIES
26 / 38
CHAPTER 6.1: NAILGUNCHAPTER 6.1: NAILGUNWe started with the nailgun libraryOriginally developed by Satellite QETargeted at Satellite environments
no support for non-Satellite pluginsreleased at the same cadence asSatellite
Designed to test the Satellite API
27 / 38
CHAPTER 6.2: APYPIECHAPTER 6.2: APYPIEnailgun was �ne when we targeted SatelliteenvironmentsKatello (and Foreman) were moving quickerDecided to write an own API library
using the published apidoc.json, thusmostly version agnostic
Switching libraries was rather easy due tothe abstraction we've built
And tests, tests will save you!
28 / 38
CHAPTER 7: USE THECHAPTER 7: USE THEFORCEFORCE
29 / 38
CHAPTER 7.1: USE THE FORCECHAPTER 7.1: USE THE FORCEOF THE ARGUMENT_SPECOF THE ARGUMENT_SPECAnsible supports complex (nested)argument_specs
elements='dict',
options=dict(…)
Allows better checking of complexparameters
30 / 38
CHAPTER 7.2: USE THE FORCECHAPTER 7.2: USE THE FORCEOF THE ENTITY_SPECOF THE ENTITY_SPEC
We always had a need to map from Ansibleparam names to Foreman API parametersThis resulted in the introduction of theentity_spec
argument_spec extended withForeman speci�c dataThe plain argument_spec can begenerated from it
31 / 38
CHAPTER 7.3: USE THE FORCECHAPTER 7.3: USE THE FORCEOF THE ENTITY_SPEC PT. 2OF THE ENTITY_SPEC PT. 2
Many modules perform simple CRUDoperations:
take user input�nd matching entitycreate/update/delete based on inputreport
We used to have write code for that, now thisis generated from the entity_spec
32 / 38
CHAPTER 8:CHAPTER 8:COMMUNITYCOMMUNITY
33 / 38
CHAPTER 8: COMMUNITYCHAPTER 8: COMMUNITYOriginally started as "my team needs this"Quickly gained contributions from ATIXToday: 35 contributors, many from Red Hatand ATIXDevelopers, Consultants, Ops, CustomersAdding their usecases and features
34 / 38
CHAPTER 8: COMMUNITYCHAPTER 8: COMMUNITYInitial contribution was hard, duplicatedcode, hard to testIncreased contribution when we moved to amore centralized codebaseHaving a collection and RPMs madeconsumption easierRecording VCR test results is still the biggestblocker
35 / 38
CHAPTER 9: OUTLOOKCHAPTER 9: OUTLOOK
36 / 38
CHAPTER 9: OUTLOOKCHAPTER 9: OUTLOOKforeman_host supporting ALL theparameterso�cial roles to support central work�owsmore modulesdocumentation autopublishing andversioningeasier contributioncollection defaults like
?module defaults
groups
37 / 38
THANKS!THANKS!
die-welt.net
@zhenech
@evgeni
38 / 38
