intermediate Python techniquesfor the automation journey
April 2015Jeremy Schulman@nwkautomaniac
Hardly Enough TheoryBarely Enough Code
Why Automate?
Don’t be the bottleneckEmpower your team
Increase agility and productivity
Keep the business running
Automate the mundaneFree up your time
Improve uptime and reliability
Your time and your expertise
Theory● Composers, constructors
and consumers
● Decomposition
● What matters most
● Zero, one, or infinity
What You will LearnCode● Iterators and generators
● Files - JSON, YAML, CSV
● Validation - JSON schema
● Templating - Jinja2○ Basics○ Programming logic○ Custom extensions
Role RelationshipsDesign things providing instructions to othersCombination of art and scienceCreative processes and best-practices
Composers
Build finished goods based on the instructionsPracticed skills - years to master the craftSpecialized domain knowledge and tools
Constructors
Use finished goodsGenerally does not require much skillSometimes requires use of tools / products
Consumers
Decomposition
Everyone has favorites:● Modeling techniques ...● Diagramming techniques ...● Process techniques ...
Breaking a complex problem or system into partsEasier to conceive, understand, program and maintainAlso known as factoring
Entity Relationship DiagramFlowchart Diagram
Apply Decomposition
Consider a switch configuration file ...
● Create shared and unique data-sets● Define variables into explicit and
derived / calculated items● Understand cohesion of data and
code● Reuse the data for other purposes,
like validation and operations● Data can be used for multi-vendor
Step 1: separate "data" from "code"
Step 2: decompose "data"
Step 3: decompose "code"
Step 4: goto step 2 until manageable
Why do this?
What Matters Most?Does it do what it is supposed to do?Are there even any requirements?Is high availability / fault tolerant a consideration?
Functionality
How fast does it run?Is speed really even an issue?Performance
Does it need to handle the growth of your environment?Do you need to deal with concurrent processing?Scale
When it breaks, can you fix it (quickly)?Can other people work on it?What happens if you leave?
Maintainability
Zero, One, or Infinity
No need to do something so just don't do itZero
I must think about everyone else that might use this or have to deal with it one day. Empathy.
Infinity
Doing it only for myself and I don't need to worry about anyone else using it or what happens when I'm gone
One
IteratorsLoop over things● list, dictionary, set, tuple● files - each line in text file● strings - each character in string● class objects supporting the iter
protocol● generators - see next slide
List iteration code example
Example output
Why use Generators?Lazy Evaluation
Evaluation strategy which delays the evaluation of an expression until its value is needed
Benefits● Reduce memory● Generate data on-demand● Generate “infinite” data● Performance increases ● Use as abstract objects
○ as function args○ treat like any other iterablehttp://en.wikipedia.org/wiki/Lazy_evaluation
GeneratorsProduce sequence of things● behave like iterators, but lazy● generator functions use yield● generator expressions are
comprehensions enclosed by parentheses
Generator expression example
Generator function example
Loading Data from Files - CSV
Example file Example code
Example outputDo you see the differences?1. id is a string, not integer2. name is not the key
How to fix this?
Demo: Putting it all togetherTask Load all of the files, of different types, into a single dictionary variable
Pythonic techniques● iterators and generators● lambda functions● reduce function● iterables.chain● functools.partial
Demo: Filtering while loadingTaskWhile reading the data files, filter out records that match a given criteria
id >= 20
Pythonic techniques● generator● compression● … extract first/only dict item● … with if statement
Templating with Jinja2 - IntroLoaded from text filesCreated dynamically by programCommonly used for configuration
{% if age > 12 %} ...
{% endif %}
Hello {{ name | capitalize }}
Hello {{ name }}
Template content is text
Programmatic controls
Filters and extensions
Variable string interpolation
Programmatic Controls - Jinja2{% if age > 20 %}{% else %}{% endif %}
if / then / else
{% for this in some_iterable %}{% endfor %}loop over iterators
{% set myvar = some_func() %}create variables
{% include "this file.j2" %}{% include file_name %}include other files
Load template searching directories
Jinja2 "Loader" determines how to find the template files
Jinja2 "Environment" allows you to control aspects template loading and rendering
Custom Filters - Jinja2Create a sorted key list by using the inner data dictionary
Bind the function to the Jinja2 environment "filters" dictionary
Example Output
The state_list variable using in the template was created in the python code using the groupby function, see next slide ...
Example Code: Report
https://docs.python.org/2/library/itertools.html#itertools.groupby
Data Validation - JSON SchemaUsed to validate dataVerify the syntactic contentsBoth inputs and outputsSchema is written in JSON
Other schema mechanism:● ASN.1 - SNMP● YANG - NETCONF● XSD - XML
Example Data
Example Schema
More interesting exampleData Schema
● One "record" per file● Record key is max 16 alpha-chars● Only the properties:
○ state○ id○ job
JSON Schema - Docs
http://json-schema.org/latest/json-schema-validation.html#toc
Automation Summary
Programming is a toolnot a career change
It's not about youit's about everyone else
Try being right nowrather than being right
More?● JSON - https://docs.python.org/2/library/json.html● JSON Schema - http://json-schema.org● YAML - http://pyyaml.org/wiki/PyYAMLDocumentation● CSV - https://docs.python.org/2/library/csv.html● Jinja2 - http://jinja.pocoo.org/● itertools - https://docs.python.org/2/library/itertools.html● jq - https://github.com/stedolan/jq ● nwkautomaniac - https://github.com/jeremyschulman/nwkautomaniac
Jeremy Schulman@nwkautomaniac
Thank you!