DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT,...

DevOps for DishwashersBringing grown-up practices to the Internet of Things

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Christopher Biggs, Accelerando Consulting

@unixbigot @accelerando_au

1 / 133

Who am I?

Christopher Biggs — @unixbigot — @accelerando_au

▷ Brisbane, Australia

▷ Former developer, architect, development manager

▷ Founder, Accelerando Consulting

▷ Full service consultancy - chips to cloud

▷ IoT, DevOps, Big Data

#YOWConnected Sep 2017 2 / 133

Why Devops?

Agenda Problems

3 / 133

Why Dishwashers?

Agenda Problems

4 / 133

"Software is eating the world"

-- Mark Andreesen

Wall St Journal, Six years ago last month.

Agenda Problems

5 / 133

Interlude: What do I mean by "DevOps"?

DevOps is NOT THIS

DevOps is not a thing you do,

it's the way you do things.

Agenda Problems DevOps?

7 / 133

DevOps is NOT THIS

Empower everyone

to maximise value.

Agenda Problems DevOps?

8 / 133

"When every Thing is connected, Everything is connected"-- Me

Welcome to the Internet of Things

pop. 10 Trillion

Agenda Landscape

10 / 133

1936 (Information Pandemic Year Zero)

10-7 device/person

Agenda Landscape

11 / 133

Mainframe era

10-4 device/person

Agenda Landscape

12 / 133

Minicomputer era

10-2 device/person

Agenda Landscape

13 / 133

Desktop era

100 device/person

Agenda Landscape

14 / 133

Mobile era

100.5 devices/person

Agenda Landscape

15 / 133

Cloud era [YOU ARE HERE]

~101 devices/person

Agenda Landscape

16 / 133

Internet of things

103 devices/person

Agenda Landscape

17 / 133

Solve the next problem, not thelast one

Agenda Landscape Challenges

18 / 133

Beware of false analogies and straight line trends

19 / 133

Observe, Orient, Decide, Act

20 / 133

"Bad people will break your stuff"

Agenda Landscape ChallengesLandscape Challenges Risks

Do you want to know more? "The Internet of Scary Things" christopher.biggs.id.au/talk

21 / 133

Everything is awful

22 / 133

Everything is awful

and the awful is on recurl http://my-dishwasher/../../../../../../../../../../../../etc/shadow

23 / 133

It's not rocket science

No really, I mean actual rockets.

24 / 133

Desiderata

Agenda Landscape ChallengesLandscape Challenges Risks Desiderata

Select appropriate tools and platforms

26 / 133

Comprehensive identity management

27 / 133

Automate for developer and user convenience

28 / 133

Testing and testability kept front-of-mind

29 / 133

Train, and Audit, and keep doing both

30 / 133

Monitor and react (automatically)

31 / 133

Platforms

People are more expensive than circuits

(Sorry, robots)

Agenda Landscape Challenges Solutions Platforms

33 / 133

Hardware is DevOps too

(Robots, I hope this makes it up to you :)

34 / 133

Open and well supported

35 / 133

Case study: ARM v7 and Debian Linux

36 / 133

Meet the #3 top-sellingcomputer of all time

37 / 133

Artisanal free-range small-batch Linux?

38 / 133

Without the Internet, it's just a Thing.

39 / 133

"Is there anybody out there?"-- Pink Floyd

(also, my lighting controller)

40 / 133

"Put the robot back in the ocean, kid."-- Oceanographers, everwhere

41 / 133

Management is not a dirty word

42 / 133

Development

Aside: Go Serverless

Landscape Challenges Solutions Platforms Dev

44 / 133

Nice languages are portable, memory-safe and asynchronous.

45 / 133

Case study: Javascript

46 / 133

Case Study - Go

Naughty languages are like a tightrope over a pit of spikes.

48 / 133

'The wi password is "abc123';cat /etc/passwd#" '

Say no to shell scripts.

49 / 133

Use, and reuse, a framework.

Yours, mine, Google, Amazon, Microsoft, Apple, whatever.

50 / 133

AWS IoT

51 / 133

Choose your own framework adventure

▷ Amazon IoT and Greengrass

▷ Google IoT

▷ Azure IoT

▷ Open Connectivity Foundation IoTivity

▷ Resin.io

▷ Mongoose-OS

52 / 133

Containment is complexity management

53 / 133

You can run Docker on a $6.95 linux computer

54 / 133

Use your CI to produce docker images as artifacts

55 / 133

Testing

Total Infrastructure Awareness

replicate your whole ecosystem

Landscape Challenges Solutions Platforms Dev QA

57 / 133

No snow ake servers.

A dev team should have access to disposable instances of everything

58 / 133

DevOps is a disaster, every day

and that's good

59 / 133

Painful testing practices beget painfully bad testing

provide easy test data

60 / 133

Quick xes are good

and cheap

61 / 133

Listen to that annoying hipster tech blogger

At least four eyeballs per line.

62 / 133

Test-before-merge

Never* "break the build"

* well, almost never

63 / 133

Fail fast

Unit tests rst, followed by slower end-to-end tests

64 / 133

Every pair of eyeballs costs $$$

(and no, you can't save $-½ by poking one out)

65 / 133

Do not poke customers with sticks (either)

(Wherein Christopher does math to "prove" a point)

66 / 133

A modern embedded system is faster than a Cray 1

But that's still reaaally slow

67 / 133

Cross-platform CI pipelines

Option zero: cross-platform languages are win

68 / 133

Option one: Emulate

69 / 133

Option two: Enrol embedded systems in your CI

70 / 133

Case Study: My Pipelines

Minimal requirement: one x86 and one ARM server (eg Raspberry Pi)

71 / 133

Stage 1: common policy checks

72 / 133

Stage 2: compile and package

73 / 133

Stage 3: Testing (on target arch)

74 / 133

Stage 4: Deploy to container registry

75 / 133

76 / 133

Case Study: My Pipelines# note the pi build job can be run on x86 because Go is awesomebuild for pi: stage: build script: - make installdeps image contents ARCH=pi artifacts: paths: - GPIOpower - GPIOpower_docker_pi.tar.gz - GPIOpower_contents_pi.tar.gz## Run tests on RasPi#test on pi: stage: test tags: - pi script: - make test

deploy to staging: stage: deploy environment: name: staging only: - masterscript:

Package separate lab and eld artifacts

78 / 133

Defeat laziness by making it easier to do the right thing

79 / 133

Regression tests, longitudinal tests

80 / 133

Dashboards as "live tests"

Containerise your BI stack and write dashboards alongsidecode

81 / 133

Dashboards learn "green" state and alert on red

Obligatory reference to Machine Learning

Do you want to know more? "Continuous Dashboarding" christopher.biggs.id.au/talk

82 / 133

Deployment

Orchestrate: Never doanything by hand.

Landscape Challenges Solutions Platforms Dev QA Deployment

84 / 133

Build a provisioning work ow

Customise a clean OS (via ethernet or emulation)

85 / 133

Robo-con gure the target system from a provisioning system

Then save a lesystem image

86 / 133

How do you create a provisioning system?

Turtles all the way down!

Do you want to know more? github.com/unixbigot/kevin

87 / 133

Case study - My orchestration scripts

1. Create a read only recovery partition

2. Install SaltStack orchestration minion (now switch protocols)

3. Set timezone, locale, etc.

4. Change default passwords

5. Configure network

6. Provision message bus clients

7. Install language runtimes (nodejs, java etc.) if needed

8. Configure VPN client

9. Fetch initial application containers

Zoom level 1: Device roles# top.sls

'G@roles:controller': - match: compound - salt.syndic - mqtt.relay - mqtt.client - net.hostapd_bridge - net.gpsd - fleetvalid.station-console 'G@roles:sensor': - match: compound - os.initramfs - os.hostname - net.aws.cli - docker - fleetvalid.docker_auth - fleetvalid.service-advertiser 'G@roles:inroad_sensor' - match: compound - fleetvalid.inroad_sensor

89 / 133

Zoom level 2: Device pro le# fleetvalid/inroad_sensor.sls

include: - fleetvalid.aggregator - fleetvalid.rfid - fleetvalid.radar - fleetvalid.vibration - fleetvalid.rfidpower

90 / 133

Zoom level 3: Device component# fleetvalid/rfid.sls

{% set image = reg.path + '/roadcurtain/' + rfid + 'rfid-' + pillar.fleetvalid.arch + ':' + pillar.fleetvalid

fleetvalid-rfid-image: docker_image.present: - name: {{image}}

fleetvalid-rfid: docker_container.running: - image: {{image}} - links: - fleetvalid-aggregator:aggregator - fleetvalid-rfidpower:power - binds: - {{rfid_device}}:{{rfid_device}} - environment: - RCAGGREGATOR: aggregator:9091 - POWER_API: power:80 - privileged: True

91 / 133

Zoom level 4: System con gurationgpsd: pkg.installed: - pkgs: - gpsd - chrony

gpsd-device-configuration: file.replace: - name: /etc/default/gpsd - pattern: ^DEVICES=.* - repl: DEVICES="{{pillar.gps.device}}" - append_if_not_found: True

gpsd-running: service.running: - name: gpsd - enable: True

92 / 133

Hey, that all sounds a bit like PaaS

Yeah, it does.

93 / 133

Amazon AWS Greengrass

IoT PaaS built on AWS IOT + AWS Lambda

94 / 133

Resin.io

IoT PaaS with Linux and Docker

Do you want to know more? https://resin.io/

"The Internet of Scary Things" christopher.biggs.id.au/talk

95 / 133

Apache MyNewt

Embedded component-based OS for wireless sensors

96 / 133

Mongoose-OS

Multiplatform embedded OS with cloud integration and remote upgrade

Do you want to know more? http://mongoose-os.com/

"IoT in two Minutes" christopher.biggs.id.au/talk "Javascript Rules My Life (CampJS 2017)" christopher.biggs.id.au/talk

97 / 133

Automate PKI enrolment

IoT Makes PKI Easy.

98 / 133

Your "secure key distribution channel" is a cardboard box

1. Build and sign a root certificate

2. Upload root cert to the SaltStack master

3. Create minion certificate

4. Install minion certificate on minion

5. Upload a copy to the master

6. All automatically

99 / 133

Containerised version control

Use your docker registry the way it was intended

100 / 133

Use Salt "grains" to de ne which container to use(i.e. live, staging, dev or other)

fleetvalid-rfid-image: docker_image.present: - name: ((image)) - force: True

fleetvalid-rfid: docker_container.running: - image: ((image)) - links: - fleetvalid-aggregator:aggregator - fleetvalid-rfidpower:power - binds: - ((rfid_device)):((rfid_device)) - environment: - RCAGGREGATOR: aggregator:9091 - POWER_API: power:80

101 / 133

Maintenance

Self-care

FLASH memory longevity tweaks

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

103 / 133

Boot to ramdisk

Sanity check, then proceed to target environment

104 / 133

Recovery mode

(ab)use DHCP

105 / 133

Liveness monitoring

If a device goes silent, notify the site custodian

106 / 133

Sickness monitoring

Use an audible or visual attention signal (think smoke-alarms)

107 / 133

While you were sleeping

Intermittent connections

108 / 133

Kill or Cure

Feature/component disable

109 / 133

Kill or Cure

The Cassini Solution

110 / 133

Monitoring (platform data)

Heartbeats

SaltStack's presence monitor

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring

112 / 133

Flatliners

Detect missing devices

(i.e. known to saltstack but not connected)

113 / 133

Health stats

SaltStack beacons - cpu/memory, often

Full process list, less often

Network stats

114 / 133

Pour it all into a data lake

And pretend to be a Bond Villain

115 / 133

Case study: Log pooling for a building safety startup

Ram disk on local ARM devices

Streaming to cloud with Filebeat

Processing with Logstash

Set a storage budget and expire to meet the budget

116 / 133

Measurement (application data)

Use orchestration message bus

SaltStack message bus is the fast, lightweight ZeroMQ

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement

118 / 133

Can you use your orchestration bus for application events?

Yes, with care

119 / 133

Extend orchestration system with custom modules

120 / 133

Record as much as you can, digest later

Shove all your client data in ElasticSearch

Purge oldest indexes until CFO stops whinging

121 / 133

Case study: Saltstack plus ELK

Bridge orchestration bus to application message bus

Engine module at top level master (or intermediate)

Gateway messages to elasticsearch, via logstash

Want MQTT? You already built a PKI to deploy it in 2 minutes

122 / 133

Case study: MQTT plus ELK

"Rapids Rivers Ponds"

MQTT brokers at each site

Broker in the cloud federates with on-site brokers

Logstash MQTT plugin subscribes to all events

Do you want to know more? "Implementing Microservice Architectures" Fred George, YOW 2014

123 / 133

Visualisation

Real time status

Liveness, resources, environment

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation

125 / 133

Measure your KPIs

Whatever makes you money, count it

126 / 133

Measure your KPIs

Set high and low water marks, alert on them

127 / 133

Measure your KPIs

Pay: Elastic and other vendors have commercial alert engines

128 / 133

Measure your KPIs

Free: Node-RED makes a good FOSS alerting engine

129 / 133

Longitudinal comparisons

View long-term trends in KPIs

130 / 133

Longitudinal comparisons

Pay attention to device longevity, wear, etc.

131 / 133

Summary▷ Lots of devices, too many to administer by hand

▷ Swimming in a soup of malware and bad actors

▷ Choose tools that support quality

▷ Pipelines for automated build/test/stage

▷ (Ab)use traditional cloud management tools for IoT Fleet

▷ Message bus all the things

▷ Big data now, play later

#YOWC d S 2017

Landscape Challenges Solutions Coda Summary

Resources, QuestionsMy SaltStack rules for IoT - github.com/unixbigot/kevin

Related talks - http://christopher.biggs.id.au/#talks

Me - Christopher Biggs

Twitter: @unixbigot

Email: christopher@biggs.id.au

Slides, and getting my advice: http://christopher.biggs.id.au/

Accelerando Consulting - IoT, DevOps, Big Data - https://accelerando.com.au/

Landscape Challenges Solutions Coda Summary Resources

133 / 133

DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT,...

Documents

YOW! Nights - YOW! Conference · 2019. 9. 23. · Kevin O’Neill - CTO PlayUp - @kevinoneill YOW! Nights - March 2013 “What needs to be done to make the ReST architectural style

YOW! Connected 2015 Conference Keynotesyowconference.com.au/artefacts/YOWConnected2015SpeakerSummary… · YOW! Connected 2015 Conference Keynotes ... devices and the impact this

conveyor belt dishwashers

DX302 Manual EMAN_Jul03_Shh Dishwashers

PROFESSIONAL DISHWASHERS

TECHNICAL INFORMATION G1000 / G2000 Dishwashersapplianceservicesecretsmembership.com_manuals.s3.amazonaws.com... · Technical Information G1000 / G2000 Dishwashers G1000 / G2000 Dishwashers

dishwashers - Electrolux

DISHWASHERS - Maytag

YOW! 2010 Day 1

Beyond JVM - YOW! Sydney 2013

Yow che GRINCH

Beyond JVM - YOW! Brisbane 2013

SERVICE MANUAL DISHWASHERS

Dishwashers M04701 M

142 | Dishwashers Dishwashers · 2017-06-26 · 142 | Dishwashers Dishwashers Sleek, flush European styling with a design for everyone. Bosch designers believe that form must always

Beyond JVM - YOW Melbourne 2013

YOW West 2017 - YOW! Conference...YOW West 2017 OUR SPONSORS Cost of a dependency Trainer Dev at –CashiesPractical Rx Lee Campbell Author of eBook and website IntroToRx.com Conference

The dishwashers | washing machine tumble dryergaggenau.co.il/wp-content/uploads/2016/06/The...The dishwashers | washing machine tumble dryer Dishwashers 400 series 252 Planning notes

WAREWASHING UNDERCOUNTER DISHWASHERS

Thermador Dishwashers Brochure 2009